Segmentation of Mango Region from Mango Tree Image

D.S. Guru and H.G. Shivamurthy

Department of Studies in Computer Science, University of Mysore, Manasagangothri,
Mysore-570006, Karnataka, India
dsg@compsci.uni-mysore.ac.in, shiv_hg@rediffmail.com

Abstract. In this paper we propose a novel framework for segmentation of

mango regions from its tree image. The proposed framework consists of mango
localization followed by mapping of boundary information to the located region
for segmentation. Initially thresholding is applied to each individual color band
R,G and B by adaptive thresholding and later they are combined back.
Application of smoothing and binarization to the combined image gives the
location of mangoes along with noise. The texture features are extracted from
each location then matched with template stored in the database to eliminate the
noisy regions. Finally, locations of the mangoes are obtained and edge
information is superimposed on to those locations for segmentation. An
experiment is performed on our own dataset and efficiency is evaluated by
computing the precision, recall and F-measure with respect to the human
segmented images considering as a ground truth.
Keywords: Precision agriculture,
thresholding, texture features.

Segmentation,

Mango

localization,

Introduction

Modernizing agricultural practices with a help of an emerging technology leads to the

environmental and economic sustainability with an optimized input in production of
agricultural products. The process of identification and interpretation of in-field
spatial variability through information and technology for effective management of
agricultural practices such as soil mapping, disease mapping, weed mapping, selective
harvesting and quality analysis, is known as precision agriculture (PA). Harvesting is
the final stage of any agricultural practice. Manual process of selective harvesting will
consume more time, manpower and not accurate. Selective harvesting is very popular
today because we harvest only the matured crop, and while harvesting we also grade
the crop based on the maturity level noticed. The clare valley and Margaret river
regions of Australia is famous for wine grape cultivation. Bramley et al. [1] state that
economic benefits that may accrue to grape growers and winemakers in Australia
through the adoption of selective harvesting. Mango is one of the commonly
cultivated commercial crops in many countries including India. Till today, the process
of mango harvesting and grading is done manually and it consumes much of human
effort and time. Nowadays, researchers are trying to automate the agricultural
R. Prasath and T. Kathirvalavakumar (Eds.): MIKE 2013, LNAI 8284, pp. 201211, 2013.
Springer International Publishing Switzerland 2013

202

D.S. Guru and H.G. Shivamurthy

activities through the help of computer vision techniques, as it is very helpful to the
formers, dealers and consumers. The mango fruits should be harvested at green
mature stage. Selective harvesting of green matured mangos is helpful for maturity
grading by distinguishing ripe and unripe mangos so that ripe mangos are used at the
earliest and unripe mangos can be stored for some time. Further, quality wise grading
of mangos can also be done during selective harvesting.
So, given an image of a mango tree, the problem of selective harvesting of mangos
can be defined as a problem of localization and segmentation of mango regions from
it. On the other hand it contributes the problem of spatial correlation since both leaf
and fruits are green in color. Mango region segmentation is a process of acquiring
knowledge about mango regions in an image. It is a very first task in a mango image
analysis process such as disease mapping, variable spraying and selective harvesting.
Quality of the subsequent tasks will depend on the success of the mango region
segmentation process. Color and texture are the two essential features in mango tree
image analysis. Texture is an efficient measure to estimate the structure, orientation,
roughness, smoothness, and regularity differences of mango regions in the mango tree
image. Two usual problems in mango region segmentation are over-segmentation:
where an image is segmented into a more number of regions than the actual mango
regions in the mango tree image and under-segmentation: where an image is
segmented into a less number of regions than the actual mango regions in the mango
tree image. The over-segmentation and the under-segmentation of mango regions
happen usually in case of images with spatially varying illumination. Also, the leaves
are occluded on mangoes in the mango tree image. Hence balancing the over and
under segmentation in unconstraint environment of the mango tree image is a
challenging issue in the area of color image segmentation.
The automation of agricultural practices for significant increase in food production
is emerging out as a new challenge for computer vision community. Ducournau et al.
[2] proposed a machine vision approach to count the number of emergent radical tips
on seed-lots, under the constrained environment and segmentation is accomplished
with the help of thresholding and morphological operators. Green vegetation region
segmentation [3] using the IHS (Intensity, Hue, Saturation) and RGB (Red, Green,
Blue) color space for color feature extraction and then apply mean shift and BPNN
(back propagation neural network) for segmentation. One of the central point of
precision agriculture is the selective treatment of weeds [4] and it is achieved in three
different stages at which each different agricultural elements is extracted. They are
segmentation of vegetation against non-vegetation, crop row elimination and weed
region extraction. The segmentation is done by thresholding on the basis of dominant
G-component, dominated B-component, minimum and maximum intensity in the
image. Segmentation of lesquerella flowers was proposed by Thorp et al. [5] based on
thresholding the images in HIS color space and boundary conditions using six
parameters, including the maximum and minimum hue, saturation and intensity in
image. The popular general unsupervised segmentation of color-texture region in
image [6] is presented and it is named as J-seg which consists of two independent
steps: color quantization and spatial segmentation.
All the segmentation problems which are related to a precision agriculture are
considered to be two class problems (the pixel belongs to vegetation/non-vegetation).

Segmentation of Mango Region from Mango Tree Image

203

In this work, we made an attempt to classify pixel belongs to a region of mango or

non-mango in a mango tree image. In our framework we use texture features for
template matching and color features for sensing the object of our interest. The
proposed framework consists of two major phases they are:
(1) Mango localization, and
(2) Mapping the edge information for segmentation.
The rest of the paper is organized as follows. In section 2, we present the overview
of the proposed method. The experimentation and results are described in the section
3. Finally paper ends with conclusions in section 4.

Proposed Framework

In this section, we propose a new framework for mango region segmentation, which
consists of two phases namely, mango localization and edge mapping. In first phase,
we apply thresholding to each color band separately then combined them. The
combined image (T-image) is said to have j-number of regions. These regions are
considered as mango regions. We perform smoothing to eliminate the non-mango
regions in the thresholded image. Then apply binarization to the resultant image to get
B-image. The B-image has say k-number of regions assumed to be the mango regions.
Finally we achieve mango localization through template matching which eliminates
the non-mango regions in the B-image and results with an image (L-image) having mnumber of mango regions where j k m. In second phase, the edge information
from the original image is superimposed on to the regions of the L-image for
segmentation. The block diagram of mango region segmentation is shown Fig .1.
2.1

Image Thresholding

The reflectance on the surface of mango regions is higher than that of non-mango
regions, so higher intensity will be preserved by the regions belonging to mangos.
This is an important clue for us to distinguish mango regions from non mango
regions. Based on this clue we apply a simple thresholding to segment the mango
regions from its tree image. We separately threshold each R, G & B components in
the mango tree image and then combine the resultant images back (T-image). The
middle value of the intensity range in a given image intensity distribution will be
taken as threshold. The thresholding process can be formulated as shown in Eq. (1)
and (2).

A(i, j), A(i, j) threshold

T(i, j)=

A(i, j) < threshold

0,

(1)

Max(int ensity)Min(int ensity)

(2)
2
Here A is the mango tree color image and T is the thresholded image. Fig.2a-2d
depict the original color image and R, G, B, components of the color image given.

Where

Threshold =

204

D.S. Guru and H.G. Shivamurthy

Fig. 1. Block diagram of the proposed framework

Fig. 2. (a) A given mango tree image (RGB). (b) The R-component of the input image. (c) The
G-component of the input image (d) The B-component of the input image.

The major difficulty in mango tree image segmentation is varying illumination

which can be seen in the mango regions at upper left corner, upper middle and lower
middle of the Fig.2a. Initially we assume the regions in the thresholded image as
mango regions. Fig.3a-3d depict the thresholded images of each R, G, B component
and combined image of the original color image respectively.

Segmentation of Mango Region from Mango Tree Image

205

Fig. 3. (a) Thresholded image of R-component. (b) Thresholded image of G-component. (c)
Thresholded image of B-component. (d) Thresholded image of given input image Fig.2a.

2.2

Smoothing and Binarization

The small regions and sharp transitions in the thresholded image are non-mango regions,
because the mango regions are relatively larger regions. The sharp transitions in intensity
levels can be seen only when there is random noise in the image. So, we perform
smoothing to remove small regions and sharp transitions (non-mango regions) in the
thresholded image of each of the R,G and B components separately and combined them.
We use a 3x3 mask for smoothing because it will not affect lager mango regions and also
it results in less blurring effect. The mango region segmentation is a two class problem, we
convert each smoothed R, G and B component into binary images where, white pixels
represent mango regions and black will represent non-mango regions. Then combine the
resultant images into a single binary image (B-image) as shown in the Fig.4. The
binarization process will fixes up a contrast break-point between pixels belonging to
mango regions and that of non-mango regions.

Fig. 4. Binarized image (B-image)

206

2.3

D.S. Guru and H.G. Shivamurthy

Template Matching

Here we use template matching technique to eliminate the non-mango regions in the
B-image. For this purpose, we extract the texture properties of the original image
from the regions located in the B-image and match with the templates stored in the
database. We retain only those regions whose texture properties are similar to
mangoes template. The template of mango is as shown in the Fig.5.

Fig. 5. Few templates of mangoes

Texture features are the very fundamental and invariant properties of images. Each
image has its own texture properties which describe different image regions present in
it. In the image classification and image segmentation literatures texture properties of
images have been efficiently used. Statistical texture features proposed by Haralick et
al. [7] are used for template matching to eliminate the non-mango region in the
binarized image (B-image). Initially Gray Level Co-occurrence matrix (GLCM) is
computed for the gray image using the pair-wise occurrences of image resolutions.
And the various texture properties are calculated using the GLCM obtained. In our
frame work three different texture features are used. Let us assume that P is the gray
level co-occurrence matrix obtained from the image region rx, expressions for
different texture features which we have used are as follows.
(1). Contrast:

f 1=

N g 1

n =0

Ng Ng

n P(i, j)
i=1 i j = n j=1

(2). Correlation:

f 2=

i j (ij)p(i, j) xy
xy

(3). Entropy:

f 3=i j p(i, j)log(P(i, j)).

Segmentation of Mango Region from Mango Tree Image

207

Notation:
p(i, j) is the (i, j) th entry in a normalized gray-tone spatial dependence matrix.
x , y , x and y are the means and standard deviation of px and py.

Ng is the Number of distinct gray levels in the quantized image.

Below is the algorithm for texture based template matching.
Algorithm: template matching.
Input:
B-image (k-regions).
Output: mango localized image
Method:
1. For Ith region in the B-image, extract the corresponding region in the original
given image.
2. Compute the texture features (Contrast, Correlation and Entropy) of the
region extracted.
3. Compute the dissimilarity between texture features extracted from obtained
region and templates stored in the database.
4. If dissimilarity is less than the threshold, then region belongs to mango
otherwise it is non-mango region, neglect it.
5. Repeat the above procedure till (I=k).
End of algorithm.
The output image of the above algorithm gives location of mangoes. To accurately
show the boundaries of each mango located and also to clearly distinguish the mangos
in the cases of occlusion, we extract the edges from located regions of the original
image and superimpose on to the localized image for segmentation. We use canny
edge detection operator for edge extraction. Finally we obtain segmented image of the
original image Fig.2a is shown in the Fig.6.

Fig. 6. Segmented image

208

D.S. Guru and H.G. Shivamurthy

Experimentation and Results

3.1

Dataset

Since, we could not find any dataset with mango tree image in the literature, we have
captured 44 natural mango tree images in a natural lighting condition using Kodak
digital camera (8.2-megopixel) with resolution 480x480dpi from different directions
and with the distance of less than five feet because, the image intensity at pixel
depends on the optical properties of the surface material, the surface shape and spatial
distribution of the incident illumination. The images we captured consists mango
leaves and branches in addition to mangos. We captured images under different
illumination conditions and occlusion of leaves on the mangoes. Sample images of
our own dataset is show in the Fig.7

(a)

(c)

(b)

(d)

Fig. 7. Sample images of mango trees

3.2

Results

The quantitative results of mango region segmentation are computed on our own data
set using precision, recall and F-measures as given below,
Pr ecision(P) =

MRS
MRS+NMRS

(3)

Re call(R) =

MRS
AMR

(4)

Fmeasure =

2*P*R
P +R

(5)

Here MRS is the mango region segmented correctly, NMRS is the non mango
region segmented as a mango region and AMR is actual mango regions present in the
image (ground truth) and human segmented image considered as a ground truth. The
more segmented results are shown in Fig.8.

Segmentation of Mango Region from Mango Tree Image

Input Image

209

Output image

Image 1

Image 2

Image 3

Image 4
Fig. 8. Segmented mango tree images with original input images

In the image (3) we exactly segment the mango region, but in the case of image
(1), image (2), and image (4) there is a over segmentation due to the occlusions of leaf
on the mango which is still a challenging issue in mango region segmentation. The
graphical representation of precision, recall and f-measures obtained for our data set is
as shown below Fig.9, Fig.10 and Fig.11 respectively.

210

D.S. Guru and H.G. Shivamurthy

Fig. 9. Precision of the proposed method

Fig. 10. Recall of the proposed method

Fig. 11. F-measure of the proposed method

Segmentation of Mango Region from Mango Tree Image

211

Conclusion and Future Work

In this paper, we have proposed a novel approach for mango region segmentation
from the mango tree image. The intensity of the pixel location depends on the optical
property and shape roughness of the objects present. Texture is the only feature to
achieve discrimination between objects in an image. Through this framework we have
exploited the intensity distribution and texture features of the objects in images for the
purpose of segmentation. The segmentation of mango regions from its tree image
taken in an unconstrained environment is a very challenging task. Presently, there are
no mango tree datasets available in the literature and hence we have created our own
dataset consisting of 44 mango tree images taken in an unconstrained environment.
The quantitative performance of the proposed method is analyzed and tabulated using
Precision, Recall and F-measures. We have achieved 72.77 % of Precision, 71.43 %
of Recall and 66.70% of F-measure. The poor results (image-20) are evidence in the
Fig.11 due to the problem of illumination, leaves shadows occluded on the mango
regions. In future, we would like to enhance the efficiency and accuracy of the
proposed method by making better use of texture features and with the addition of
features such as shape, spatial geometry and solidity etc.

References
1. Bramley, R.G.V., Proffit, A.P.B., Hinze, C.J., Pearse, B., Hamilton, R.P.: Generating
benefits from precision viticulture through selective harvesting. In: Proceedings of the 5Th
European Conference on Precision Agriculture, pp. 891898 (2005)
2. Ducournau, S., Feutry, A., Plainchault, P., Revollon, P., Vigouroux, B., Wagner, M.H.: An
image acquisition system for automated monitoring of the germination rate of sunflower
seeds. Computers and Electronics in Agriculture 44, 189202 (2004)
3. Zheng., L., Zhang., J., Wang, Q.: Mean-shift-based color segmentation of images containing
green. Vegetation Computers and Electronics in Agriculture 65, 9398 (2009)
4. Burgos-Artizzu., X.P., Ribeiro., A., Tellaeche., A., Pajares., G., Fernndez-Quintanilla, C.:
Analysis of natural images processing for the extraction of agricultural elements. Image and
Vision Computing 28, 138149 (2010)
5. Thorp, K.R., Dierig, D.A.: Color image segmentation approach to monitor flowering in
lesquerella. Industrial Crops and Products 34, 11501159 (2011)
6. Deng, Y., Manjunath, B.S.: Unsupervised Segmentation of Color-Texture Regions in
Images and Video. Pattern Analysis and Machine Intelligence 23, 800810 (2001)
7. Haralick, R.M., Shanmugam, K., Dinstein, I.: Texture Feature For Image Classification.
IEEE Transactions on Systems, Man and Cybernetics SMC-3, 610621 (1973)