Automated analysis of immunohistochemical images
based on curve evolution approaches
G. Livanos, M. Zervakis
Department of Electronics and Computer Engineering
Technical University of Crete
Chania, P.C. 73100, Crete, Greece
Abstract The HER2/neu oncogene is notable both for its
role in the pathogenesis of breast cancer and its role as a target of
treatment. Qualitative or quantitative protein evaluation has
been achieved using immunohistochemistry (IHC) on frozen and
archival tissues, a widely adopted technique due to the
standardization of the internal procedural steps and its easy and
low-cost applicability to any laboratory. The goal of the present
study is to introduce an efficient tool for the automated detection
of HER2 protein overexpression in tissues, providing accurate,
instant, yet objective interpretation outcomes through a
formalized procedure. The comparison of results with
classifications by specialists who evaluated the same tissue
samples dataset confirms the efficiency and prospect of the
methodology.
Keywordssegmentation; protein overexpression; membrane
staining; active contours; image clustering; color models;
immunohistochemistry;
I. INTRODUCTION
Breast cancer constitutes one of the most frequent types of
cancer among women, yet it can be efficiently treated and
cured when it is diagnosed in early stages. Over the last years,
medical advances have led to the identification of numerous
tumor biomarkers facilitating the understanding of the
molecular basis of tumor progression and treatment response.
Prognostic markers aim to objectively estimate the patients
overall outcome, while predictive markers focus on the
objective evaluation of the possible benefits from a specific
clinical intervention. HER2/neu oncogene amplification
and/or overexpression has been highly associated with breast
tumor evolution because of its prognostic role, as well as its
ability to predict response to specific antibodies. Therefore,
the evaluation of HER2 status of the cancer is considered
extremely important [1]. Overexpression of this receptor is
observed in tumors with much higher level than in normal
tissue and is associated with increased disease recurrence,
poorer relapse-free survival and worse prognosis. A healthy
breast cell has 2 copies of the HER2 gene. Some kinds of
breast cancer arise when a breast cell has more than 2 copies
of that gene, which start to over-produce the HER2 protein,
forcing the affected cells to grow and divide too rapidly. Due
to the fact that this receptor plays a key role in connecting the
biological and clinical behavior of several types of cancers, it
is an ideal therapeutic target. Herceptin (trastuzumab) is a
978-1-4673-5791-3/13/$31.00 2013 IEEE
G. C. Giakos
Department of Electrical and Computer Engineering
The University of Akron
Akron Ohio, 44325, USA
monoclonal antibody that targets the extracellular section of
the HER2 receptor and has been proved quite effective in
treating patients as a single means or in combination with
traditional chemotherapy, being, however, active only against
HER2-overexpressing tumors [2].
The American Society of Clinical Oncology has
emphasized on the need for the assessment of HER2 in all
breast tumors, either in the early stages of diagnosis or in
repetitive intervals during the recurrence period. HER2 status
can be evaluated using imaging modalities at the DNA level
(using FISH, PCR, or Southern Blot), at the mRNA level
(using Northern Blot or RTPCR) or at the protein level (using
IHC or Western Blot) [3]. Yet, a preferred detection approach
in clinical practice should fulfill the needs for applicability to
archival pathologic tissue slices, ability to reveal their
morphological details, convenience in being routinely used in
most histology labs, and extraction of interpretable and
reproducible results. Based on these requirements, two of the
most reliable and widely applied methods include the
evaluation of gene amplification in the cell nuclei by
fluorescence in situ hybridization (FISH) and the estimation of
protein overexpression at the tumor cell membrane by
immunohistochemistry (IHC) [4]. These approaches have the
advantage of being able to correlate between HER2 expression
and morphologic features in tissue sections. FISH allows
selective staining of various DNA sequences with fluorescent
markers and accomplishes the detection, analysis, and
quantification of specific structural abnormalities within
nuclei, while IHC uses specific antibodies to stain proteins in
situ, providing the identification of many cell types.
IHC aims at detecting specific antigens in tissues or cells
based on an antigen-antibody reaction that can be visualized
by a marker (fluorescent dye, enzyme, radioactive element or
colloidal gold) and is widely used due to its low cost and high
applicability. When reporting results, the degree of HER2
protein-overexpression is scored according to the intensity of
membrane staining and the percentage of tumor cells stained.
The evaluation procedure is usually performed qualitatively by
a pathologist, who carefully observes the IHC samples via
microscopy and manually calculates the presence of cancer
cells in the breast tissue, assigning scores according to
appropriate criteria and international standards. Yet, the
interpretation of such results is subjective and causes certain
inconsistencies upon the diagnosis, as the result is highly
dependent on the experience of the specialist and the quality
of the tissue preparation stage.
In order to make immunohistochemical studies more
objective, quantitative techniques based on computer-assisted
microscopy and image analysis must be developed. In
specialized or hospital laboratories this is achieved via
commercial software and equipment of high cost, preventing
individuals from automated evaluation of gene expression.
The main stages of such an IHC procedure, for a
representative sample image, are depicted in Figure 1.
In [5], a novel technique is introduced for membrane
segmentation based on hue value thresholding and virtual cell
membrane fitting through morphology and weighted
barycentre membrane points initialization procedure.
Matkowskyj et al [6] exploited the utilities of commercial
software and the difference of signal energy between the
images derived from control tissue slide (identically treated
slide) and the experimental tissue slide (primary antibody-
exposed slide) in order to quantitatively evaluate IHC staining.
In [7], an alteration of the traditional Hough transform was
utilized as a seed-finding method for each cell along with a
partial differential equation solver to approximate the contour
of the membrane.
Fig. 1. Main steps of an automated procedure for the assessment of HER2
status from a representative immunohistochemical input image.
In [5], a novel technique is introduced for membrane
segmentation based on hue value thresholding and virtual cell
membrane fitting through morphology and weighted
barycentre membrane points initialization procedure.
Matkowskyj et al [6] exploited the utilities of commercial
software and the difference of signal energy between the
images derived from control tissue slide (identically treated
slide) and the experimental tissue slide (primary antibody-
exposed slide) in order to quantitatively evaluate IHC staining.
In [7], an alteration of the traditional Hough transform was
utilized as a seed-finding method for each cell along with a
partial differential equation solver to approximate the contour
of the membrane.
Our research is focused on the automated detection of Her-
2/neu protein in tissues, in order to diagnose breast cancer at
its earlier stages so as to guide targeted therapy, helping the
patience cure with higher probability. Advanced image
analysis techniques are adopted in order to accurately segment
the cells within the sample IHC images and precisely extract
their membrane contour and degree of staining. The
contribution of this work refers to the standardized and
accurate determination of the impact of cancer on female
organisms through the processing of IHC microscope images
via an objective, efficient and powerful tool that will reliably
complement the pathologists diagnosis and decision. The
novelty of the proposed methodology lies on the fully
automated membrane contour evaluation without prior
knowledge about the tissue structure and on the combination
of information provided through different color models, the
discrimination of objects of interest derived by clustering
approaches, and the energy minimization efficiency for
approximating region borders adopted in curve evolution
approaches. Our proposed methodology involves color
deconvolution through model conversion and thresholding, in
order to enhance intensity differences between the regions of
interest in the test images. This is followed by k-means
clustering on intensities, which reveals the key segments for
the evaluation procedure, succeeded by edge following and
linking via the active contours algorithm, in order to extract
the complete border of the cell membranes. The accurate
derivation of cell boundaries facilitates the calculation of
percentage and intensity of cell staining, producing the tissue
characterization according to the IHC scoring system. The
preliminary results of our segmentation technique reveal that
this fusion of color image information achieves accurate
segmentation even in the presence of highly intersecting
regions, such as immunohistochemical captions of breast
tissue.
II. SEGMENTATION ALGORITHM
A. Principles of image segmentation
Points, lines, regions, boundaries are among the key
features to be evaluated through the segmentation procedure.
Several approaches have been proposed in the bibliography
including point and line detection techniques, where the
detected edges are linked in order to accurately represent the
shape of each object, thresholding methods (histogram,
adaptive, multi-level), which divide the image into segments
according to distinct bands of pixel intensities, as well as
region growing/splitting methodologies, which iteratively
classify neighboring pixels of "seed-points" into a region
through appropriately selected similarity criteria [8]. Edge
segmentation methodologies appear very sensitive to noise,
while thresholding techniques lead to ambiguous boundaries
and homogenous regions, with few and non-intersecting
regions. Finally, region based segmentation approaches
introduce computational complexity and are sensitive to the
initialization of seed-points.
In this study, the input images constitute of three key
regions: the cell nuclei characterized by a blue color (IHC
staining using the DAB protocol), the cell membrane graded
from moderate to dark brown in the sections of high HER2
overexpression (characteristic marked color for DAB protocol)
and the intermediate tissue regions, which can be found at
different color bands on the image histogram and must be
neglected in the evaluation and classification process. Breast
cancer tissue-microscopy images usually contain texture, noise
due to the tissue preparation and staining procedure, artifacts
introduced by the instrument (blurring, bad contrast and
magnification) and numerous overlapping items, thus
introducing additional limitations to segmentation techniques
and preventing them from converging to accurate object
boundaries. Thereupon, in order to improve the quality of the
image partitioning results, an efficient preprocessing procedure
needs to be performed, enhancing the qualitative information
of the input image.
B. Preprocessing of immunohistochemical images
As an initial step, a transformation to a different color space
is required. The initial description of an IHC image in the RGB
model appears adequate for digital representation but is not
appropriate for color segmentation. HSV (Hue-Saturation-
Value) and Lab (Luminance - a-b color-opponent dimensions)
models decorrelate the pixel intensity from the pure color
components, facilitating the detection of specific color bands.
Exploiting the color discrimination offered by the HSV model,
the application of adaptive thresholding to the H and S
channels is capable of isolating the regions of interest within
the image. This process provides initial information regarding
the distributions of membrane and nuclei areas. Subsequently,
histogram equalization of the pure color bands (a and b) is
performed within these regions on the Lab image, as to
enhance the differences of these regions of interest and
improve their separability.
The actual pixel classification is performed using the k-
means classification scheme, acting as an intermediate and
interconnecting step between the basic segmentation algorithm
and the preprocessing stage. This step aims at identifying
pixels belonging to the cell membrane and nuclei and enables
the assessment of HER2 overexpression. Following this initial
segmentation, the cell regions are further refined by means of
an efficient edge-linking approach presented in the next
section.
C. Application of curve evolution approach active contours
Contour-based techniques are well established in
international bibliography, providing accurate and robust
results even in noisy environment, even though they suffer
from initialization, local minima and stopping criteria
problems. The fundamental principle of these techniques lies
on the linking of edge points extracted via an edge detection
scheme, attempting to exploit curvilinear continuity in order to
iteratively approximate the object borders starting from an
initialized closed curve [9]. Global minimum energy-searching
methods have been proved effective in overcoming local
minima problems, leading to robust convergence regarding the
final contour extraction.
Chan and Vese [10] proposed a powerful and flexible
methodology for active-contours object detection combining
curve-evolution techniques, level sets and the Mumford-Shah
functional, accomplishing the detection of corners and any
topological changes. The model begins with a contour in the
image plane defining an initial segmentation and then this
contour gradually evolves according to a level set method,
until it meets the boundaries of the foreground region.
According to the model, a curve C is represented via a
function (the level-set function) as C={(x, y)|(x, y)=0},
where (x, y) are coordinates in the image plane while the
evolution of the curve is given by the zero level curve at time t
of function (x, y, t) . Negative values of denote points
outside the curve while positive values of originate from
points belonging to the internal area of the curve, as depicted
in the following scheme:
Fig. 2. Representation of a curve using a level set function (x, y, t).
At any given time, the level set function simultaneously
defines an edge contour (=0) and a segment of the image
(0) and is being evolved according to the partial differential
equation (1), iteratively converging to a meaningful
segmentation of the image.
= F , (x,y,0)= 0 ( x, y )
t , (1)
where F denotes the speed of the curve evolution. Assuming
that the image I consists of two regions of approximately
constant distinct intensities I1 and I2 and that the object of
interest is represented by the region of value I1 (inside curve
C), the fitting energy functional is denoted according to
equation (2):
2 2
F (c1 , c2 , C ) + F2 (C ) = 1 I ( x, y) c1 dxdy + 2 I ( x, y) c2 dxdy , (2)
insideC outsideC
+ Length(C ) + v Area(inside C )
where C is any variable curve except for the object boundary
C0, constants c1 , c2, are the averages of I inside and outside C
respectively,and ,,1,2 are non-negative fixed user defined
parameters. The contour of the foreground region is the
solution of the minimization problem inf C (F(c1, c2, C))
0F(c1, c2, C0) . A detailed description of the complete
mathematical background can be found in [10].
III. EXPERIMENTAL RESULTS
In the proposed methodology, the cell nuclei are
represented by a band close to the blue color (middle of the
HSV cone), while the cell membrane lies on the brown color
band (beginning and end of the HSV cone). The masked and
equalized image in the Lab color space feeds the k-mean
classifier, which then feeds the active contours segmentation
algorithm. Each active contour is initialized by fixed-sized
circle regions in the centre of each estimated cell region, as
originated from the application of k-means clustering. In order
to improve the accuracy of the extracted segment boundary,
morphological operators are adopted to smooth the computed
curve, remove noisy and inaccurate borders and fill the holes
within the foreground region. Finally, the active contour
approach is applied on the edges obtained from k-means
segmentation. The main steps of the proposed cell
segmentation methodology are summarized in the following
flowchart:

Fig. 3. Flowchart of the proposed methodology.
The segmentation results are illustrated in Figure 4, revealing
efficient approximation of the cell membrane-contour and
accurate cell nuclei counting. Despite the increased level of
region intersection, the ambiguity of area borders is reduced at
the perimeter of each cell.
Fig. 4. Active-contours segmentation results. From left to right, from top to
bottom: Equalized and masked Lab-space image, boundary extraction after
algorithmic convergence, segmented image with all regions, segmentation of
cells after application of morphology and removal of intermediate tissue
regions.
IV. CONCLUSIONS FUTURE WORK
The present study proposes a novel attempt for breast
cancer cell-nuclei segmentation and membrane border
extraction, by fusing information derived from conventional
and advanced image processing techniques. The preliminary
results from the analysis of immunohistochemical images
confirm the prospect of the methodology, aiming at a fully
automated, qualitative and quantitative assessment of oncogene
overexpression that compromises with the specialists'
diagnosis. A dataset of numerous microscopy images is already
being tested and evaluated in order to validate the outcome of
this work with numerical data based on objective metrics.
Challenges in the proposed segmentation scheme include the
reduction of computational cost due to the high resolution and
complexity of the test images, as well as the introduction of
additional information originating from texture analysis.
Increased speed in active-contours extraction can be achieved
using fast level-set implementation, more clever initialization
strategies or multiscale approaches through downsampling or
wavelet analysis. In addition, a technique of fusing color and
texture is one of our future plans for the enrichment of the
current methodology.
ACKNOWLEDGMENTS
Research was supported by YPERThEN project, which
is funded by the EU INTERREG programme and funds from
Greece and Cyprus. It was partly supported by the OASYS
project funded by the NSRF 2007-13 of Greek Ministry of
Development. We thank Dr. Dourou at the Aristotle University
of Thessaloniki, for providing examples of IHC images and
evaluating the HER2 status of the sample breast cancer tissue.
REFERENCES
[1] Sylvie Menard, Elda Tagliabue, Manuela Cambiglio, Serenella M. Pupa,
Role of HER2 overexpression in breast carcinoma, Journal of Cellular
Physiology, vol. 182, pp. 150-162, 2000.
[2] David G. Hicks, Raymod R. Tubbs, Assessment of the HER2 status in
breast cancer with fluorescence in situ hybridization: a technical review
with interpretive guidelines, Human Pathology, vol. 36, pp. 250-261,
2005.
[3] Shahla Masood, Marilyn M. Bui, Prognostic and predictive value of
HE2/neu oncogene in breast cancer, Microscopy Research and
Technique, vol. 59, pp. 102-108, 2002.
[4] Z. Theodosiou, I. Kasampalidis, G. Livanos, M. Zervakis, I. Pitas, K.
Lyroudia, Automated Analysis of FISH and Immunohistochemistry
mages: A Review, International Society for Analytical Cytometry,
Cytometry, Part A, vol. 71, pp. 439-450, 2007.
[5] E. Ficarra, E. Macii, G. De Micheli, L. Benini, Computer-aided
evaluation of protein expression in pathological imaged, Proceedings of
the 19th IEEE Symposium on Computer-Based Medical Systems, IEEE,
The Computer Society, 0-7695-2517-1, IEEE, 2006.
[6] Kristina A. Matkowskyj, Dan Schonfeld, and Richard V. Benya Young,
Quantitative Immunohistochemistry by Measuring Cumulative Signal
Strength Using Commercially Available Software Photoshop and
Matlab, The Journal of Histochemistry & Cytochemistry, vol. 48, pp.
303-311, 2000.
[7] C. De Solorzano, R. Malladi, S. A. Leliea, VRE & S. J. Lockett,
Segmentation of nuclei and cells using membrane related protein
markers, Journal of Microscopy, Vol. 201, pp. 404 415, 2001.
[8] R. C. Gonzalez and R.E. Woods, Digital Image Processing 2nd Edition,
Prentice Hall, New Jersey, 2002
[9] Jitendra Malik, Serge Belongie, Thomas Leung, Jianbo Shi, "Contour
and texture analysis for image segmentation", International Journal on
Computer Vision, Issue 43, pp 7-27, 2001.
[10] Tony F. Chan, Luminita A. Vese, "Active Contours Without Edges",
IEEE Transactions on Image Processing, Vol 10, No 2, 2001.