Open Access
Add to BibSonomyAbstract
Efficient image classification of microscopic fluorescent spheres is demonstrated with a supervised backpropagation neural network (NN) that uses as inputs the major color histogram representation of the fluorescent image to be classified. Two techniques are tested for the major color search: (1) cluster mean (CM) and (2) Kohonen’s self-organizing feature map (SOFM). The method is shown to have higher recognition rates than Swain and Ballard’s Color Indexing by histogram intersection. Classification with SOFM-generated histograms as inputs to the classifier NN achieved the best recognition rate (90%) for cases of normal, scaled, defocused, photobleached, and combined images of AMCA (7-Amino-4-Methylcoumarin-3-Acetic Acid) and FITC (Fluorescein Isothiocynate)-stained microspheres.
©2001 Optical Society of America
1. Introduction
Advances in fluorescent probe manufacturing and staining techniques have allowed the multiple application of fluorophores on the same sample site resulting in the formation of a multicolored image [1–6]. Detection of chromosomal and genomic defects has been made easier and more accurate as a result of this new capability [2–3]. Multicolor fluorescent staining has also been utilized to generate high-contrast confocal fluorescent images of internal organs in the whole-mount mouse embryo [5–6]. As the fluorescence marking technology develops further and finds new applications in biology and medicine (e.g. cell counting, mutation monitoring in organogenesis, imaging through a scattering medium), the availability of a rapid and accurate method to recognize and classify fluorescent images also becomes necessary.
This paper seeks a color pattern recognition technique that is suitable for multicolor images of fluorescent microscopic objects. Color-based image classification is inherently invariant to transformations in the image that are due to rescaling, translation, distortion, and rotation. It is also robust against the effects of partial occlusion. Image classification based on color histograms was first demonstrated by Swain and Ballard with Color Indexing [7]. A color model histogram of the target pattern to be searched is obtained and similarity between several tests and the target is measured by histogram intersection. Schemes [8–10] that improved on Color Indexing were later proposed that incorporate color, geometry, and color constancy. Recent techniques jointly employ other image cues such as texture, shape and location of colors [11–12]. These color image classification techniques have thus far been demonstrated only on images of non-microscopic samples illuminated by natural or artificial light. Applications include data mining in web-based applications such as Query by Image Content (QBIC), video cut determination and quality inspection of materials [11–14].
Classification of microscopic objects requires special attention because their images are influenced by diffraction effects (defocusing, aberrations, scaling) and optical noise (edge ringings, scattering, digitization errors) [15]. In addition, fluorescent objects photobleach, that is, their efficiency to emit fluorescent light decreases with time for the same excitation power [16]. The fluorescent object under consideration is also the light source itself, thus the usual problem of recognition under varying illumination conditions is absent.
In this work, the recognition rates of Color Indexing in the classification of fluorescent microsphere images is compared with our proposed method: a supervised backpropagation neural network (NN) that uses as inputs the one-dimensional (1-D) major color histogram of the image to be classified. The major colors are extracted using two methods: (1) by taking the mean of major color clusters, and (2) by employing a self-organizing feature map. In order to limit the effects of diffraction and optical noise, the major color histogram creates a reduction of feature components, while a neural network classifier generalizes even with noisy inputs [17]. We demonstrate that our recognition method is more robust than Color Indexing against changes caused by scaling (viewing the object at different magnifications), defocusing, and photobleaching. To our knowledge, color pattern recognition for images of fluorescent microscopic objects has not yet been investigated in the presence of optical noise which degrade their signal-to-noise ratio.
The rest of the paper is organized as follows: our method is expounded in Section II. Details and results of experiments are discussed in Sec. III and Sec. IV gives our conclusions.
2. Method
2.1 Images
Four types of images were obtained from two fluorescent microsphere suspensions (Duke Scientific Corp) of FITC (diameter=20µm, excitation wavelength λE=490 nm, fluorescence wavelength λF=520 to 530nm) and AMCA (diameter=10µm, λE=364 nm, λF=420 nm). The AMCA fluorospheres were observed using an epi-fluorescence microscope (Olympus BH-2) using a dichroic mirror with a short wavelength cut-off of λc=420 nm, to generate the AMCA-blue images. The FITC fluorospheres were viewed using a dichroic mirror with λc=515 nm, to produce the FITC-green images. The same FITC microspheres were also viewed with another dichroic mirror where λc=590 nm, to generate the FITC-red images. Images of blank samples (no fluorosphere found in the field of view) were also obtained for the background images. Images were captured by a 1-CCD Sony XC-711 Camera and digitized by an 8-bit frame storeboard (DIPIX Power grabber). Note that the use of a 1-CCD color camera presents a worst-case scenario in terms of imaging weak fluorescent samples.
Eight image classes were considered for recognition: R (FITC-red), G (FITC-green), B (AMCA-blue), Bk (background), Gb (FITC-green+AMCA-blue), Rg (FITC-red+FITC-green), Rb (FITC-red+FITC-blue), Rgb (FITC-red+FITC-green+AMCA-blue). Combinations of spheres in the last four classes were made using imaging software to simulate multifluorescing samples. Figure 1 shows examples of each of the 8 image classes for classification.
Figure 1. Fluorescent microsphere image classes used in recognition experiment. From left to right: R (FITC-red), G (FITC-green), B (AMCA-blue), Bk (background), Gb (FITC-green+AMCA-blue), Rg (FITC-red+FITC-green), Rb (FITC-red+FITC-blue), Rgb (FITC-red+FITC-green+AMCA-blue).
To test a classifier’s robustness to fluorescence microscopy imaging artifacts, the fluorospheres were captured under four imaging conditions: (1) NORMAL, fresh slides are used, (2) DIMMED, photobleached fluorospheres are used, (3) DEFOCUSED, the spheres are imaged slightly out of focus, and (4) SCALED, the spheres are imaged using two different magnifications (Type: Dplan Apo UV) namely, 10× [numerical aperture (NA)=0.4] and 40×(NA=1.0). The images were cut out in different sizes ranging from 8×9 to 400×400 (SCALED) and the images were not strictly centered to make the number of background pixels arbitrary.
2.2 Color Indexing by Histogram Intersection
Classification by Color Indexing involves the color histogram of the image. To remove the dependence of the histogram on brightness variations, it is necessary that the image RGB be transformed into a color space where brightness is decoupled from chromaticity. Here we used normalized color coordinate rg - I, where the brightness is defined as I=R+G+B, and the chromaticities r and g are given by r=R/I and g=G/I. Note that b=B/I is redundant because r+g+b=1, thus it is sufficient to describe the chromaticities by only two coordinates. The color histogram of an image is then the two-dimensional rg histogram. By limiting the analysis to the chromaticity space, the technique is made robust to brightness change due to photobleaching. The Histogram Intersection between the test and the model is calculated and the test sample is assigned to the model which gives the highest intersection. The Histogram Intersection, D, is defined by
where S is the sample histogram, M is the model histogram, b is the bin index and B is the total number of bins. A model for each of the eight classes was computed from the average histogram of 30 images from each class. The rg-histograms were discretized into 64×64 bins.
2.3 Major Color Histogram
Unlike Color Indexing, our proposed method uses a histogram binned only to a few major colors. This approach aims to reduce the effect of noise on the resulting feature vector by forcing a pixel to bin into one of the major colors. The major colors were obtained in RGB (camera Red, Green and Blue) space by two ways: (1) by taking the cluster mean (CM) and (2) computing the Kohonen self-organizing feature (SOFM). Four unique color classes corresponding to R,G,B and Bk are present. For CM, the mean RGB of 30 images from each of the four classes are taken as the four major colors.
In SOFM, a single-layer network with 200 initial outputs computes the dot products of the input, x, and the weights, w. Initially set to small random numbers, the weights are updated based on the internal monitoring of performance between neurons. The declared winner is the neuron that best matches the input. Similarity matching is found using the minimum Euclidean distance criterion:
where i(x) is the index of the winning neuron, n is the iteration time, index j=1, 2, .., N is number of neurons in the lattice, and ‖.‖ refers to the Euclidean norm of the argument vector. The winning neuron is allowed an output while the rest are not. During weight update, only the winner neuron and its neighbors are allowed to adapt. If the neurons are members of the neighborhood L, its weight adapts according to Δw=η(x-w), where η is the learning rate, otherwise, the weights remain the same. Both η and L are functions of time. In our experiments, the η value was initialized at 0.9 and then made to decrease after the first 1000 iterations until it reaches the value of 0.01. Learning is achieved when ‖x(n)-w j‖ becomes less than 0.05-a choice that is based on the assumption that there should not be too many major colors learned to avoid feature overlaps but also not too few so as to neglect the differences among colors of the same hue but of different saturation. In this manner, seven major colors were obtained.
Major color histograms were obtained as follows: the dot product of each pixel RGB with the major colors is computed and the major color with the largest product is given a vote of one and the rest is given zero. The votes for each major color are tallied and the resulting histogram is divided by the total number of pixels. The thresholding and normalization of the histogram take care of the system’s invariance to illumination strength and scaling. Without the application of thresholding, it is highly possible that insignificant pixels such as those associated with the background contribute significantly to the bin count. A pixel with a color that is close to a major color will have its largest projection along the said major color and smaller projections on the other colors. With thresholding, the bin contribution of an insignificant pixel to other colors is reduced or eliminated.
Normalization on the other hand, guarantees that the histogram profile is preserved even when the object is imaged under different magnification. In effect, scaled versions of the images also give the same histogram profile due to normalization.
2.4 Neural Network Training
To classify the major color histograms into 8 image classes, a supervised feedforward backpropagation network was used. For CM, the network has a 5-4-8 architecture, while for SOFM, the architecture is 8-9-8. The extra unit in the input layer is a bias unit constanly set at 1.0. Supervised backpropagation training involves the presentation of input-output pairs to minimize the error (cost function) Eq=(1/2)·k (-dk)2, where {yk} are the NN outputs after iteration number q, and {dk} are the desired outputs, and index k=1, 2, 3. The values of the weights at (q+1) are determined according to:
where k is the learning rate, dEq/dw ji is the error derivative from unit j to i, and α is the momentum coefficient. Activation functions used were sigmoids. Desired outputs where assigned such that only one unit, corresponding to the class of the input, is trained to fire while the rest are trained to go to zero. The neural networks were trained with NORMAL images only.
3. Results and Discussions
Figure 2 shows the rg histograms of the eight classes. The first four histograms are the four unique image classes R,G,B and Bk and each has a distinct cluster in rg space. Combinations of R,G, and B have, as expected, histograms which are combinations of histograms of each class present. Table 1 lists the major colors computed by CM and Table 2, by SOFM.
Figure 2. Histogram models in rg-space of the eight fluorescent image classes. From left to right, R, G, B, Bk, Gb, Rg, Rb, Rgb. Frequency of pixel occurrence is shown in a rainbow color map with red as the highest and blue the lowest.
The three classifiers tested are: (1) Color Indexing using rg histogram (CI), (2) Neural Network using histograms from Cluster Mean (CM+NN) and (3) Neural Network using histograms from SOFM (SOFM+NN). Each classifier was tested on the following number of novel images for each class: R 107, G 93, B 152, Bk 107, Gb 30, Rg 30, Rb 30 and Rgb 30.
Table 1. RGB components of the major colors computed using Cluster Mean.
Table 2. RGB components of the major colors computed using SOFM
Tables 3 to 5 show the confusion matrices and overall recognition rates for CI, CM+NN and SOFM+NN. The best performing classifier is SOFM+NN with a total recognition rate of 90%, followed by CI with 78% and then by CM+NN with 66%.
Table 3. Confusion matrix and recognition rates (%RR) for Color Indexing(CI).
Table 4. Confusion matrix and recognition rates (%RR) for Cluster Mean+Neural Network (CM+NN).
Table 5. Confusion matrix and recognition rates (%RR) for SOFM+Neural Network (SOFM+NN).
Next, the classifiers were tested for cases of defocused (DEFOCUSED), photobleached (DIMMED) and scaled (SCALED) images of R, G, B only. For DEFOCUSED, there are 60, 56 and 106 images of R, G, and B, respectively. Table 6 shows a summary of the recognitions rates for the three classes. In all cases, the best performing classifier was SOFM+NN. Color Indexing shows very poor recognition except with photobleached images. As expected, the use of the brightness-independent rg space in computing the histograms for CI makes the classification of photobleached image successful in such a method.
Table 6. Summary of recognition rates (%RR) for defocused, photobleached and scaled or magnified images.
IV. Conclusions
Color images of fluorescent microspheres were classified using Color Indexing and a neural network-based classifier to determine which analyser is more efficient for image recognition of microscopic fluorescent objects that are degraded by optical noise. Normal, photobleached, defocused and scaled images were used for testing. The three possible classifiers utilized were: (1) Color Indexing using rg histogram (CI), (2) Neural Network using histograms from Cluster Mean (CM+NN), and (3) Neural Network using histograms from SOFM (SOFM+NN). Our results show that the best color-based classifier is one which uses an SOFM for major color extraction, a thresholded major color histogram as feature, and a neural network for classification. The SOFM+NN exhibited a recognition success rate of 90% for the test images. Feature reduction, histogram thresholding and normalization reduces the effect of diffraction, optical noise and scaling on the histogram profile of a fluorescent image while a neural network classifier allows generalization even with noisy inputs.
References and Links
1. S. Andersson-Engels, J. Johansson, and S. Svanderg, “Multicolor fluorescence imaging systems for tissue diagnostics,” Bioimaging Two-Dimensional Spectroscopy, Proc. SPIE 1205, 179–189 (1990).
2. M.R. Speicher, S. Gwyn Ballard, and D. Ward, “Karyotyping human chromosomes by combinatorial multifluor FISH,” Nature Genetics 12, 368–375 (1996). [CrossRef] [PubMed]
3. E. Schrock, S. du Manoir, T. Veldman, B. Schoell, J. Weinberg, M.A. Ferguson-Smith, Y. Ning, D.H. Ledbetter, I. Bar-Am, D. Soenksen, Y. Garini, and T. Reid, “Multicolor spectral karyotyping of human chromosomes,” Science 273, 494–498 (1996). [CrossRef] [PubMed]
4. S. Abrams, “Fluorescent Markers: GFP Joins the Common Dyes,” Biophotonics International 5, pp 48–54 (March/April 1998).
5. C. Saloma, C. Palmes-Saloma, and H. Kondoh, “Site-specific confocal fluorescence imaging of biological microstructures in a turbid medium,” Phys Med Bio 43, 1741–1759 (1998). [CrossRef]
6. C. Palmes-Saloma and C. Saloma, “Long-depth imaging of specific gene expressions in wholemount mouse embryos with single photon excitation confocal fluorescence microscope and FISH,” J. Structural Biology 131, 56–66 (2000). [CrossRef]
7. M. Swain and D. Ballard, “Color indexing,” International J. Computer Vision 7, 11–32 (1991). [CrossRef]
8. B. Funt and G. Finlayson, “Color constant color indexing,” IEEE Trans Pattern Analysis Machine Intelligence 17, 522–529 (1995). [CrossRef]
9. D. Slater and G. Healey, “Global color constancy: recognition of objects by use of illumination-invariant properties of color distributions,” J. Opt. Soc. Am. A 11, 3003–3010 (1994). [CrossRef]
10. P. Ennesser and G. Medioni, “Finding Waldo, or focus of attention using local color information,” IEEE Trans. Pattern Analysis and Machine Intelligence 17, 805–809 (1993). [CrossRef]
11. B. Mel, “SEEMORE: Combining color, shape, and texture histogramming in a neurally inspired approach to visual object recognition,” Neural Computation 9, 777–804 (1997). [CrossRef] [PubMed]
12. J. Lampinen and S. Smolander, “Self-organizing feature extraction in recognition of wood surface defects and color images,” Intl. J. Pattern Recognition and Artificial Intelligence , 10, 97–113 (1996). [CrossRef]
13. C. Faloutsos, W. Equitz, and M. Flickner et al., “Efficient and effective querying by image content,” J. Intelligent Information Systems 3, 231–262 (1994). [CrossRef]
14. B.-L. Yeo and B. Liu. “Rapid scene analysis on compressed video,” IEEE Trans. Circuits and Systems for Video Technology 5, 533–544 (1995). [CrossRef]
15. S. Inoue, Video Microscopy, (Plenum Press, New York1986).
16. B. Herman, Fluorescence Microscopy2nd Ed. (Springer-Verlag, Singapore1998).
17. M. Soriano and C. Saloma, “Improved classification robustness for noisy cell images represented as principal-component projections in a hybrid recognition system,” Appl. Opt. 37, 3628–3838 (1998). [CrossRef]
