1. Introduction

Systems that convert a microscope into an imaging spectrometer, delivering the emittance, reflectance or color signal of a sample at pixel resolution, normally require affixing special optics to the microscope, fitting a scanning mechanism, or taking a large number images of the sample [1–4]. Hence such spectral imaging systems tend to be expensive. Here, we demonstrate a simple system for estimating the fluorescence spectrum at an arbitrary (pixel) location of a fluorescence image from its captured image color using just two or three image captures taken with minimal equipment. Our assemblage consists only of a 3-chip CCD camera attached to the fluorescence microscope and 1 to 2 lightly-tinted color filters. Basis spectra or eigenspectra are derived from an ensemble of known fluorescence color signals using Principal Components Analysis (PCA). Compared with Singular Value Decomposition, PCA normally yields fewer sets of eigenspectra that are needed to reconstruct color signals since the ensemble is mean-centered and the eigenvalues are more representative of the real variances of the ensemble. Fluorescence spectrum is estimated from its image color by mapping its color channel values (e.g. red, green and blue signals) to eigenspectra coefficients; Section 2.1 derives the transformation matrix. Estimating a spectrum from only a few measurement points is an underdetermined problem. However, since most fluorophores have very similar spectral characteristics, we assume that a weighted sum of the first few eigenspectra (ranked in descending order of eigenvalues) will give a good first estimate of the actual spectrum. The number of color channels that can be obtained limits the number of eigenspectra that can be used for spectrum recovery. This limitation and the fact that eigenspectra, in general, have negative values, causes the estimated spectrum to have negative values. We improve the estimate by employing a minimum negativity constraint, discussed in Section 2.2, that allows us to: (1) compute additional coefficients, and (2) put correction terms to the initial estimated coefficients derived from image color. Our approach is similar to [5] but with the addition of a minimum negativity constraint to ensure an all-positive reconstructed fluorescence color signal. Because our technique requires only at most two image captures, sample exposure time is minimal and the fluorescence spectra of live and motile samples can be examined before severe photobleaching sets in.

2. Method

2.1 Mapping from RGB to eigenspectra coefficients

Typically, fluorescent samples are excited with short wavelength light and observed with a filter that excludes the excitation wavelengths. The observed color signal C(λ) of a fluorescent sample is the product of its fluorescence emittance spectrum E(λ) and the effective transmission spectrum of the filter used, F_eff(λ), i.e.,

where F_eff(λ) is the product of the transmission curve of a dichroic mirror F_DM(λ) and a barrier filter F_B(λ),

A color camera viewing the sample generates a camera channel output Q_m given by

where S_m(λ) is the spectral sensitivity for the m-th channel. For a 3-channel color camera, Q₁, Q₂, Q₃ corresponds to the color components Red, Green and Blue (RGB), respectively.

We use an existing ensemble of color signals from known fluorescence emittance spectra of dyes and transmittances of common dichroic mirrors. We also limit our ensemble of C(λ)’s to those which are smooth and unimodal or at most, bimodal. This restriction is valid for biological samples since majority of the recorded spectra of fluorescing marine organisms or terrestial plants are unimodal and only few species have two peaks or secondary shoulders [6]. The eigenspectra e_i(λ) is obtained by applying the PCA on the ensemble. Our assumptions ensure that only a few eigenspectra are needed for its spectral reconstruction.

The reconstructed color signal of C(λ), C̃(λ)in terms of the first few significant eigenspectra is:

where C_mean(λ) is the mean of the spectral ensemble and N is the number of eigenspectra used.

Coefficients a_i are computed by

Combining Eqs (3) and (4) we get:

or Q = Ta + Q_mean where Q is the vector at the left side of Eq (6), Q_mean is the second term in the right side and

T is the transformation matrix that maps a_i to the Q_m of the image. Since we wish to recover C(λ), a’s are unknown.

For a colored image the camera signals Q are the RGB components. Hence, the only unknown variable in Eq. (6) is the coefficient vector a. By matrix operation, a can be found using

where T^-1 is the inverse of T. It is important to note that the inversion matrix T^-1 in Eq. (8) is defined only if T is a square matrix. Because the size of T is equal to M × N, T^-1 exists only if N and M are equal.

To increase M the sample is image-captured with a lightly-tinted colored filter placed before the camera. A color camera normally has M = 3, (for R, G, B) but with the filter inserted, the fluorescent sample is effectively imaged under 6 independent channels (M = 6). Changing filters and recapturing images further increase M by multiples of 3. The filters to be used must not be spectrally flat, as in the case of neutral density filters, because this will merely cause RGB values to be scaled by a constant and will lead to singularities in T^-1. Thus only lightly-tinted colored filters are suitable because, in effect, they substantially alter the spectral sensitivities of each of the camera channels.

Figure 1 shows the proposed spectral imaging system. A 3CCD camera is attached to a microscope and in-between them are changeable colored filters. A personal computer which stores the pre-computed inversion matrix T^-1, digitizes the N/3 captured images of fluorescent samples.

 

Fig. 1. Experimental setup for obtaining fluorescence color signal from image color. A microsocope is fitted with a 3CCD camera with a lightly-tinted color filter. Camera output is digitized by a computer.

Download Full Size | PDF

For each pixel value, a is computed using Eq. (8). We have found that 5 basis spectra give the least residual error in the calculation of T^-1 which implies that only two colored images are needed for spectral recovery, one taken by the colored CCD camera and another taken with a lightly-tinted color filter inserted before the camera.

2.2 Minimum-negativity constraint

Since only the first M = N eigenvectors are utilized, C̃ can erroneously contain negative values which may be due to the lack of higher-order eigenvectors. The negative values may be minimized by recovering the lost high-order eigenvectors i.e. by increasing N. Furthermore, the first N computed coefficients may still be incremented to produce an all-positive C̃. We minimize a function f defined as the sum of the squared value of the negative parts of C̃ plus an adjustment term which uses n ≥ N eigenvectors:

where α_I’s are the correction coefficients and H(λ) is the Heaviside step function:

H(λ) excludes the positive values of C̃ allowing the summation in Eq. (9) to include only the negative values. The α_j’s are derived by getting the partial derivative of Eq. (9) with respect to the unknown α_j and setting it to zero:

Performing Eq. 11 up to n ^th α_j will lead to

or B α = M where “∙” in Eq. (12) represents the dot product performed over the negative parts of C̃ only. B is the square matrix of dot products of e’s, and M is the column matrix on the right side of Eq. (12). Because the vector α is the only unknown, it is computed as

The all-positive reconstructed spectrum is then given by

Reconstruction error is measured using Normalized Mean Square Error (NMSE) defined as $\mathit{NMSE}=\sqrt{\frac{\left\{{\displaystyle \sum _{\lambda }}{\left[C_m\left(\lambda \right)-C_{\mathit{rec}}\left(\lambda \right)\right]}^2\right\}}{\left\{{\left[{\displaystyle \sum _{\lambda }}{C}_m\left(\lambda \right)\right]}^2\right\}}}$ and correlation coefficient $\mu =\frac{〈\left(C_m-C_{\mathit{mean}}\right)\left(C_{\mathit{rec}}-C_{\mathit{mean}}\right)〉}{\sqrt{〈\left(C_m-C_{\mathit{mean}}\right)\left(C_m-C_{\mathit{mean}}\right)〉〈\left(C_{\mathit{rec}}-C_{\mathit{mean}}\right)\left(C_{\mathit{rec}}-C_{\mathit{mean}}\right)〉}}$ where C_m and C_mean is the measured (true) and average color signal spectrum of the fluorescent sample, respectively. Perfect reconstruction corresponds to NMSE = 0. Because μ is the normalized measure of the strength of the linear relationship between the measured and the reconstructed spectrum, μ = 0 for uncorrelated spectra and μ = 1 for equivalent ones. Correctly-positioned but oppositely-valued peaks corresponds to μ = -1. The procedure only has one free parameter to adjust, n, which is the number of correction coefficients to recover and it may be increased beyond N until NMSE is closest to zero, μ is closest to 1 and f in Eq. (9) is lowest.

3. Experiment

The image is produced with an Olympus BH2 fluorescence microscope, captured by a Hitachi HV-C20 3CCD camera and digitized by a Matrox Framegrabber. The spectral sensitivities of the camera channels and the relative transmittance of the colored filter are posted at: http://www.nip.upd.edu.ph/ipl. The camera is pre-calibrated to give a linear response. We demonstrate our technique using two samples: (1) 0.04 μm carboxylate-modified fluorospheres (Molecular Probes) having excitation/emission at 505/515 nm, and (2) Chlorophyll b in plant leaf (Bambusa Multiplex Rivierorum). Both are viewed under a UV objective (0.65 numerical aperture, 40×) and a “G-excitation” dichroic mirror which allows the reflection of excitation light starting at 545 nm and the transmission of fluorescent emission beginning at 590 nm. The recovered color signal from the fluorosphere is clipped at the cutoff of the G-excitation dichroic mirror. We did not use the next available dichroic mirror (B-excitation) because it excludes the excitation wavelength of the fluorosphere. Calculations were performed using MATLAB.

Emittances of 121 fluorescent dyes were taken from Molecular Probes [7]. Together with the transmittances of three different dichroic mirrors, a total of 363 color signals are computed. Camera channel sensitivities were obtained from the manufacturer. All spectra are discretized at 1 nm-resolution. PCA is performed on the ensemble of color signals to yield 301 eigenspectra and the T^-1 was computed via Eq. (7) using the first 5 eigenspectra and the camera spectral sensitivity data.

The reference chlorophyll color signal was measured using SPECIM Imspector V9 Imaging Spectrometer while the reference fluorosphere spectrum was obtained from Molecular Probes [7]. To limit noise effects, the RGB from a 10×10 block representation of the sample image is averaged and used to recover the fluorescence color signal. Figure 2 shows images of Chlorophyll b (2a) and fluorospheres (2b) samples. Right column of Fig. 2 presents the measured spectra (black), the reconstructed spectra using 5 coefficients (blue) without minimum negativity, and the corrected spectra using 10 coefficients (red) with minimum negativity constraint applied. The recovered peaks are accurate to within ±5 nm of the actual peak. Figure 3 shows the dependence of f, NMSE and μ values for the two samples with the number n of recovered coefficients.

 

Fi.g 2. Fluorescence color signal reconstruction of (a) Chlorophyll b; (b) fluorospheres. In images on the right column box indicates position where spectrum is recovered. Graphs on the left column are actual spectra (black), reconstructed spectra using 5 coefficients (blue), improved spectra using 10 coefficients recovered using minimum negativity constraint (red).

Download Full Size | PDF

 

Fig. 3. Quality measures. (a) Negative values in C_rec (b) μ and (c) NMSE vs number of recovered coefficients for leaf chlorophyll, (squares) and fluorospheres (circles).

Download Full Size | PDF

4. Conclusions

We have demonstrated an inexpensive system for estimating the spectra at any arbitrary pixel location of a fluorescence image at a spectral resolution that is dependent on the information content of the spectral data ensemble. The technique uses a linear mapping of color camera outputs to the basis coefficients. Minimum negativity constraint is utilized to improve the estimate and reduce negative parts of the recovered spectra.