1. Introduction

Multi-color fluorescence microscopy imaging is widely used to simultaneously observe multiple substances at the subcellular, cellular, and tissue levels in biological samples [1–4]. This technology usually labels different biological substances with different fluorescent chromophores, and then separates them by sequential excitation [5], spectral splitting [6], or spectral imaging [7,8]. Recently, we developed a multi-color wide-field fluorescence imaging method called colorimetry camera-based fluorescence microscopy (CCFM) [9], where the monochrome camera in a typical fluorescence microscopy system is replaced with a colorimetry camera containing both gray pixels and color pixels (including R, G, B, NIR pixels). Both fluorescence intensity and spectrum information could be recorded by the colorimetry camera with a single-shot raw image, and then a multi-color fluorescence image can be decoded from this raw image by demosaicking and fluorescence separation. This method simplifies the optical setup and requires no additional image alignment for multi-color fluorescence microscopy, while keeping a high spatial resolution and low color crosstalk. Therefore, CCFM provides a new opportunity for high-throughput and low-cost multi-color fluorescence microscopy applications.

Currently, a major problem in the CCFM method is the poor color separation ability in low light levels. Insufficient fluorescence signal decreases the raw image quality and the precision of demosaicking. Moreover, the colorimetry camera is developed from the scientific Complementary Metal–Oxide–Semiconductor (sCMOS) technology, which seriously suffers from pixel-dependent noise [10,11]. In colorimetry camera, pixel-dependent noise causes intensity variation in adjacent pixels, including pixels in different color channels. This variation will further decrease the precision of demosaicking. These two factors result in inaccurate images in different color channels, and reduce the color separation ability of the CCFM method. Note that poor color separation ability would limit the application field of this method. For example, for high-throughput single-molecule Fluorescence Resonance Energy Transfer (smFRET) [12–14], a poor color separation ability will reduce the calculation precision of FRET conversion rate, which is crucial for FRET-based experimental analysis, such as resolving the state of molecules.

To improve the color separation ability for low light CCFM, the first idea is to improve the demosaicking precision by introducing noise correction in the demosaicking process. Researchers have developed various methods for joint demosaicking and denoising [15–21]. But these methods consider the noise properties of all pixels to be the same, while the noise in sCMOS camera is pixel-dependent. Furthermore, when applied to a new camera including our colorimetry camera, these methods need to be additional adapted based on color filter array (CFA) pattern. So we cannot use these joint demosaicking and denoising methods directly. sCMOS camera researchers have developed noise correction methods for pixel-dependent noise [22,23]. But these noise correction methods are developed for monochrome cameras and do not consider CFA pattern or demosaicking in colorimetry camera. Therefore, it is necessary to develop a new algorithm considering both sCMOS pixel-dependent noise and demosaicking in colorimetry camera to improve the demosaicking precision. Another idea to improve color separation ability for CCFM is to optimize dye combination. Optimizing dye combination based on camera characteristics will help to increase the imaging differences between fluorescence channels on colorimetry camera. Although previous researches performed fluorescence imaging with color cameras (including R-G-B [24], R-G-B-NIR [25,26], W-R-G-B-NIR [9]), but they did not study dye combination selection for color separation crosstalk reduction.

In this paper, we develop a dye combination selection method and a joint demosaicking and denoising algorithm for CCFM to improve its low-light color separation ability. Firstly, we propose a dye combination selection method based on analyzing the photon response difference between color channels in colorimetry camera, and certify this method by experimentally comparing different dye combinations. Secondly, we develop a joint demosaicking and sCMOS denoising algorithm to correct sCMOS noise during the demosaicking process. Finally, we use these two strategies to improve the color separation ability of CCFM for four-color fixed cos-7 cells imaging. We find that low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and the crosstalk rate is < 5% when the signal is ∼ 100 e-.

2. Theory and methods

2.1 Crosstalk in CCFM

Crosstalk is one of the most important indicators for evaluating multi-color fluorescence imaging methods. It refers to the probability of identifying one kind of fluorescence signal as another. For CCFM, crosstalk is positively correlated with the pixel value fluctuation (i.e. noise) in decoded fluorescence images. We analyze the crosstalk of the CCFM by analyzing the noise of decoded fluorescence images.

The relationship between camera channel images (i.e. images after demosaicking before fluorescence separation) X_C (X_W, X_R, X_G, X_B, X_NIR) and the decoded fluorescence images S_FM (S_F1, S_F2, S_F3…) can be approximated as:

 

Fig. 1. The relationship between camera channel images and decoded fluorescence images. X_C and S_FM represent camera channel image and decoded fluorescence image, respectively. The column vector of NCI matrix (N) can also be approximated as the rQE values of each camera channel at the peak wavelength of the fluorescence emission spectrum. In the right figure, the solid lines represent the rQE curves of each camera channel (including W, R, G, B, and NIR), and the shaded curves represent the emission spectrum of dyes.

Download Full Size | PDF

According to the sCMOS noise model [10], the pixel value of camera channel image should obey the Gaussian distribution $\widetilde {X}\mathrm{\sim G}({X,\sigma_X^2} )$, when Poisson noise is approximated to Gaussian noise. X and $\sigma _X^2$ represent the true signal value and noise of a pixel, respectively. $\sigma _X^2$ can be expressed as:

Putting $\widetilde {X}\mathrm{\sim G}({X,\sigma_X^2} )$ into Eq. (1), the decoded fluorescence images ${\widetilde {S}_{FM}}$ should also obey Gaussian distribution: ${\widetilde {S}_{FM}}\mathrm{\sim G}({{S_{FM}},\sigma_{FM}^2} )$. S_FM and $\sigma _{FM}^2$ can be calculated as:

For a certain fluorescence signal X, a high $\sigma _{FM}^2$ value means a noisy decoded fluorescence image and high CCFM crosstalk. Based on Eq. (4), to reduce the noise of decoded fluorescence image, $\textrm{abs}({{N^ + }} )$ and $\sigma _X^2$ should be reduced, which represents optimizing the dye combination and correcting the noise of camera channel images, respectively. Note that other factors, such as fluorescence image decoding algorithm, will also bring noise and crosstalk to the decoded fluorescence images, but they are not discussed in this paper.

2.2 Dye combination selection

To reduce the crosstalk of CCFM, we perform dye combination selection based on σ_FM /S_(NCI) analysis. Since different dye combinations will constitute different NCI matrices, the quality of different dye combinations could be compared by calculating the σ_FM /S_(NCI) of decoded fluorescence images. To obtain the NCI matrix for a dye combination, the NCI of different dyes should be estimated. There are two methods to estimate NCI, one method is based on imaging experiments and the other method is based on rQE curve.

For the method of imaging experiments, images of single-color samples need to be captured and used to calculate camera channel images. Then, the NCI of each pixel is calculated as the ratio of each camera channel image to the white channel image, and the mean NCI value of all non-background pixel is used as the NCI of the dye. This method is more accurate because it considers the effects of imaging system, such as filters, transmission efficiency of lens, etc. But this method requires additional sample preparation and imaging experiments, making it complicated to implement. We used this method when comparing dye combination experimentally or optimizing the dye combination of previous work [9].

For the method of rQE curve, the peak emission wavelength of a fluorescent dye is used as the NCI wavelength of the dye (WL_NCI), and we read the rQE value at WL_NCI (see Fig. 1) as the NCI value of the dye. This method is much easier to implement, but it does not consider the effects of actual imaging factors. However, since the rQE curve of the colorimetry camera usually changes more slowly than a dye emission spectral curve, the accuracy of this method is generally acceptable. This method can be used for pre-analyzing the suitability of CCFM for specific dye combination, when performing fluorescence imaging experiments is cumbersome. We used this method to find the optimal dye combination by changing the WL_NCI in the range of 400 nm ∼ 850 nm at every 10 nm interval.

2.3 Image processing

There are two kinds of image processing procedures in this work. One for general biological fluorescence microscopy images, and another for the uniformly illuminated fluorescent solution images. All the image processing procedures were performed based on self-written MATLAB (MathWorks) code.

For the procedures of processing biological fluorescence microscopy images, we develop a new procedure to improve the multi-color separation ability of colorimetry camera, and compare it with our previous procedure [9]. The new procedure is mainly divided into three steps, and the flow chart is shown in Fig. 2(a). Firstly, pixel-level noise correction is performed on the CFA image. Secondly, the CFA image is decomposed into camera channel images X_C (including X_W, X_R, X_G, X_B, and X_NIR) by a sCMOS demosaicking method while considering sCMOS noise. Finally, decoded multi-channel fluorescence images are obtained using NCI evaluation [9]. We denote the first and second step as the joint demosaicking and sCMOS denoising method. These two steps were used to substitute the demosaicking method residual interpolation [27] used in our previous procedure [9].

 

Fig. 2. Flow chart of biological microscopy image processing. (a) Complete flow chart. (b) Flow chart of sCMOS demosaicking. Note that in the sCMOS demosaicking method, sCMOS noise is considered and corrected.

Download Full Size | PDF

CFA image denoising includes two steps. One step is fixed pattern noise (FPN) correction, and the other step is white pixel read noise correction. FPN correction is performed based on our previous work [10] (See Supplement 1 for the detailed method). White pixel read noise correction is mainly based on the noise correction for sCMOS camera (NCS) method [23], and it includes three steps. Firstly, all W pixels in the FPN corrected CFA image (X_dF) are extracted to form a down-sampled gray image. Secondly, NCS is used to perform read noise correction on the down-sampled gray image. Finally, the pixel values of the NCS corrected image are used to replace the corresponding W pixel values in the original X_dF.

sCMOS demosaicking method mainly base on residual interpolation method for demosaicking [27], and includes the sCMOS denoising (NCS method) [23], the detailed process is shown in Fig. 2(b). NCS method combines high-frequency denoising with maximum likelihood estimation, and realizes sCMOS image correction by iteratively optimizing the cost function. When calculating maximum likelihood estimation function, NCS needs to use the raw image value of each pixel. However, for the colorimetry camera used in this work, the raw images for each camera channel are incomplete. Therefore, it is necessary to obtain the interpolated images of each camera channel by demosaicking, and then denoise the interpolated images by minimizing an adapted cost function.

Firstly, the white pixel corrected CFA image (X_dW) is used as original image, to calculate the W channel interpolated image (X_it) by the guide image calculation method in residual interpolation [27]. And the initial estimated image (X_e,init) is calculated by a linear combination of X_it and X_dF:

Secondly, change the estimated image (X_e) by minimizing the cost function f, and obtain the optimized image (X_O) after minimization. The minimization is performed by the function ‘fminunc’ in MATLAB. The cost function f is calculated as:

LLS_C represents the negative log-likelihood function considering both raw signal value and noise characteristics. The calculation principle of LLS_C is similar to the simplified negative log-likelihood in NCS. The difference is that, in our method, only the pixels of the decoded camera channel (e.g. W channel for white channel image calculation) are used:

σ_N represents the noise part filtered by OTF mask:

dm_NP-C represents the difference between the estimated image and the interpolated image:

Finally, use X_O as X_e to repeat the second step for 10 ∼ 15 times, and the final white channel image (X_W) can be obtained.

For other camera channels (including R, G, B, and NIR), firstly, using X_dF as the original image and X_W as the guide image, calculate the interpolated image (X_it) by the red or blue channel image calculation method in residual interpolation [27]. Then calculate the initial estimated image as Eq. (5). Secondly, calculate the cost function as described above, except the channel C should be changed from W channel to R, G, B, or NIR channel. Finally, after 10 ∼ 15 iterations for each camera channel, the final color channel image (X_R, X_G, X_B, or X_NIR) can be obtained.

NCI evaluation for separating fluorescence images is the same as our previous work [9]. Firstly, single-color images of each fluorescence channel are captured. Secondly, camera channel images are calculated from raw images, and the NCI (including N_R, N_G, N_B, N_NIR) of non-background pixel is counted as the NCI probability distribution of each fluorescence channel. Finally, based on these NCI probability distributions, NCI maximum likelihood estimation is performed to calculate the intensity ratio in each fluorescence channel for every pixel. And the decoded fluorescence images of each channel can be calculated by multiplying this intensity ratio map with the white channel image.

For the procedure of uniformly illuminated fluorescent solution images, because there is no biological structure or morphology information in these images, general demosaicking methods are ineffective. We directly used the signal value in raw CFA images to substitute the camera channel images in the biological fluorescence image procedure. Then the pixel fluorescence intensity probability distribution function (PDF) of each fluorescence channel could be counted, and used for crosstalk or σ_FM /S calculation. In particular, to verify the σ_FM /S_(NCI) value, we imaged the mixture of fluorescent solutions, and used the ratio of the standard deviation to the mean value of pixel fluorescence intensity probability distribution function (PDF) as experiment σ_FM /S_(EXP). The detailed information is in Supplement 1.

2.4. Imaging experiment

Microscopy imaging experiments were performed on an Olympus IX73 microscope system with a 60X/NA1.42 oil lens. A 405 nm laser (LWVL405, Laserwave), a 532 nm laser (LWGL532, Laserwave), and a 640 nm laser (LWRL640, Laserwave) were combined by a fiber combiner with vibration motor to form a multi-color homogeneously illuminated excitation laser at the object plane [28]. The fluorescence signal was filtered by a dichroic mirror (ZT405/488/532/640rpc-XT, Chroma) and a fluorescence filter (ZET405/488/532/642m, Chroma), and then collected by a colorimetry camera (Retina 200DSC, SN: 006, Tucsen Photonics).

The fluorescent solutions were prepared by dissolving the following 9 kinds of fluorescent dyes with emission wavelengths ranging from 400 nm to 850 nm in double distilled water: Calcein blue (C153992, Aladdin), Calcein (C866390, Macklin), Rhodamine 123 (R299309, Aladdin), Rhodamine 6G (R105623, Aladdin), Rhodamine B (R104963, Aladdin), Sulforhodamine101 (S131262, Aladdin), Nile blue (N141384, Aladdin), Sulfo-CY5.5 carboxylic acid (R-C-3575, Xi'an Ruixi Biotechnology), Sulfo-CY7.5 carboxylic acid (R-H-3059, Xi'an Ruixi Biotechnology). To dissolve the CY5.5 and CY7.5 in water, Sulfo-CY5.5 carboxylic acid and Sulfo-CY7.5 carboxylic acid were used. The dissolved concentrations were adjusted to make their electronic signal levels close to each other on the camera. When performing imaging experiment, 20 µL of the fluorescent solution was dropped between two coverslips (12545E, Fisherbrand) and directly placed on the microscopy imaging system.

For fixed cos-7 cells imaging, nuclei, PMP70, and mitochondria were labeled with DAPI, AF532, AF750, respectively. And microtubules were labeled with DL650 or AF568. The cos-7 cells sample preparation method was same as previous work [9].

2.5. Imaging assessment

Crosstalk in CCFM was mainly characterized by calculating the decoded fluorescence images of single-color samples [7,9]. We applied image analysis procedures mentioned in Section 2.3 to calculate multi-color fluorescence images from the raw CFA images of single-color samples. Since we know exactly that only a certain type of emitters should be correctly identified in these calculated multi-color fluorescence images, we can quantify the crosstalk by calculating the fraction of signals that are wrongly identified as the signal from other types of emitters.

Supposing crosstalk characterization with the samples labeled with fluorescence dye fc, and its decoded fluorescence images on each fluorescence channel are SL_fc,m (SL_fc,1, SL_fc,2, SL_fc,3…). The crosstalk of fluorescence dye fc on fluorescence channel mc (mc ≠ fc) should be

For images of multi-color samples, ROIs only containing one biological structure could be used for crosstalk characterization [29]. Since nuclei, PMP70, and mitochondria usually overlapped with other structures in our images, only ROIs of microtubules were used for multi-color sample crosstalk characterization. More specifically, 10 ROIs of 7 × 7 pixels area were used to calculate crosstalk based on Eq. (10). And the mean value of the maximum crosstalk for each ROI was used as the representative crosstalk.

Peak Signal-to-Noise Ratio (PSNR) is a common-used parameter to assess image quality. It is defined as

Signal intensity was characterized as the mean electronic signal in white channel images, in this work. For uniformly illuminated fluorescent solution images, signal intensity is the average value of all W pixels. For biological images, signal intensity is the average value of all non-background pixels in white channel image.

3. Results and discussion

3.1 Dye combination selection

3.1.1 Dye combination selection based on σ_FM /S_(NCI)

By comparing the σ_FM /S_(NCI), the WL_NCI of the best dye combinations was investigated, as shown in Fig. 3(a). When the total fluorescence channel number (M) is 3, the WL_NCI of the best dye combination is blue (430 nm), green (550 nm), and red (610 nm). For this dye combination, the difference between these three dyes can be clearly separated by R, G, B channels in the colorimetry camera. When M is 4, the WL_NCI of the best dye combination is blue (440 nm), green (550 nm), red (620 nm), and near-infrared (830 nm). When M is 5, the WL_NCI are 470 nm, 570 nm, 590 nm, 670 nm, and 770 nm. Note that there are two relatively close wavelengths, 570 nm and 590 nm, and they mainly utilize the rapid changes of R and G channels in the colorimetry camera around 580 nm (see Fig. 1 right, the rQE function).

 

Fig. 3. Dye combination selection results. (a) Best dye combination for different total fluorescence channel number (M). The WL_NCI of the best dye combination is represented as a solid area labeled with F. (b) Selecting two dyes for a dye combination based on σ_FM /S_(NCI). 2-D maps in (b) are the σ_FM /S_(NCI) value for a specific dye combination. σ_FM /S_(NCI) value of white dashed box area in (b) is relatively low, which denotes a good choice for dye combination selection. Color bar in (b) are 0.15 ∼ 0.3, 0.2 ∼ 0.4, 0.25 ∼ 1, and 0.25 ∼ 1 from left to right, respectively.

Download Full Size | PDF

As for dye selection in experiments, we took common-used Alexa Fluor series dyes (see Supplement 1, Table S1) as an example, and combined them with DAPI to achieve the dye combination described above. The peak emission wavelength of DAPI is around 461 nm, which means other dyes in blue band are no longer suitable for this dye combination. When M is 3, the most suitable dye combination should be DAPI, AF532, AF594 or AF568. When M is 4, the most suitable dye combination should be DAPI, AF532, AF594, and AF790. When M is 5, the most suitable dye combination should be DAPI, AF546, AF568, AF647, and AF750.

Additionally, since the NCI value of adjacent wavelengths are generally close, using a dye combination with emission wavelength around the best WL_NCI can also be a good choice. For instance, when M is 3, the best WL_NCI for the blue band is 430 nm and adjacent wavelengths (410 nm ∼ 480 nm) are also good choice.

In order to show that the adjacent emission wavelengths are also ideal choice, we considered 4 cases where two dyes need to be selected, and their two-dimensional functions of σ_FM /S_(NCI) changed with WL_NCI are shown in Fig. 3(b). We obtained the NCI value of DAPI, AF532, DL650, and AF750 (Table S2) based on previous work [9], and used some of these dyes as the identified dyes for the cases where M is 3 or 4. For each case in Fig. 3(b), we found there is an area instead of a point with relative low σ_FM /S_(NCI) value, and we considered the area as good dye combination area. As shown in Fig. 3(b), when M is 2, the WL_NCI of two dyes should be within 400 nm ∼ 600 nm and 600 nm ∼ 850 nm, respectively. Especially, when the WL_NCI of two dyes are within 400 nm ∼ 480 nm and 600 nm ∼ 700 nm, it is a good choice. When M is 3 and one of the dyes is DAPI, the WL_NCI of the other two dyes should be within 520 nm ∼ 570 nm and 590 nm ∼ 680 nm, respectively. When M is 4 and two of the identified dyes are DAPI and AF750, the WL_NCI of the other two dyes should be within 510 nm ∼ 580 nm and 600 nm ∼ 640 nm, respectively. When M is 4 and two of the dyes are DAPI and AF532, the WL_NCI of the other two dyes should be within 600 nm ∼ 650 nm, 770 nm ∼ 850 nm, respectively.

Moreover, based on the above two analysis results when M is 4, the dye combination used in our previous work [9] could be optimized. The peak emission wavelength of DL650 (672 nm) is not within 600 nm ∼ 640 nm. According to Table S1, it is better to use AF568, AF594, or AF610 instead of DL650.

Taking AF568 as an example, we reconsidered this dye combination optimization based on Fig. 1. For the dye combination we used in our previous work [9] (including DAPI, AF532, DL650, AF750), the photon response of DAPI and AF532 is significantly stronger than other dyes in blue and green camera channel, respectively. So DAPI and AF532 are easier to separate, and the difficult problem should be separating AF750 and DL650. The peak emission wavelength of AF568, DL650, and AF750 are around 603 nm, 672 nm, and 775 nm, respectively. Comparing the photon response of colorimetry camera around these three wavelengths, the rQE of white and red channels are close (see Fig. 1 right). But the blue and NIR channels are clearly different, so they are important for separating another two dyes. More specifically, the rQE of blue and NIR channels around 607 nm, 672 nm, and 775 nm are < 0.2, ∼0.4, and >0.55, respectively. Obviously, there is a larger gap between 607 nm and 775 nm, which makes AF568 to be a better candidate dye for this dye combination than DL650.

3.1.2 Verification of dye combination selection method

Then, we performed a fluorescent solution imaging experiment to verify the dye combination selection method based on σ_FM /S_(NCI). Nine kinds of fluorescent solutions with emission wavelengths ranging from 400 nm to 850 nm were imaged individually, including Calcein blue (D1), Calcein (D2), Rhodamine 123 (D3), Rhodamine 6G (D4), Rhodamine B (D5), Sulforhodamine101 (D6), Nile blue (D7), CY 5.5 (D8), and CY 7.5 (D9). Their NCI values are shown in Supplement 1, Table S3, and used to calculate σ_FM /S_(NCI).

We first verified the σ_FM /S_(NCI) value of decoded fluorescence images by comparing the σ_FM /S_(NCI) with the σ_FM /S_(EXP). Note that σ_FM /S_(NCI) was calculated from single-color solution images, while σ_FM /S_(EXP) was calculated from single-shot mixture solution images. Three groups of dye combinations were chosen, Rhodamine 6G & Nile blue (D4 & D7), Rhodamine B & CY 7.5 (D5 & D9), and Sulforhodamine101 & CY 7.5 (D6 & D9), respectively. The comparison results are shown in Fig. 4(a). We found that the σ_FM /S_(EXP) is close to the σ_FM /S_(NCI) for all three dye combination groups, which means the σ_FM /S_(NCI) can be used to estimate the precision of the decoded fluorescence images calculated from single shot multi-color images.

 

Fig. 4. The verification of dye combination selection based on σ_FM /S_(NCI). (a) Comparison of σ_FM /S_(NCI) with σ_FM /S_(EXP). (b) The experimental crosstalk and (c) the σ_FM /S_(NCI) of different dye combinations for 3 different dye selection conditions. For (b) and (c), the coordinates represent the serial numbers of dyes (D1 ∼ D9), and each small square represents a dye combination. The dye combination with the lowest value (σ_FM /S_(NCI) or experimental crosstalk) is circled in red. Compared to (b) and (c), a dye combination with low σ_FM /S_(NCI) would also have low experimental crosstalk. Color bar: (b): 0 ∼ 20%; (c): left 0.08 ∼ 0.3; middle 0.1 ∼ 0.4; right 0.1 ∼ 1. Signal X for (b, c) is ∼ 400 e-.

Download Full Size | PDF

Next, we verified the dye combination selection methods based on σ_FM /S_(NCI). We investigated whether we could select a low crosstalk dye combination by comparing σ_FM /S_(NCI) value of different dye combinations. We considered three conditions where two dyes need to be selected from D1 ∼ D9 to form a combination. Conditions include M is 2, M is 3 and one of the dyes is Calcein blue (M =3, F: D1), M is 4 and two of the dyes are Calcein blue and CY 7.5 (M =4, F: D1 & D9). For each condition, we calculated the experimental crosstalk (Fig. 4(b)) and the σ_FM /S_(NCI) (Fig. 4(c)) of all possible dye combinations. Compared to the results of Fig. 4(b-c), we found the experimental crosstalk and the σ_FM /S_(NCI) change in the same trend for all three conditions. Specifically, the lowest σ_FM /S_(NCI) dye combination group has the lowest experimental crosstalk. These results indicate we can speculatively obtain a dye combination with low experimental crosstalk by calculating and comparing σ_FM /S_(NCI) value of different dye combinations.

Apart from the experimental results, the main idea of the dye combination selection method is to increase the relative response differences of different fluorescence dyes in the colorimetry camera. Although we performed the analysis and experiments based on Retina 200DSC, the dye combination selection method is also applicable to other types of color cameras. Besides, since we used the electronic signal in the camera as the standard unit for different fluoresce channels, some optical factors (including camera QE, dye excitation efficiency, optical transmission efficiency, etc.) are not considered in the dye combination selection method. When these optical factors are important for the experiment, they could be factored into the noise model by changing the signal unit from electronics to photons.

3.2 Noise correction

There are two kinds of pixel-dependent noise in sCMOS cameras, FPN and read noise [10]. We first corrected the FPN of the colorimetry camera, and the corrected DSNU and PRNU are over 3 times lower than uncorrected. The detailed results are shown in Supplement 1 and Fig. S1.

Then, we demonstrated the effects of read noise correction based on the images of single-color microtubules labeled with DL650, as shown in Fig. 5. Images were processed by the joint demosaicking and denoising method, or the residual interpolation method [27] for denoising comparison. We found the PSNR of the noise-corrected white channel image and red channel image increased from 16.2 dB and 13.8 dB to 19.2 dB and 19.4 dB, respectively. And the noise-corrected images are smoother than uncorrected images. These results indicate the joint demosaicking and denoising method could improve the imaging quality of camera channel images. Besides, there are more single white points (high noise pixels) in the uncorrected images, especially in the temporal pixel fluctuation map, as shown in Fig. 5(a-b). The temporal pixel fluctuation map describes the variation of each pixel in a group of continuous images. Comparing the temporal pixel fluctuation maps in (a-b), nearly all the high noise pixels were corrected by the joint demosaicking and denoising method. Moreover, we observed the values of two high noise pixels in the 100 consecutive frames, as shown in Fig. 5(c). Clearly, the variation of the noise-corrected pixels is smaller. Since single high noise pixels directly reflect the impact of pixel-dependent read noise, these results indicate pixel-dependent read noise in colorimetry camera can be well corrected by the joint demosaicking and denoising method. In total, all these results demonstrate that both read noise and FPN in colorimetry camera can be effectively corrected.

 

Fig. 5. Noise correction for microtubules imaging. (a) Images processed by residual interpolation method. (b) Images processed by joint demosaicking and denoising method. (c) Pixel value fluctuations of two high noise pixels in 100 consecutive frames. The temporal pixel fluctuation map in (a, b) is calculated as the standard deviation of each pixel in white channel images over 100 frames. Temporal pixel fluctuation map is an obvious grid shape, because W pixels in the white channel image are more obviously affected by noise compared with other pixels calculated from demosaicking algorithm. ROI images in (a, b) are enlarged from white images and temporal pixel fluctuation maps. Pixel 1 and 2 in (c) are the two high noise pixels circled in the ROI images in (a, b). Compared to (a) and (b), the noise-corrected images are smoother and have fewer single white points (high noise pixels). Scale bar in (a, b): 2 µm. Color bar: white image and red image, 4 ∼ 30 e-; temporal pixel fluctuation map, 1 ∼ 10 e-.

Download Full Size | PDF

Apart from these correction results, since the colorimetry camera used in this work is developed from sCMOS technology, pixel noise non-uniformity was considered when performing noise correction. Although we developed the joint sCMOS denoising and demosaicking method based on NCS, other newly developed sCMOS denoising methods, such as ACsN [22] could also be used, which may further improve the image PSNR.

3.3 Crosstalk of low light multi-color fluorescence imaging

Based on previous analysis and experimental results, we tried to decrease the crosstalk of the CCFM method introduced in our previous work [9] by optimizing dye combination (i.e. replacing DL650 with AF568) and performing noise correction (i.e. using joint demosaicking and denoising algorithm). We compared four methods with different dye combinations and image processing methods, as shown in Table 1. For each method, four single-color fixed cos-7 samples labeled with different dyes were imaged. To discuss the crosstalk under different intensities, we captured groups of images with different exposure times. Each group contains 4 single color images with different dyes and similar signal intensity, and these images were used to characterize the maximum crosstalk. To reduce random errors, we repeatedly captured 5 groups of images for each intensity level, and used the mean value of the maximum crosstalk and the signal intensity to draw Fig. 6(a).

 

Fig. 6. The crosstalk of single-color samples. (a) The dependence of maximum crosstalk on fluorescence signal intensity. (b) A single frame (raw image) and a mean image averaged from 100 consecutive frames of mitochondria labeled with AF750. (c) Decoded fluorescence images are calculated based on the raw image in (b) with Method 1 (DL650 group without noise correction) or Method 4 (AF568 group with noise correction) in Table 1. Note that only AF750 channel image in (c) corresponds to the mitochondria image, and the images of other channels are crosstalk caused by noise or other factors. Each data point in (a) was calculated from an average of 5 image groups. Scale bar in (b, c): 5 µm. Color bar in (b, c): 0 ∼ 20 e-. The signal level in (b, c) is about 16 e-.

Download Full Size | PDF

Table 1. Methods for multi-color CCFM crosstalk comparation

View Table

Comparing Method 1 and Method 3 (or Method 2 and Method 4), crosstalk decreases after using AF568 replacing DL650, this indicates AF568 is a better dye candidate for this dye combination as we discussed in Section 3.1.1. Comparing Method 1 and Method 2(or Method 3 and Method 4), we found the joint demosaicking and denoising algorithm could also reduce crosstalk. In general, low crosstalk can be obtained with 1.5 ∼ 2 fold fewer fluorescence photons when only the noise correction method or dye combination optimization method was performed. However, when the noise correction method and dye combination optimization method were performed together, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and the crosstalk drops from > 10% to < 5% for 100 e- signal. That means the method of combining noise correction and dye combination optimization has a significant effect on reducing the crosstalk of CCFM. Note that Method 1 was used in our previous work, and the maximum crosstalk is 4.6% for ∼ 375 e- signal [9].

To further analyze the method of combining noise correction and dye combination optimization, we compared Method 1 and Method 4 by decoding a same group of images. The NCI probability distributions calculated from Method 1 or Method 4 were used to decode 100 consecutive frames of single-color mitochondria images with a signal level of ∼16 e- (Fig. 6(b–c)). We found the crosstalk of Method 1 is > 10% on the AF532 and DL568 channels, and the maximum crosstalk is 17%. While for Method 4, the maximum crosstalk is only 8%. Besides, the PSNR of the AF750 channel image increases from 14.8 dB to 17.9 dB after using Method 4. These results further demonstrate that the method of combining noise correction and dye combination optimization will improve the imaging quality of multi-color CCFM.

3.4 Four-color fixed cos-7 cells imaging

To verify the benefits of crosstalk reduction on CCFM, we performed single shot four-color fluorescence imaging to compare Method 1 with Method 4 in Table 1. Fixed cos-7 cells samples labeled simultaneously with DAPI (nuclei), AF532 (PMP70), DL650 or AF568 (microtubules), and AF750 (mitochondria) were imaged.

Comparing the decoded multi-color fluorescence images in Fig. 7, we found structures in image from Method 4 are much closer to structures in single-color images (Supplement 1, Fig. S3). For example, the signal in DAPI channel is concentrated in single-color image or image from Method 4. But for image from Method 1, scattered DAPI signal could be found anywhere (see Fig. 7(a)).

 

Fig. 7. Four-color CCFM imaging of fixed cos-7 cells labeled simultaneously with DAPI (nuclei), AF532 (PMP70), DL650 or AF568 (microtubules), and AF750 (mitochondria). (a) Before dye optimization and noise correction. (b) After dye optimization and noise correction. Four-color fluorescence images were calculated from the raw CFA images. For each method, a ROI area (in yellow rectangle) containing only PMP70 and microtubules was enlarged for crosstalk analysis. After enhancing the contrast by 3 times, it clearly shows that the color of the fluorescence image in (a) is messier than image in (b). Mic. Crosstalk in (a, b) represents the max crosstalk of microtubules signal. Scale bar: 5 µm. Color bar: 0 ∼ 200 e-. Signal level for each decoded fluorescence images: 20 ∼ 40 e-.

Download Full Size | PDF

ROIs only containing microtubules (as shown in Supplement 1, Fig. S4) were chosen to characterize crosstalk. The maximum crosstalk of microtubules for Method 1 is 18%, while for Method 4 is only 10%. We also chose two sub-regions only containing PMP70 and microtubules for comparison. For Method 4, most of the PMP70 and microtubules signal are classified into AF532 and AF568 channel, respectively. And there are almost no signal in DAPI and AF750 channel. For Method 1, microtubules signal is classified into not only DL650 channel but also AF750 channel. Besides, crosstalk signal in DAPI channel is clear.

These findings confirm our previous conclusion that the method of combing noise correction and dye combination optimization could decrease crosstalk in multi-color CCFM. These results also demonstrate that the crosstalk reduction would help to separate different biological substances more easily, and make the decoded fluorescence images more convincing. Besides, after considering the imaging quality, the imaging speed of CCFM should be at least 2 times faster than traditional sequential excitation methods for four-color fluorescence imaging (See Supplement 1 for details).

4. Conclusion

In summary, we developed a method to improve the color separation ability of CCFM by performing noise correction and optimizing dye combination selection. The method found that dye combinations could be analyzed by σ_FM /S_(NCI) calculation, which was verified by imaging uniformly illuminated fluorescent solutions. By developing a joint demosaicking and denoising algorithm that takes into account the noise characteristics and CFA pattern of the colorimetry camera, the effects of pixel-dependent noise in colorimetry camera can be well corrected, including improving the image PSNR by 3 ∼ 6 dB and suppressing the defects brought by high-noise pixels. We used this method to improve our previous four-color CCFM imaging experiment [9]. When only one strategy (noise correction or dye combination optimization) was performed, low crosstalk can be obtained with 1.5 ∼ 2 fold fewer fluorescence photons. When these two strategies were combined, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and the crosstalk is < 5% when the signal level is ∼ 100 e-. This improvement allows CCFM to maintain good color separation ability with low exposure time, and brings the potential for imaging speed improvement. We believe that this study pushes CCFM for low-light fast multi-color fluorescence imaging applications, and provides a new opportunity for more biological studies such as fast live-cell imaging.