1. Introduction

Fluorescence lifetime imaging (FLIM) has become a powerful tool for biomedical discoveries. The fluorescence lifetime is a robust indicator for better observing living cells [1]. FLIM provides abundant cellular microenvironmental information that is unattainable in fluorescence intensity imaging. It can probe biophysical parameters such as temperature, O₂, viscosity, metabolism, pH, and ion concentrations [2,3]. In combination with Förster resonance energy transfer (FRET) principles, FLIM-FRET has been widely used to assess protein-protein interactions and conformational changes [2–4]. Meanwhile, FLIM is more robust than intensity imaging because it is less susceptible to experimental parameters such as the fluorophore concentration, excitation intensity, and optical path. FLIM can distinguish fluorophores with identical emission spectra. It can also identify tagged fluorochromes on specified cells from background signals or unbounded fluorochromes [2]. It has been widely used for cellular study, assessing drug delivery, cancer diagnosis, and ophthalmological diagnosis [5,6].

The time-correlated single photon counting (TCSPC) technique is the gold standard for FLIM imaging due to its superior timing resolution, photon efficiency, and signal-to-noise ratio performances [7]. However, traditional TCSPC-based FLIM systems are slow, especially for acquiring high-resolution (HR) images. TCSPC requires a low photon-counting rate (typically < 5% of the laser repetition rate) to avoid pile-up effects [8]. There are two different instrumental configurations for acquiring FLIM images: laser scanning microscopy (LSM) and wide-field illumination microscopy (WFIM) [9]. LSM FLIM systems acquire images through scanning the focal point across samples. Obtaining HR images requires a minimized focal point and a smaller scanning step, leading to a long acquisition, photobleaching fluorophores, and causing phototoxicity. On the other side, WFIM systems adopt a parallel light field to excite samples, offering a higher frame rate and causing less damage to samples. However, WFLIM’s spatial resolution is poor, and images are easily contaminated by out-of-focus surrounding light. Besides, there is a trade-off between the spatial and temporal resolutions due to the crosstalk [10]. Hence, it is a great challenge to acquire HR images rapidly.

Meanwhile, since FLIM is an indirect imaging technique, traditional algorithms such as the least-squares fitting method (LSF), maximum likelihood estimation methods (MLE), Bayesian analysis, or Laguerre deconvolution [3,11,12] are applied to analyze the fluorescence lifetime. They are model-based iterative optimization methods (generally slow) and rely on prior knowledge to set proper models and initial conditions. Other fast and model-free analysis algorithms were also developed, including the center-of-mass method (CMM) and phasor method with graphic representation [13,14]. However, the image acquisition and data processing are too slow for HR or 3D images, hampering widespread applications such as cellular dynamics imaging or high-throughput screening.

New sensor and microscopy technologies have been introduced to improve FLIM systems’ performances, including short-deadtime high-throughput single-photon sensors [15] and multifocal scanning systems [16]. This paper proposed an alternative computational strategy using deep learning techniques to rapidly obtain HR FLIM images from raw image data without modifying the system configuration. Our method combines DL-based fluorescence lifetime determination and single image resolution enhancement (DL-SIRE) techniques. DL techniques provide a new approach to delivering fast analysis with different models. They have achieved unprecedented accuracy and fast speed [17 –19]. ‘SIRE’ refers to reconstructing a single HR image directly from its LR counterpart. It is a notoriously ill-posed problem because an infinite number of HR images correspond to a given LR image. DL’s powerful hierarchical representation learning ability allows DL-SIRE to reconstruct state-of-the-art HR images through learning a single image’s intrinsic and high-level statistical features [20,21]. DL-SIRE has been widely used in face recognition, remote sensing, and video surveillance [22,23]. It has become a research trend in improving the spatial resolution of optical and fluorescence microscopes [24–28]. However, DL-SIRE has not yet been used in FLIM systems, and there are two main challenges. First, there is a lack of massive FLIM data for training neural networks. Unlike natural images, FLIM images are three-dimensional (spatial and temporal) data. Acquiring massive HR FLIM images is labor-intensive and time-consuming. Second, FLIM images simultaneously possess complex spatial and temporal features.

To address the above challenges, 1) we proposed a method to generate a large-scale semi-synthetic FLIM dataset for training neural networks. The samples in the FLIM dataset include distinct morphological characteristics of subcellular structures and complex decays with a wide range of fluorescence lifetimes. 2) a hybrid neural network architecture, SRI-FLIMnet, was proposed to map LR FLIM data to corresponding HR images. SRI-FLIMnet serves two purposes: A) to directly map a decay histogram to its corresponding lifetime without iterative optimization processes and B) to nonlinearly transform LR FLIM to HR FLIM images. LR images can be fast obtained through both LSM and WFIM FLIM systems. SRI-FLIMnet was evaluated using semi-synthetic FLIM images and validated with experimental FLIM data. Results confirm SRI-FLIMnet’s superiority in reconstructing HR images. SRI-FLIMnet is expected to fast capture HR FLIM images and find potential applications in high-throughput and real-time FLIM systems.

2. Dataset preparation

Deep learning techniques are data-driven approaches. A large amount of suitable training data is necessary for developing a robust DL algorithm. However, for DL tasks in microscopy, there are limited datasets. Unlike open-source natural image datasets used for DL computer vision applications [29], most previously reported studies only considered specific tissues or cells, consisting of several dozens to hundreds of samples for network training [24–28]. Data argumentation techniques should be applied to increase the training data. In addition, it is challenging to generalize previously reported DL algorithms for general applications. The lack of FLIM datasets severely handicaps the development of DL FLIM techniques. Considering that FLIM data possess many spatial and temporal features, traditional data argumentation techniques cannot achieve desirable performance. One effective method to circumvent this difficulty is to generate model-based synthetic data to train networks. Data synthesis is a mature and widely used method to approximate realistic data and serve network training purposes. It has been successfully applied to many DL tasks, including light detection and ranging (LiDAR), fluorescence lifetime estimation, and super-resolution microscopy [17,18,27]. In this section, a theoretical framework was proposed to obtain massive semi-synthetic FLIM data. The forward TCSPC model was first developed to describe the time-stamped photon detection process. Then semi-synthetic FLIM images were generated by converting immunofluorescence intensity images from the Human Protein Atlas (HPA) dataset to 3D FLIM data using the TCSPC model. Meanwhile, the degrading model between HR and LR images was also developed to obtain LR-HR training pairs. We demonstrate that semi-synthetic data can be used for neural network training and successfully applied to different experimental data.

2.1 TCSPC model

In TCSPC, a train of laser pulses excites the sample periodically. In each cycle, at most one photon is detected by a single-photon detector to avoid the pile-up effect. According to the photon counting theory, such a process is governed by a non-homogeneous Poisson process [30]. For a pixel (i, j) in an M×M×L FLIM image, the rate function ${\lambda _{i,j}}$ can be expressed as:

2.2 Semi-synthetic FLIM images

The HPA dataset provides a comprehensive subcellular map of human proteomes in cells, tissues, and organs [31]. It contains more than 80,000 cellular images with a large morphological diversity obtained by confocal immunofluorescence microscopy systems. The images in HPA dataset cover a wide range of cellular morphologies. We used the HPA image classification dataset [32], including more than 30,000 samples with 27 different cell types. Each sample contains four color channels, and each color channel contains one cellular organelle. The blue, red, yellow, and green channels contain nuclei, microtubules, endoplasmic reticulums, and proteins (such as endosomes, lysosomes, or mitochondria). Figure 1 shows some original image samples in the HPA dataset and generated 3D FLIM data. The 1^st column shows RGB images of three different samples, and their corresponding color-channel images are shown in the 2^nd column. The cellular images contain more statistical features, including more refined details, complex spatial variations, and more sophisticated correlations than open-access natural images (mostly used by computer vision communities for DL training). To generate FLIM data with complex decay features and various cellular morphologies, The samples in HPA datasets are converted to semi-synthetic FLIM data. the pixel values of intensity images are converted to time-stamped photon counts using the forward TCSPC model in Eqs. (1)–(3). The pixel values from 0 to 255 were linearly scaled to 0 to 5100. The photon count range corresponds to the typical intensity range in practical experiments and offers a sufficient SNR for accurate lifetime estimations. Instead of restricting to specified fluorophore lifetimes, each color channel was randomly assigned a single lifetime in the range from 0.5 to 5.5 ns to include more general cases. The semi-synthetic FLIM data were obtained by summing up all the considered color channels. Since different color channels have diverse intensities and spatial overlaps, each pixel naturally shows complex decays with different lifetime components. Meanwhile, they also have a variety of shapes. The total photon count in a pixel of a FLIM image remains the same as the intensity image, while the temporal photon distribution follows an exponential decay with given lifetime parameters. In simulations, M = L = 256, D = 1, and the total recorded time is 10 ns. The IRF is modeled by a Gaussian distribution with FWHM = 167.3 ps. This matches the two-photon FLIM system we used for experiments (see details in Section 5). The added noise can be reasonably neglected because Poisson noise dominates in TCSPC measurements.

 

Fig. 1. Original intensity images in the HPA dataset and the semi-synthetic FLIM data. The 1^st column is the RGB intensity images of three samples. Their corresponding four color-channel images are shown in the 2^nd column; each organelle is labeled by one color. The 3^rd and 4^th columns show the synthetic 3D FLIM data cloud and lifetime images. In 3D point plots (3^rd column), the dark blue to bright yellow point color indicates a high to low photon density.

Download Full Size | PDF

The generated data cloud and the corresponding fluorescence lifetime images calculated by the least-square fitting method are shown in the 3^rd and 4^th columns of Fig. 1, respectively. Each point corresponds to a detected photon’s spatial and temporal positions in the data cloud. FLIM images have lower contrast than RGB intensity images in revealing cell structures. Many delicate structures are blurred or even disappear, and this is because fluorescence lifetime is independent of the intensity variation. To ensure diversity, FLIM data with only one, two, or three decay components are also generated. The whole dataset contains 30,000 FLIM samples for training purposes, providing sufficient data for network training.

2.3 Degrading model

An accurate degrading model for developing LR-HR image pairs correctly is crucial for DL-SIRE’s performance [21]. For general DL-SIRE frameworks, the degrading model is:

3. Network design and training

3.1 Network architecture

It has been a great challenge to process the spatial and temporal features of FLIM data simultaneously. One intuitionistic approach is to design a 3D neural network architecture to map 3D FLIM data to HR images. However, the spatial and temporal information could interfere, leading to difficulty in network training. It would also cause an enormous computation burden and a prolonged inference speed. Our proposed SRI-FLIMnet adopts a temporal and spatial decoupling approach to address this problem. The SRI-FLIMnet, as shown in Fig. 2, uses a hybrid network architecture composed of two sub-nets: τ-Net and SRI-Net. The former aims to estimate fluorescence lifetimes, whereas the latter is to produce spatial-resolution improved lifetime images so that the crosstalk between spatial and temporal dimensions can be eliminated. τ-Net uses a one-dimensional convolutional (1DCNN) neural network to calculate pixel-wise fluorescence lifetimes [19]. 1DCNN is efficient and hardware-friendly, suitable for fast analysis, and can accurately analyze single- and multi-exponential decay models. The design and characterization of 1DCNN have been reported elsewhere [18, 19]. τ-Net contains five 1D convolutional blocks. Each block has a convolution layer, a batch normalization layer, and a nonlinear activation layer (Rectified Linear Unit, ReLU). The hyperparameters of all layers in τ-Net are annotated in Fig. 2. We adopt the residual dense network (RDN) with hybrid dilated convolution as the backbone for SRI-Net. RDN can fully exploit local and global hierarchical features from LR images [33]. It consists of four parts, the shallow feature extraction (SFE) layers, residual dense blocks (RDB), global feature fusion (GFF) layers, and the sub-pixel convolution up-sampling (SCUS) block [23]. SFE layers extract high-level features of input LR images for later network blocks. RDBs are for learning and fusing local features with different levels. In an RDB, 8 2D convolutional layers are densely connected. The input feature of the first layer is 64, and the growth rate of the feature number in the following layers is 64, too. A concatenation layer is put at the end of the RDB, followed by a residual connection across the whole block. To achieve a better performance, RDN requires a deeper architecture with many RDBs to obtain a large receptive field and high nonlinearity. The receptive field is a critical factor in the network’s performance because it decides how much contextual information is used to reconstruct high-frequency components. However, a deeper architecture consumes more computation resources and is, therefore, slower in training and inference. To overcome this problem, we further improved the network by introducing the hybrid dilated convolution in each RDBs [34]. By inserting zeros in the convolution kernel, dilated convolutions can exponentially expand the receptive field without extra parameters. In each RDB, the dilation rate for 8 convolutional layers is [1, 2, 5, 9, 1, 2, 5, 9]. We particularly used mixed saw-toothed dilation rates for different layers to expand the convolution kernels to different scales while avoiding the gridding effect and information loss brought by dilated convolutions [35]. The GFF block is for global feature fusion and global residual learning, and the final SCUS block up-samples images to the size of corresponding HR images. Figure 2 only shows the network architecture for a k = 4. Note that for k = 2 and 3, there are only one 2D convolutional layer and one Pixel-Shuffle layer in the Up-Scaling part for obtaining the desirable data size of output images. The corresponding feature number of the convolutional layer are 64×k², and the kernel size and stride remain 3 and 1, respectively.

 

Fig. 2. Network architecture of the proposed SRI-FLIMnet with a scaling ratio k = 4. SRI-FLIMnet contains two subnets, τ-Net, and SRI-Net, responsible for lifetime evaluation and nonlinear transformation for improving spatial resolution, respectively. The parameters of convolutional layers are labeled in the parentheses as (feature number, kernel size, stride). SRI-Net contains 8 residual dense blocks (RDB) with hybrid dilated convolution. In each RDB, 8 2D convolutional layers with a kernel size of 3 and a stride of 1 are densely connected. Both the input feature number and growth rate are 64. In the Up-Scaling part of SRI-Net, the parameters are (1024, 3, 1) for both convolutional layers. The input data of SRI-FLIMnet is 64×64×256, and the final output data size is 256×256.

Download Full Size | PDF

3.2 Loss function

For τ-Net, the Mean-Square Error (MSE) loss or L₂ norm loss is used for network training:

3.3 Network training

We implemented SRI-FLIMnet using the Pytorch library in Python 3.8 and performed network training and inference on a workstation containing an Intel Core i9-10900X CPU and an Nvidia Quadro RTX5000 GPU card. The two subnets were trained independently. For τ-Net, the training dataset contains 50,000 decay histograms generated from Eqs. (1)–(3). For each histogram, the total photon counts vary from 100 to 10,000, and the number of lifetime components ranges from 1 to 4, with a lifetime ranging from 0.5 to 5.5 ns. All random variables follow uniform distributions. The optimizer is Adam with a learning rate of 1e-4, and the batch size is 256. An early stop with 20 patient epochs was added during the training process to prevent over-fitting. The total training time is about 5 mins for τ-Net. For SRI-net, the semi-synthetic dataset mentioned above was used for training. The sizes of LR FLIM data are 128×128×256, 85×85×256, and 64×64×256 for k = 2, 3, and 4, respectively, which are obtained using Eq. (5). The optimizer is Adam. The initial learning rate is 1e-4, and it dropped 0.9 times every 20 epochs. The batch sizes for the k = 2, 3, and 4 are 6, 12, and 24, respectively, to fully use GPU memory. The total training epoch for different scaling ratios is 200, and the training time is about 400 h, 160 h, and 50 h for k = 2, 3, and 4. Once the networks were well trained, the inference time for different scaling ratios typically took several seconds to map the 3D LR FLIM data to 256×256 HR images.

4. Characterization of the neural network

After the training procedure, SRI-FLIMnet was evaluated with a new testing dataset. The performance of SRI-FLIMnet on lifetime determination was first quantitatively assessed. Figure 3 (a) shows three normalized decay histograms used for testing τ-Net. The samples denoted as Decays 1–3 have different SNR, lifetimes, and decay components. As mentioned above, the decay components may vary in different pixels in a FLIM image, and we do not know the exact number of decay components. Consequently, applying multi-exponential decay model analysis can cause model-mismatch problems and wrong interpretation. As suggested in Ref. [37], the amplitude-weighted average lifetime τ_A was used as a lifetime indicator, suitable for FLIM-FRET measurements. τ_A is defined as ${\tau _A} = \mathop \sum \limits_{i = 1}^D {\alpha _i}{\tau _i}$, where D is the number of lifetime components. τ-Net directly predicts τ_A for input histograms. The accuracy of lifetime determination in terms of SNR is shown in Fig. 3(b). The model-free algorithm, CMM and model-based LSF were also compared to SRI-FLIMnet. For LSF, the mono-exponential model is used for evaluating τ_A of multi-exponential decays. The SNR of a decay histogram equals to $\sqrt N $ with N the total photon count. The SNR(dB) is calculated as $SNR({dB} ) = 20log\sqrt N $. The mean absolute error (MAE) determines the analysis accuracy. Figure 3(b) indicates that τ-Net shows the best accuracy, much better than traditional CMM and LSF. The median MAE of τ-Net is 0.169 for SNR 20 ∼ 30 dB (100 ∼ 1000 photon counts). As a comparison, the median MAE of LSF is 0.46 for SNR 38 ∼ 40 dB (6400 ∼ 10, 000 photon counts). The accuracy of τ-Net for low SNR is even better than that of LSF for high SNR. The lifetime determination performance over the whole dynamic lifetime range was investigated in Fig. 3 (c). Generally, given a fixed SNR, decays with larger lifetimes are nosier because photons are distributed in more time bins, resulting in fewer photons in a single time bin. So, the accuracy is poor for decays with larger lifetimes. Nevertheless, τ-Net keeps a low MAE and small changes compared with the other two traditional methods over the whole lifetime ranges. The results in Fig. 3 indicate τ-Net is robust and accurate to calculate lifetimes. In addition, the calculation speed for τ-Net is more than 300-fold faster than LSF.

 

Fig. 3. Fluorescence lifetime determination performance of SRI-FLIMnet. (a) Decay samples used for testing τ-Net. The lifetime components for Decays 1–3 are 2, 3, and 4; the lifetimes for Decays 1–3 are [0.6, 1.5], [0.8, 2, 2.6], and [1.2, 1.8, 3.1, 4.5]; the total photon counts are 400, 2000, and 5000. The red dashed line represents the IRF. (b) Lifetime determination performance under different SNR conditions. In each group, the lifetime ranges from 0.5 to 5.5 ns and the lifetime components randomly vary from 1 to 4. The SNR ranges from 20 ∼ 40dB, corresponding to 100 ∼ 10,000 photon counts. (c) Lifetime determination performance for different average lifetimes. In each group, the lifetime components randomly vary from 1 to 4, and SNR ranges from 20 ∼ 40dB. In (b) and (c), 2000 samples were taken into consideration in each group.

Download Full Size | PDF

The performance of spatial resolution improvement for SRI-FLIMnet was further investigated. The testing FLIM dataset contains 1,000 samples. As an example, we evaluated reconstructed HR images with k = 4. In this case, suppose that the dwell time in each scanning step is fixed, the image acquisition speed improves 16 times for LSM-FLIM systems. At the same time, LR images undergo severe degradation as much spatial and temporal information is lost. Figure 4 shows the input LR, interpolated HR, SRI-FLIMnet-reconstructed HR, and ground truth (GT) FLIM images of four samples chosen from the testing dataset to illustrate the results better. The four samples labeled S₁ to S₄ include different morphological features such as a sizeable cellular population and complex cell patterns. They also have distinct lifetime distributions. As shown in Fig. 4, the 64×64 LR images predicted by τ-Net are shown in the 1^st row, with a poor spatial resolution and a strong mosaic pattern. Compared with the GT images in the 4^th row, the boundaries are unclear, and many delicate structures are missing. The lifetime distributions also have significant deviations. The 2^nd row shows referenced interpolated 256×256 HR images using the bicubic interpolating method, indicating that the direct interpolation fails to improve the spatial resolution and introduces more artifacts. For example, the boundaries in all samples are severely distorted. In contrast, SRI-FLIMnet-reconstructed HR images in the 3^rd row have sharper and smoother edges. They also recover more refined textures compared to LR images.

 

Fig. 4. Spatial resolution improved FL images of four samples with distinct morphology and lifetime distributions. The samples are denoted by S₁ – S₄. The low-resolution (LR), bicubic interpolation, reconstructed HR by SRI-FLIMnet, and ground-truth HR (GT) FL images are shown in the 1^st row to 4^th rows, respectively. The white dashed boxes in each subfigure depict the regions of interest (ROIs) for further analysis.

Download Full Size | PDF

Figure 5 shows the zoomed-in regions of interest (ROIs) marked by white dashed boxes in Fig. 4 to illustrate the performance of SRI-FLIMnet better. The results indicate that SRI-FLIMnet can generate shapes, boundaries, and general lifetime distributions closer to GT images from minimal information in LR images. SRI-FLIMnet-reconstructed images significantly outperform those using traditional interpolation methods. However, SRI-FLIMnet fails to reconstruct pixel-level refined structures such as gaps between cells in S₁ and irregular boundaries in S₃. In addition, it cannot recover random lifetime variations in cells. For example, the spotty lifetime patterns in S₄ are blurred and smoothed. Therefore, SRI-FLIMnet has a solid ability to generate visually clear shapes and boundaries, whereas it cannot precisely reconstruct pixel-wise details and variations.

 

Fig. 5. Zoomed ROIs corresponding to the white dashed boxes in Fig. 3. The LR images lose most of the high- and low-frequency information and show strong mosaic patterns. The reconstructed HR images are more visually pleasing and close to the GT images than the interpolating method.

Download Full Size | PDF

The precision of SRI-FLIMnet-reconstructed HR images was also evaluated. Figure 6 shows absolute error maps (Fig. 6(a) – (d)) and lifetime distribution histograms (Fig. 6(e) – (h)) of S₁ – S₄ shown in Fig. 4. The absolute error maps are the differences between reconstructed images and GT images. Since different samples have various lifetime ranges, absolute error maps are normalized to 0 ∼ 1 for a better comparison. Interestingly, the largest errors always occur at outer boundaries for all four samples, where the spatial differentiation reaches its maximum value. The spatial differentiation has a significant effect on reconstruction precision. An area with sizeable spatial differentiation tends to have an apparent absolute error and vice versa. Figures 6 (e)- (h) show lifetime distribution histograms of S₁ – S₄, respectively. The two histograms of SRI-FLIMnet-reconstructed and GT images have primarily overlapped distributions for all four samples, revealing that SRI-FLIMnet can correctly restore the lifetime distribution in a wide dynamic range. However, there is a slight difference in lifetime distribution between reconstructed and GT images. The reconstructed images have more pixels with lifetime centering in the peak values. In addition, the lifetime distributions of REC images are relatively narrower compared to corresponding GT images.

 

Fig. 6. Normalized absolute error maps (a) - (d) and lifetime distributions of the GT and SRI-FLIMnet reconstructed HR images (e) - (h) for S₁ – S₄. The normalized absolute error maps have an arbitrary unit.

Download Full Size | PDF

We further evaluated SRI-FLIMnet on the whole testing dataset. Table 1 summarizes the results of k = 2, 3, and 4. Two image quality metrics, SSIM and image MAE (IMMAE), were used for quantitative evaluation. The SSIM quantifies how SRI-FLIMnet can reconstruct close LR images to corresponding GT images. In contrast, the IMMAE measures the overall deviation of the lifetime distribution of reconstructed images from that of GT images. The IMMAE for an image is calculated as$|{{I^{Rec}} - {I^{GT}}} |/{M^2}$, where ${\textrm{I}^{\textrm{Rec}}}$and ${\textrm{I}^{\textrm{GT}}}$denote the reconstructed and GT images, respectively. M is the length of the spatial dimension. The dynamic lifetime range for calculating SSIM is 0 ∼ 5.5 ns. In Table 1, the results are the average values of 10,000 samples. Both metrics of SRI-FLIMnet significantly outperform those of bicubic interpolation algorithms for different scaling ratios. In comparison, The SSIM of SRI-FLIMnet is approximately twice that of the bicubic interpolation method, whereas the IMMAE of SRI-FLIMnet is also much better. The results reveal that SRI-FLIMnet has superior performance in reconstructing morphologies and lifetime distributions. It is worth noting that the testing samples we considered have complex structures and an extensive dynamic lifetime range. Both metrics for SRI-FLIMnet can be further improved if we only consider specified cell types and narrow the lifetime range.

Table 1. Quantitative Evaluation of Different Scaling Ratios on Testing Dataset

View Table

5. Experimental validations

5.1 Sample preparation

We validated the trained SRI-FLIMnet using FLIM data of bacteria-infected mouse raw macrophage cells cultured in DMEM (Dulbecco’s modified Eagle medium) containing 10% FCS (fetal calf serum), maintained with 5% CO₂ in a humidified chamber at 37°C. Cells were seeded onto glass coverslips within 24-well cell culture plates. Lactobacillus and Shigella Sonnei bacteria were used in FLIM experiments. They were engineered to express a green fluorescent protein, which was harvested at an early exponential phase. Cells cultures were infected at a multiplicity of infection (MOI) of 100. About one hour of infection, the wells were washed 3 times with phosphate buffer saline (PBS) to remove extracellular bacteria. Then, cells were further incubated with DMEM containing 50 μs/mL gentamicin. The DMEM with gentamicin was removed at the chosen time, and cells were washed 3 times with antibiotic-free PBS. Cells were fixed with 3.7% paraformaldehyde for 15 min and permeabilized with 0.1% Triton X-100 for 5 min after being washed 3 times with PBS. Cells were stained with phalloidin Alexa Flour 546 after being washed 3 times with PBS and then were covered by coverslips with a ProLong antifade solution for microscopic examination.

5.2 Experimental data acquisition and analysis

A two-photon FLIM system was used to obtain FLIM data. The system contains a confocal microscope (LSM 510, Carl Zeiss), a femtosecond Ti: sapphire laser (Chameleon, Coherent) with an 850 nm wavelength as the excitation source, and a TCSPC acquisition system (SPC-830, Becker & Hickl GmbH) for time-tagging detected photons. The laser repetition rate is 80MHz, and the laser pulse duration is less than 200 fs. The fluorescence signals were collected using a 63× water-immersion objective lens (N.A. = 1.0) after passing a 500-550 nm bandpass filter. The TCSPC resolution is 0.039 ns, and each measured histogram contains 256 time-bins. FLIM data (with a dimension of 256×256×256) were taken as ground truth HR images. LR FLIM data is obtained through pixel binning as described in Eq. (5). We used a k = 4 as an example. Figure 7 shows 4 different samples denoted by S₁ to S₄. The peak intensities of the samples range from 4,000 to 16,000 photon counts, providing a sufficient SNR for determining lifetimes. The bacteria have a smaller lifetime of around 2 ns, contrasting to macrophage cells. FLIM images can better identify cells infected by bacteria and localize bacteria. The 1^st column shows LR lifetime images of the samples. The 2^nd and 3^rd columns are reconstructed HR images by bicubic interpolation and SRI-FLIMnet, respectively. The last column is GT images for reference. SRI-FLIMnet can generate clear boundaries, whereas bicubic interpolated HR images show distinct jagged artifacts. This can be observed in all samples. For S₂, small rod-shaped bacteria structures were well recovered in reconstructed HR images from LR images. The boundaries between cells are difficult to distinguish in the LR and interpolated HR images of S₄. Reconstructed HR images, however, show clearer boundaries close to GT images. SRI-FLIMnet also has good performance in reconstructing correct lifetime distributions. The lifetime distributions of LR images shift towards a smaller lifetime due to the pixel binning of decay histograms among neighboring pixels. This problem is resolved in reconstructed HR images, which generate similar lifetimes with GT images. More apparently, the outer boundaries with a higher lifetime are also recovered. These results demonstrate that SRI-FLIMnet can reconstruct more useful HR images from their LR counterparts without prior knowledge.

 

Fig. 7. Spatial resolution improved images of experimental FLIM data of bacteria-infected mouse raw macrophage cells. Four different samples denoted by S₁ to S₄ are shown in the 1^st to 4^th columns. The 1^st row shows the normalized intensity images of the four samples. Their FL images are shown in the 2^nd row. The reconstructed HR images by SRI-FLIMnet are shown in the 3^rd row. The last row is the GT HR images for reference.

Download Full Size | PDF

Figure 8 shows the normalized absolute error maps and lifetime distribution histograms in Fig. 7. Like Fig. 6, large discrepancy only occurs at boundaries. The lifetime distributions are also close to GT images. Figures 7 and 8 indicate that SRI-FLIMnet trained by semi-synthetic datasets can be well applied to new experimental datasets. The semi-synthetic datasets and the proposed SRI-FLIMnet are expected to find more DL-based FLIM applications. However, it should be noted that SRI-FLIMnet is limited to recovering very fine structures (high-frequency contents) and producing smooth images. This can be seen from the reconstructed images of S₁ to S₄ in the 3^rd column, in which irregular and rough outer edges of cells are over-smoothed.

 

Fig. 8. Normalized absolute error maps (a) - (d) and lifetime distributions of the GT and SRI-FLIMnet reconstructed HR images (e) - (h) for experimental data S₁ – S₄. The normalized absolute error maps have an arbitrary unit.

Download Full Size | PDF

6. Discussion and conclusion

We first developed a theoretical method to generate massive semi-synthetic FLIM data in this work. It is challenging to use traditional FLIM experiments to image every cellular morphology and various fluorophores with an extensive range of lifetimes and complex decays. Therefore, there are no available large-scale FLIM datasets in scientific communities. Our work offers a feasible way to fill the gap and develop FLIM-related deep learning techniques. A degrading model was also proposed to obtain LR-HR image pairs with different scaling ratios, essential for spatial resolution enhancement tasks. The semi-synthetic FLIM dataset can be further optimized according to specific tasks by considering cell types, fluorophores, SNR, and other noise sources to perform better. It is expected that our method will significantly boost deep learning techniques for FLIM applications.

In addition to the semi-synthetic FLIM data preparation, a high-efficient neural network architecture, SRI-FLIMnet, was designed to map LR FLIM data to HR images. Achieving HR for FLIM is more challenging and complicated. The lifetime should be evaluated from measured decay histograms. Meanwhile, LR FLIM data has significant information loss in spatial and temporal dimensions. To address these issues, SRI-FLIMnet adopts a combined strategy to estimate lifetimes and improve spatial resolution for LR images sequentially. Two subnets, τ-Net and SR-Net are responsible for the two steps, respectively. This configuration is flexible to tailor the network for different situations. For example, it has been demonstrated that a narrower input lifetime range will provide higher estimation accuracy in τ-Net [18]. If samples’ lifetime range is known in advance, τ-Net can be fast re-trained within a minute to get better performance. Both semi-synthetic and experimental FLIM datasets were applied to show that SRI-FLIMnet can show more apparent boundaries and closer lifetime distributions to GT images. However, it should be noted that SRI-FLIMnet cannot precisely reconstruct images at the pixel level. Theoretically, mapping LR to HR images is an ill-posed problem, as LR images do not contain detailed information. Deep learning approaches can learn statistical features but cannot fully recover all information. Even when we use generative models like generative adversarial networks (GANs), the network would only generate plausible and artificial details, which may misrepresent original images.

In summary, we propose a deep learning-based spatial resolution improved FLIM algorithm. The proposed SRI-FLIMnet trained by a semi-synthetic dataset shows superior performance in reconstructing HR images from LR 3D FLIM data. The performance is also verified by experimental data for bacterial infected mouse raw macrophage cells. Our approach offers a practical computational method to fast obtain HR images without conducting extra experiments and modifying the system setting.