1. Introduction

Typical FD-FLIM system implementations employ heterodyne or homodyne detection using a lock-in amplifier (or equivalent electronics) and require that the detector’s gain be modulatable and in some way synchronized to the modulation of the light source [1–6]. This approach typically provides the fastest image acquisition rate when used with a camera based imaging system. However, camera based FD-FLIM approaches are difficult to use in applications where the physical dimensions of the detector are prohibitive, as in the case of endoscopic imaging. In such applications, a point scanning method is typically adopted where each pixel is acquired serially with a point detector rather than in parallel with a camera. Point scanning heterodyne or homodyne detection was economical when electronics with 100MHz bandwidth were cost prohibitive, as it allows for detection of higher frequency modulation and phase shifts at lower sampling frequencies. Nowadays, however, the cost of higher bandwidth electronics has decreased, making the increased complexity associated with point scanning heterodyne and homodyne methods less justified. As higher bandwidth electronics became more accessible, other excitation and detection schemes emerged, in which simultaneous optical or electrical measurement of the light source is required as a reference or trigger [7–11]. These approaches, however, increase FD-FLIM implementation cost, complexity, and physical size by requiring an extra analog input channel and maintenance of a scattering solution, or alignment of a reflector and additional optics.

Some recent point scanning FD-FLIM implementations do not require heterodyne or homodyne detection. For instance, the authors of [7] implemented an FD-FLIM system with an internally modulated light emitting diode (LED) for excitation. They utilized a sampling rate of 2GS/s, modulation frequencies between 10-60MHz, and averaged 128 acquisitions per measurement to improve signal quality. This strategy, however, would not scale well to real-time FD-FLIM imaging applications due to the lack of real-time processing and the need of heavy signal averaging. The authors of [12] used a field programmable gate array (FPGA) and refer to their system as digital frequency domain FLIM. Their approach resembles time correlated single photon counting (TCSPC). A given measurement consisted of building a histogram of discrete photon arrival events using a cross correlation approach implemented in an FPGA using detectors capable of single photon counting. They then applied phasor analysis to the histogram to obtain phase and magnitude data for various frequencies. The work described in [13] also implemented a photon counting approach, but acquired sinusoidal waveforms instead of histograms from the fluorescence emission. The system in [14] utilizes analog mixing electronics along with a 2-photon laser and a fixed gain PMT to acquire FD-FLIM images. Although the FD-FLIM portions of this system are relatively low cost, the system is not capable of using multiple excitation wavelengths simultaneously. The system is also not able to acquire data suitable for fitting multiple exponential components. While our method lacks the sensitivity of the single photon counting approaches, it is simpler to implement and capable of imaging at 12.5kHz pixel rates using multiple modulation frequencies simultaneously. Also, while the single photon counting approaches do not require analog to digital converters (ADCs), they require more expensive photodetectors and carefully limiting the excitation power to avoid pile-up artifacts. Our approach, on the other hand, is suited to applications where the sample of interest can tolerate relatively more excitation power in order to increase imaging speed. The system described in [10,11] provides excitation at two wavelengths simultaneously using an interferometer and spinning polygon mirror. This method, however, requires measuring a reference signal from each excitation wavelength and additional analog mixing electronics. Our method does not require complex optical systems or additional reference signals to enable dual-wavelength excitation.

In this work we report a novel FD-FLIM instrumentation design that is capable of simultaneous excitation at multiple wavelengths (dual-excitation reported here), simultaneous fluorescence detection at multiple emission spectral bands, time-resolved fluorescence measurement using low-cost, fixed-gain, 100MHz bandwidth avalanche photodiodes (APD), and real-time processing. The implementation of this FD-FLIM instrumentation design is summarized as follows. The sample is excited with two CW diode lasers that are modulated with different digital pulse trains generated with an FPGA. Fluorescence signals from four emission spectral bands are simultaneously detected by APDs, digitized at 250MS/s, and processed at 250MS/s. The laser digital modulation signals are generated with logic clocked with the first ADC sample clock. Using the ADC clock signal for laser control synchronized the laser digital modulation with digitization of the fluorescence emission. This synchronization reduced temporal jitter between successive spatial pixels to much less than the sampling frequency, which simplified instrumentation, system calibration, and data processing.

2. Background

FD-FLIM theory is well documented and understood [15]. A fluorescent sample is first excited by a light source(s) modulated at specific frequency(s). The sample then emits fluorescence at the same frequency(s), but with a phase delay and reduced amplitude (modulation) relative to the excitation. The phase and modulation fluorescence lifetimes, $\tau _{\phi }$ and $\tau _{m}$, respectively, are calculated from the phase delay and modulation at each excitation frequency as shown below in Eq. (1)–3. The $\tau _{\phi }$ and $\tau _{m}$ terms are also referred to as apparent lifetimes in this work. Multi-exponential component lifetimes can be obtained from the phase delay and modulation using non-linear least squares analysis [15]. The notation $(\lambda _{\text {em}},\lambda _{\text {ex}},y,x,f^\text {ex})$ denotes emission band $\lambda _{\text {em}}$, excitation wavelength $\lambda _{\text {ex}}$, image spatial coordinates $y,x$ (assuming a 2D image), and modulation frequency $f^\text {ex}$(Hz) at the given $\lambda _{\text {ex}}$. The $I_{em (\lambda _{\text {em}},\lambda _{\text {ex}},y,x,f^\text {ex})}$, $I_{ex (\lambda _{\text {ex}},y,x,f^\text {ex})}$, $m_{(\lambda _{\text {em}},\lambda _{\text {ex}},y,x,f^\text {ex})}$, and $\phi _{(\lambda _{\text {em}},\lambda _{\text {ex}},y,x,f^\text {ex})}$ terms denote the fluorescence emission intensity, excitation intensity, modulation, and phase delay of the emission relative to the excitation, respectively.

FD-FLIM system instrumentation limitations usually prohibit direct measurement of absolute values of $\phi$ and $m$ required to calculate $\tau _{\phi }$ and $\tau _{m}$. Typically, a reference fluorophore with a known lifetime is used to calibrate the FD-FLIM system (for example, [12]). By imaging a reference fluorophore with a known lifetime and decay kinetics (preferably mono-exponential for simplicity), correction factors (i.e., the instrument frequency response) can be computed to obtain absolute values of $\phi$ and $m$. These calibration procedures are specific to the particular implementation of the FD-FLIM system. The calibration procedures for our system are discussed in more detail in section 3.2.

3. Materials and methods

3.1 Instrumentation

Figure 1 shows the system schematic. Figure 2 shows a picture of the part of the system referred to as the engine in this work. A detailed table of components is located in the Supplement 1. The system is controlled via a desktop computer (also referred to as the host computer) and a Xilinx Zedboard. The Zedboard is an evaluation board that contains a Xilinx ZYNQ-7000 series system on a chip (SoC). The ZYNQ-7000 SoC contains a processor space (PS) part and a programmable logic (PL) part on the same chip. The PL is also referred to as the FPGA in this work. The Zedboard PS contains two ARM CPU cores. At power-up, the Zedboard PS program and FPGA bitfile are loaded automatically from an SD card. After the power-up initialization, the FPGA configures the FMC104 digitizer’s clocking components to generate a 250MHz clock signal and to distribute it to all 4 ADCs. The ADC_0 clock signal is used as the clock signal for the laser digital modulation signal logic in the FPGA. The MEMS Digital Driver is also initialized during this time by custom FPGA logic. Once Zedboard initialization is complete, the computer sends a command via Ethernet to the Zedboard PS to begin an image acquisition. The Zedboard PS relays the start signal to the FPGA to begin acquiring data. Once the FPGA receives the start signal, the FPGA simultaneously (a) generates SPI (serial programming interface) control signals to drive the MEMS (micro-electromechanical system) mirror to raster scan the sample, (b) generates two different digital modulation signals and outputs them to the two CW diode lasers, and (c) begins acquiring and processing the digitized fluorescence emission signal. The 375nm and 445nm CW diode laser digital modulation signals are outputted from the JC1_P and JC2_P pins on the Zedboard, respectively. Each laser uses a SmartDock fiber coupler (Toptica Photonics) to couple its output into a single mode fiber (SMF). The 375nm laser’s output from its SMF is collimated by L1, and reflected onto the common excitation and emission path by a longpass dichroic mirror DM1. The 445nm laser’s output from its SMF is collimated by L2, and reflected by a notch DM2 onto the common excitation and emission path. L3 is used as an objective lens when raster scanning the sample. The fluorescence emission from the sample is collected by L3, transmitted through DM1, DM2, and LPF1, coupled into a 200µm core multimode fiber (MMF) by L4, collimated by L5, and split into four emission channels by DM3-5. The bandpass filters BPF1-4 and longpass filter LPF2 are used to further filter the fluorescence emission for each band. Each filtered emission band is then coupled into a 200µm core diameter MMF with L5. The output of each MMF is secured to an avalanche photodiode (APD) with a custom adapter plate that holds the optical fiber in close proximity to the active area of the APD. The analog output of each APD is connected in series to an amplifier, highpass filter, and lowpass filter. The output of each lowpass filter is connected to an analog input channel on the FMC104 digitizer.

 

Fig. 1. Schematic of the FD-FLIM system. Individual components are listed in the Supplement 1. The FD-FLIM approach described in [16] was expanded by adding a 445nm diode laser and a fourth APD to allow for 2 excitation wavelengths and 4 simultaneous emission channels. The FPGAs and digitizer of the previous system [16] were replaced with the Zedboard and FMC104 to simplify the design and reduce hardware and software costs.

Download Full Size | PDF

 

Fig. 2. Picture of the FD-FLIM engine from Fig. 1 on a standard optics breadboard table with 1-inch spaced holes. 6-inch green scalebars are depicted using the Zedboard as a size reference. The FD-FLIM engine consists of the Zedboard, FMC104, MEMS digital driver, custom mounting hardware, and other accessories (power supplies, etc.). The FD-FLIM engine is shown in the bottom right inset picture in normal operating conditions with a protective top plate and 80mm cooling fan.

Download Full Size | PDF

The analog output from each of the 4 APDs is digitized with 14-bit resolution at 250MS/s by the FMC104. Custom FPGA code was written in VHDL for reading the digitized data from the FMC104 to the Zedboard PL. An overview of the digitizer control logic is located in the Supplement 1.

The digitized data is processed first on the Zedboard PL and then on the computer. The DFT is computed in the Zedboard PL as described later in section 3.3. The DFT output data is transferred to the DDR3 RAM attached to the Zedboard PS via DMA (two pairs of Xilinx AXI4-Stream Data FIFO 2.0 and AXI Direct Memory Access 7.1 cores). Once the image is finished scanning, the DFT output data is transferred from the Zedboard PS DDR3 RAM to the computer via Ethernet using a custom bare metal c-program running on one processor core of the Zedboard PS. The second Zedboard PS core was not used. Then, the system calibration and apparent lifetime calculations ($\tau _\phi$ and $\tau _m$) are performed on the computer as explained in section 3.2. The computer operating system is Linux Mint 20.3. All FPGA codes were developed in Xilinx Vivado 2019.1, and custom FPGA codes were written in VHDL. The Zedboard PS code was written using the lwIP (lightweight industrial protocol) echo server example in Vivado SDK 2019.1 as a starting point. All custom host computer codes responsible for image acquisition and processing were written in Python3. The Python sockets module was used on the computer to interface with the Zedboard PS over Ethernet. All graphical user interface (GUI) codes were written in Python3 using PyQt5 and PyQtGraph.

The digital modulation signals for both the 375nm and 445nm lasers are generated concurrently with data acquisition in the Zedboard PL. Synchronization of the laser control logic and the ADC sampling clocks is required for operation of the FD-FLIM system as described in [16]. However, unlike our previous work, both the digitization and laser control logic are located on the same FPGA. Synchronization was accomplished by using the 250MHz clock from the first ADC (AI0, also referred to as ADC_0) as the clock for the laser control logic as shown in the Supplement 1. Each laser’s digital modulation signal is generated in the Zedboard PL using two logically AND-ed counters as described in our previous work [16]. The two frequencies for each laser’s digital modulation signal will be written as $f^\text {ex}_{\approx DC}$ for the DC approximation frequency and $f^\text {ex}_\text {LT}$ for the fundamental lifetime frequency. The laser digital modulation frequencies for $\lambda _\text {ex}=375\text {nm}$ were $f^{375}_{\approx DC}=1.82\text {MHz}$ and $f^{375}_\text {LT}=27.78\text {MHz}$. The frequencies for $\lambda _\text {ex}=445\text {nm}$ were $f^{445}_{\approx DC}=2.02\text {MHz}$ and $f^{445}_\text {LT}=31.25\text {MHz}$. See the Supplement 1 for more details on the frequency selection of the digital modulation signals.

3.2 Data processing - theory

FD-FLIM data is calibrated for system effects before applying Eq. (1)–3 to obtain the phase and modulation lifetimes. System effects include the presence of background noise, unequal power at each modulation frequency, and arbitrary phase shifts at each modulation frequency due to delays in the electronics and optical path lengths. The emission from each excitation source is processed independently. Processing methods for each excitation wavelength are identical to those described in our previous work [16]. Data for system calibration is obtained by imaging reference fluorophores with known lifetimes. POPOP was used for lifetime calibration of $\lambda _{\text {ex}}=375\text {nm}$: $\lambda _\text {em} = 405 / 35\;\text {nm}$, $\lambda _\text {em} = 487 / 37\;\text {nm}$, and $\lambda _\text {em} = 553 / 93\;\text {nm}$ data. A mono-exponential lifetime of 1.200ns was assumed for POPOP. Fluorescein was used for lifetime calibration of $\lambda _{\text {ex}}=445\text {nm}$: $\lambda _\text {em} = 484 / 37\;\text {nm}$ and $\lambda _\text {em} = 553 / 93\;\text {nm}$ data. A mono-exponential lifetime of 4.000ns was assumed for fluorescein. Rose bengal was used for lifetime calibration of both excitation wavelengths in the $\lambda _\text {em} = 646 / 69\;\text {nm}$ emission channel. A mono-exponential lifetime of 0.076ns was assumed for rose bengal. The three fluorescence references (POPOP, fluorescein, rose bengal) are imaged at the beginning of each imaging session, and the reference data is used for system intensity and lifetime calibrations of all FD-FLIM images acquired during the particular imaging session. Fluorophores are detailed later in 3.6.1. The remainder of this section lists key terms and equations from the previous work, and describes the updates for dual excitation.

First the fluorescence emission is digitized by the FMC104 and sampled in the FPGA acquisition logic: $d^{\text {measured}}_{(\lambda _{\text {em}},y,x,t)}$. Then the DFT is calculated in the FPGA at the prescribed frequencies, giving $D^{\text {measured}}_{(\lambda _{\text {em}},y,x,f)}$. After the DFT is calculated, another dimension is introduced for the excitation wavelength $\lambda _\text {ex}$, and an additional label is added to the frequency $f$ to indicate the excitation wavelength $f^\text {ex}$ where ex is either 375 or 445 (nanometers). The data is then rewritten as $D^{\text {measured}}_{(\lambda _{\text {em}},\lambda _\text {ex},y,x,f^\text {ex})}$. Then the complex frequency domain data (one real and imaginary 64-bit number per frequency) is transferred from the FPGA to the computer.

Once $D^{\text {measured}}_{(\lambda _{\text {em}},\lambda _\text {ex},y,x,f^\text {ex})}$ is transferred to the host computer, system calibration is performed (detailed in the Data Processing - Theory section of the Supplement 1). The final calibrated data consists of normalized intensities and apparent phase and modulation lifetimes. $\hat {I}^{C-\text {inter}}_{(\lambda _\text {em},\lambda _\text {ex},y,x)}$ is the inter, or between-emission-channel intensity. For example, $\hat {I}^{C-\text {inter}}_{(405,375,7,4)}=0.5$ indicates that half of the fluorescence from 375 nm excitation was detected in emission channel 405 nm at pixel location (7,4) in the given image. $\hat {I}^{C-\text {intra}}_{(\lambda _\text {em},\lambda _\text {ex},y,x)}$ is the intra, or within-emission-channel intensity. An intra-intensity of $\hat {I}^{C-\text {intra}}_{(484,375,10,50)}=0.7$ indicates that 70% of the fluorescence intensity in channel 484 nm was from 375 nm excitation at pixel location (10,50) in the given image.

The same terms introduced in the background section are used to denote the apparent phase and modulation lifetimes. The LT subscript is added to the frequency term $f^\text {ex}_\text {LT}$ to indicate that this is a modulation frequency used for lifetimes. Now the $\phi (\lambda _{\text {em}},\lambda _{\text {ex}},y,x,f^\text {ex}_\text {LT})$ and $m (\lambda _{\text {em}},\lambda _{\text {ex}},y,x,f^\text {ex}_\text {LT})$ terms denote the phase and modulation after system calibration.

Finally, the $\overline \tau$ term denotes the average lifetime from fitting a bi-exponential model to the calibrated FD-FLIM data as explained in 3.6.2. The $\overline \tau$ term is introduced to better compare the in vivo lifetimes with other studies.

3.3 Data drocessing - FPGA

FPGA processing was developed to allow for real time imaging, to lower equipment costs, and to facilitate adapting the FD-FLIM system to a clinical setting. The 4-channel 250MS/s digitized fluorescence emission from the FMC104 has a throughput of 1.862GiB/second, assuming each 14-bit number was transferred as a 16-bit number from the FPGA to a computer in a typical imaging system setup without FPGA processing. This relatively large data throughput would require an interface such as PCIe (peripheral component interconnect express) for real time performance. By performing the DFT at the modulation frequencies of interest on the FPGA, the data throughput is reduced from 1.862GiB/s to roughly 9MiB/s before transferring the data to the computer. The FD-FLIM system calibration, intensity, apparent phase lifetime, and apparent modulation lifetime calculations are then performed on the computer.

The DFT algorithm implemented on the Zedboard PL (detailed in the Supplement 1) is an updated version of our previous work [16]. The DFT was implemented in VHDL with core math operations implemented with free Xilinx IP cores that were included in Xilinx Vivado 2019.1. The resource usage summary of the overall design and various components is listed in the Supplement 1. The DFT is calculated in two decimated sums as shown below in Eq. (9). $N$ is the maximum number of time points in the DFT (65536), and $M$ is the actual number of time points in the DFT ($M$=20000 for a pixel frequency of 12.5kHz). The $z[n]$ and $Z[k]$ terms are shorthand for $d^{\text {measured}}_{(\lambda _{\text {em}},y,x,n)}$ and $D^{\text {measured}}_{(\lambda _{\text {em}},\lambda _{\text {ex}},y,x,k)}$, respectively, as the DFT calculations are identical for all $y$ and $x$, with different $k$ values for various $\lambda _{\text {em}}$ and $\lambda _{\text {ex}}$ combinations. Index terms $n$ and $k$ are substituted for their time and frequency counterparts, $t$ and $f$, respectively.

3.4 Alternating mode

The fluorescence emission from each excitation consists of frequency components at the DC approximation frequency $f^\text {ex}_{\approx DC}$, the lifetime frequency $f^\text {ex}_\text {LT}$, DC approximation frequency harmonics ($2f^\text {ex}_{\approx DC}$, $3f^\text {ex}_{\approx DC}$, etc.), lifetime frequency harmonics ($2f^\text {ex}_\text {LT}$, $3f^\text {ex}_\text {LT}$, etc.), and mixed harmonics ( $f^\text {ex}_\text {LT} \pm f^\text {ex}_{\approx DC}$, $f^\text {ex}_\text {LT} \pm 2f^\text {ex}_{\approx DC}$, $\dots$, $2f^\text {ex}_\text {LT} \pm f^\text {ex}_{\approx DC}$, $2f^\text {ex}_\text {LT} \pm 2f^\text {ex}_{\approx DC}$, $\dots$, $3f^\text {ex}_\text {LT} \pm f^\text {ex}_{\approx DC}$, $3f^\text {ex}_\text {LT} \pm 2f^\text {ex}_{\approx DC}$, etc.). The fluorescence emission signal also contains noise across the entire frequency spectrum due to the discrete nature of light detection. Although separate non-overlapping digital modulation frequencies are used for each laser, the detection noise from one $\lambda _\text {ex}$ had a noticeable effect on the signal from the other $\lambda _\text {ex}$ within the same emission channel $\lambda _\text {em}$. Alternating mode was implemented to provide full noise isolation within a given emission channel between the emission from each $\lambda _\text {ex}$ at the expense of halving the acquisition time for the given $\lambda _\text {ex}$. Alternating mode was implemented by making $\lambda _{\text {ex}}=375\text {nm}$ active for the first half and $\lambda _{\text {ex}}=445\text {nm}$ active for the last half of the dwell time of each spatial pixel, respectively. The DFT logic was modified to include enable/disable logic for alternating mode. When alternating mode is active, 0’s are multiplexed in place of the digitized fluorescence emission to effectively zero pad the DFT logic for the inactive excitation wavelength. Similarly in the laser control logic, the inactive excitation wavelength’s laser control signal is driven low to disable it during the inactive half of the spatial pixel. Alternating mode was used for all data presented in this work.

3.5 Excitation light exposure and MPE

The excitation light exposure of the system was calculated in order to ensure that in vivo images of skin could be acquired without tissue damage. The maximum permissible exposure (MPE) for skin was calculated according to the guidelines for the American National Standards Institute (ANSI) for Safe Use of Lasers [17]. The thermal MPE over a limiting aperture with a diameter of 3.5mm was the limiting exposure case. The MPE for each excitation source is $\text {MPE}_{375\text {nm}} = 0.56 \cdot \left ( T_{3.5 \phi } \right ) ^ {0.25} \frac {\text {J}}{\text {cm}^2} = 0.63\ \frac {\text {J}}{\text {cm}^2}$ and $\text {MPE}_{445\text {nm}} = 1.1 \cdot \left ( T_{3.5 \phi } \right ) ^ {0.25} \frac {\text {J}}{\text {cm}^2} = 1.24\ \frac {\text {J}}{\text {cm}^2}$. The worst case exposure of the system was calculated assuming all pixels were acquired in a 3.5mm diameter aperture; thus, $T_{3.5 \phi }$ corresponds to the exposure time for the 3.5mm aperture. All images acquired in this work utilized the same parameters unless otherwise specified: alternating mode, 12.5kHz pixel rate, 4mm by 3mm field of view, and a raster scanned image size of 160 by 160 pixels (plus 80 flyback pixels) for a total imaging time of 3.072 seconds. The average power at the sample position, including the digital modulation signals and alternating mode, was 9mW for $\lambda _\text {ex} = 375\text {nm}$ and 14mW for $\lambda _\text {ex} = 445\text {nm}$. The digital modulation signals for both lasers were disabled during flyback by the FPGA logic to limit excess exposure to the sample being imaged. The number of pixels in the 3.5mm aperture was: $N_{3.5 \phi } = \pi \left [ 3.5\text {mm} / 2 \right ] ^2 / \left [ \frac {4\text {mm}}{160} \cdot \frac {3\text {mm}}{160} \right ] = 20525$. The exposure time for the 3.5mm aperture was $T_{3.5 \phi } = N_{3.5 \phi } / 12.5\text {kHz} = 1.642\ \text {seconds}$. The exposures for each excitation wavelength were $\text {Exp}_{375} = 9\text {mW} \cdot T_{3.5 \phi } / \left [ \pi ( 3.5\text {mm} / 2 ) ^2 \right ] / 1\mathrm {e}{4} \frac {\text {J}}{\text {cm}^2} = 0.154 \frac {\text {J}}{\text {cm}^2}$ and $\text {Exp}_{445} = 14\text {mW} \cdot T_{3.5 \phi } / \left [ \pi ( 3.5\text {mm} / 2 ) ^2 \right ] / 1\mathrm {e}{4} \frac {\text {J}}{\text {cm}^2} = 0.239 \frac {\text {J}}{\text {cm}^2}$. The worst case exposure of the system was less than the MPE for both excitation sources in the worst case exposure scenario: $\text {Exp}_{375} = 0.154 \frac {\text {J}}{\text {cm}^2} < 0.63 \frac {\text {J}}{\text {cm}^2}$ and $\text {Exp}_{445} = 0.239 \frac {\text {J}}{\text {cm}^2} < 1.24 \frac {\text {J}}{\text {cm}^2}$. The actual exposure of the system was lower than the worse case, as the field of view was larger than the 3.5mm limiting aperture, the imaging time was longer, and the lasers were disabled during the 50% flyback portion of the raster scan.

3.6 Fluorescent sample preparation and FD-FLIM imaging

3.6.1 Standard fluorophores

The following fluorophores were imaged to evaluate the fluorescence lifetime measurement accuracy of the FD-FLIM system: 1µM 9CA (9-Anthracenecarbonitrile, Sigma Aldrich 152765) in ethanol (E7023, Sigma-Aldrich), 1µM ANT (anthracene, Sigma Aldrich A3885) in ethanol, 0.1µM Coumarin 6 (C6, Sigma Aldrich 546283) in ethanol, 1µM DPA (9,10-Diphenylanthracene, Sigma Aldrich D205001) in ethanol, 0.1µM FAD (flavin adenine dinucleotide, Sigma Aldrich F6625) in PBS (deionized water mixed with phosphate buffered saline P5368, Sigma-Aldrich), 1µM FLU (Fluorescein, 46960, Sigma Aldrich) in PBS, 0.1µM FLU di (Fluorescein, 46960, Sigma Aldrich) in di (deionized water), 100µM NADH (nicotinamide adenine dinucleotide, Sigma Aldrich 10107735001) in PBS, 0.1µM POPOP ([1,4-Bis(5-phenyl-2-oxazolyl)benzene], Sigma Aldrich P3754) in ethanol, 0.1µM Rub (Rubrene, Sigma Aldrich 554073) in methanol, and 100µM RoseB (Rose Bengal, Sigma Aldrich 330000) in PBS. Fluorophore solutions were not degassed. Concentrations for all fluorophores were diluted from stock concentrations to avoid saturating the APDs and digitizer after the APD gains were set. The gain for each APD was set by acquiring data in vivo from an human finger and gradually increasing the gain until the analog output of the APD saturated the analog input of the FMC104 digitizer. The same APD gains were used to image all reference fluorophores and in vivo samples in this work. Literature lifetimes for the aforementioned fluorophores are listed in the fluorescent sample preparation and imaging section of the Supplement 1. Fluorophores were placed in quartz cuvettes (R-3010-T, Spectrocell) for imaging. Overhead room lights in the lab remained on throughout imaging to mimic ambient light conditions in a clinical setting. Cuvettes were positioned so that the field of view also contained an empty region from outside the cuvette to be used for background subtraction. One set of reference fluorophore data (POPOP in ethanol, fluorescein in PBS, and rose bengal in PBS) was acquired at the start of the imaging session and used to calibrate all images acquired in the imaging session, as described previously in 3.2.

3.6.2 In vivo human finger and human oral mucosa

The FD-FLIM system’s ability to image autofluorescence of biological tissue was demonstrated by imaging in vivo human finger skin and human oral mucosa. First, the reference fluorophores (POPOP in ethanol, fluorescein in PBS, and rose bengal in PBS) were imaged as described previously in 3.2. Then, in vivo images were acquired from the tip of a human index finger and the apex of a human tongue. One image without anything at the sample position (blank image) was acquired after each in vivo image for background subtraction. Overhead room lights in the lab remained on throughout imaging to mimic ambient light conditions in a clinical setting. Fluorescence average lifetimes were measured for the in vivo data from a dual-exponential decay function fitted to the magnitude and phase frequency responses measured at the three used modulation frequencies.

4. Results and discussion

4.1 Reference fluorophores

The reference fluorophore data acquired with the FD-FLIM system is summarized in Table 1, Table 2, and Table 3. The reported mean calibrated inter-channel ($\hat {I}^{C-\text {inter}}$) and intra-channel ($\hat {I}^{C-\text {intra}}$) normalized intensities were in agreement with the excitation and emission spectral characteristics of each fluorophore. NADH was excited only at 375nm, with the strongest emission in the 484nm band and significant emission also in the 553nm band. C6 was more strongly excited at 445nm, with strong emission in both the 484nm and 553nm bands. FAD and FLU were excited at both 375nm and 445nm, with dominant emission ($~70-75\%$) in the 554nm band. ANT was only excited at 375nm, with dominant emission ($~90\%$) in the 405nm band. DPA was excited only at 375nm, with the strongest emission in the 405nm band and significant emission also in the 484nm band. RUB was more strongly excited at 445nm, with the strongest emission in the 553nm band and significant emission also in the 646nm band. 9CA was only excited at 375nm, with the strongest emission in the 484nm band and significant emission also in the 405nm and 553nm bands.

Table 1. Fluorophore normalized intensities and lifetimes imaged with the FD-FLIM system (part 1/3). The left column has the fluorophore names, $\lambda _{\text {ex}}: \lambda _{\text {em}}$ lists the excitation and center emission wavelengths, $\hat {I}^{C-\text {inter}}::\hat {I}^{C-\text {intra}}$ lists the mean and standard deviation in percent (0-100%) rounded to the nearest integer of the calibrated inter-channel and intra-channel normalized intensities, the $\overline \tau$ column contains the average lifetimes, the $\tau$ column indicates if the entries in the remaining right columns are the apparent phase or modulation lifetimes ($\tau _{\phi }$ or $\tau _m$), the remaining $f^\text {ex}_\text {LT}$ columns list the mean and standard deviation of the apparent phase or modulation lifetimes (nanoseconds). For intensities and lifetimes, the values are listed as mean(standard deviation). Fluorophore solutions were not degassed. Lifetime entries in the table were omitted if the standard deviation was larger than 25% of the mean, except in cases where the standard deviation was $<0.5\text {ns}$. Fluorescence average lifetimes, $\overline \tau$, were measured from a dual-exponential decay function fitted to the magnitude and phase frequency responses measured at the three used modulation frequencies.

View Table | View all tables in this article

Table 2. Fluorophore normalized intensities and lifetimes imaged with the FD-FLIM system (part 2/3). See Table 1 for a detailed description of the table values.

View Table | View all tables in this article

Table 3. Fluorophore normalized intensities and lifetimes imaged with the FD-FLIM system (part 3/3). See Table 1 for a detailed description of the table values.

View Table | View all tables in this article

The apparent phase and modulation lifetime values were in agreement with the values reported in the literature, including in our previous work [16]. For NADH, the mean values of $\tau _\phi$ range from 0.42ns-0.48ns, close to the expected lifetime of 0.44ns. The values of $\tau _m$ are overestimated due to the short lifetime of NADH relative to the modulation frequencies used in the system (limited to <100MHz) and how $\tau _m$ is calculated. In such cases the distribution of modulation $m$ values will be centered close to 1, but with some values being $>1$ and $<1$ depending on the signal quality. Values of $m>1$ were omitted because they resulted in complex values for $\tau _m$ when applying Eq. (3). This omission lowered the mean value of $m$ and increased the mean value of $\tau _m$ when results were reported in Table 1, Table 2, and Table 3. More accurate sub-nanosecond lifetime estimations is possible if higher modulation frequencies are used; however, the bandwidth of the system was purposely restricted to 100MHz in order to use cost-effective detectors and sampling electronics, as discussed in the DFT configuration section of the Supplement 1. C6 lifetimes are slightly longer than the expected 2.4ns, with $\tau _m$ being the most overestimated. As expected, we observed monoexponential behavior with similar $\tau _\phi$ and $\tau _m$ values across all modulation frequencies for C6. The $\tau _\phi$ and $\tau _m$ lifetime values for FAD are close to the expected literature value of 2.70ns. Since FAD presents multiple exponential components, we observed as expected $\tau _m > \tau _\phi$ for each $(\lambda _\text {em}, \lambda _\text {ex}, f^\text {ex}_\text {LT})$. Furthermore, the apparent lifetimes of FAD at $\lambda _\text {ex} = 375\text {nm}$ were similar to our previous work [16], although the emission channels were not identical. The mean values of $\tau _\phi$ and $\tau _m$ for FLU in diwater ranged from 3.20ns to 3.70ns, which is shorter than the $\approx 4\text {ns}$ lifetime of FLU in PBS that was used for calibration [18]. This is expected as the pH of diwater is lower than PBS (although the pH of the solutions were not measured). ANT, DPA, and 9CA lifetime values were close to the expected literature values. The RUB $\tau _m$ and $\tau _\phi$ lifetimes are approximately 2-3ns shorter than the literature degassed lifetime of 9.9ns. Shorter lifetimes were expected for RUB, as the solution of RUB was not degassed before imaging.

Overall, these results demonstrated the following capabilities of this FD-FLIM system. First, it was possible to excite fluorophores with significantly different absorption spectra. Second, it was possible to image both fluorescence intensity and lifetime at multiple emission spectral bands simultaneously. Third, it was possible to measure fluorescence lifetimes within a broad range of values (0.4-12ns) using only three modulation frequencies spanning a narrow bandwidth of 100MHz.

4.2 In vivo human finger and human oral mucosa

In vivo human finger skin and in vivo human tongue apex mucosa were imaged to demonstrate the FD-FLIM system’s ability to image endogenous fluorescence. Calibrated normalized intensity maps and average lifetime maps are shown in Fig. 3 and Fig. 4 for human finger skin and human tongue mucosa, respectively. The background subtracted data $D_(\lambda _\text {em}, \lambda _\text {ex}, y, x, f^\text {ex})$ was spatially averaged by a 3x3 pixel window before calculating the average lifetimes to improve signal quality. The calibrated normalized intensity maps were not spatially averaged.

 

Fig. 3. In vivo image of human index-finger tip skin. Calibrated inter-channel intensities (Eq. (4), $\hat {I}^{C-\text {inter}}_{(\lambda _\text {em},\lambda _\text {ex},y,x)}$) are shown in the top two rows. Calibrated intra-channel intensities (Eq. (5), $\hat {I}^{C-\text {intra}}_{(\lambda _\text {em},\lambda _\text {ex},y,x)}$) are shown in the third row. Fluorescence average lifetime maps are shown in the last two rows for each $\lambda _\text {ex}$. Emission channels are organized in columns, with the center wavelength labeled at the top of the plots.

Download Full Size | PDF

 

Fig. 4. In vivo image of a human tongue mucosa. Calibrated inter-channel intensities (Eq. (4), $\hat {I}^{C-\text {inter}}_{(\lambda _\text {em},\lambda _\text {ex},y,x)}$) are shown in the top two rows. Calibrated intra-channel intensities (Eq. (5), $\hat {I}^{C-\text {intra}}_{(\lambda _\text {em},\lambda _\text {ex},y,x)}$) are shown in the third row. Fluorescence average lifetime maps are shown in the last two rows for each $\lambda _\text {ex}$. Emission channels are organized in columns, with the center wavelength labeled at the top of the plots.

Download Full Size | PDF

The dual-excitation, multispectral FD-FLIM maps of the finger tip reflect the endogenous fluorophores (collagen, elastin, NADH, FAD) expected in human skin (Fig. 3). Collagen and elastin fluorescence emission is expected in the 405nm band under 375nm excitation, with average lifetimes of $\sim$2ns for elastin and $\sim$4ns for collagen. This is consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity and average lifetime (range: $\sim$2.5-3ns) maps observed in the 405nm band. Elastin, NADH and FAD fluorescence emission is expected in both the 484nm and 553nm bands under 375nm excitation, with average lifetimes of $\sim$4-5ns for elastin, $\sim$0.5-2.5ns for NADH, and $\sim$0.5-3ns for FAD. This is also consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity and average lifetime (range: $\sim$3-4ns) maps observed in both the 484nm and 553nm bands. None of these endogenous fluorophores emit at the 646nm band under 375nm excitation, which is consistent with the corresponding $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity map. Elastin and FAD fluorescence emission is expected in both the 484nm and 553nm bands under 445nm excitation, with average lifetimes of $\sim$4-5ns for elastin and $\sim$0.5-3ns for FAD. This is also consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity and average lifetime (range: $\sim$3.5-4.5ns) maps observed in both the 484nm and 553nm bands. Some FAD fluorescence emission is expected in the 646nm band under 445nm excitation, which is consistent with the average lifetime map (range: $\sim$2.5-3.5ns) observed in the 646nm band.

The dual-excitation, multispectral FD-FLIM maps of the tongue apex reflect the endogenous fluorophores (collagen, NADH, FAD, porphyrin) expected in human oral mucosa (Fig. 4). Some collagen fluorescence emission is expected in the 405nm band under 375nm excitation. This is consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {e}=375nm)}$ intensity and average lifetime maps observed in the 405nm band, which show patches of tissue with both measurable fluorescence intensity and lifetime values of $\sim$3.5-4.5ns. NADH and FAD fluorescence emission is expected in both the 484nm and 553nm bands under 375nm excitation. This is consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity and average lifetime (range: $\sim$2.5-3.5ns) maps observed in both the 484nm and 553nm bands. Some porphyrin fluorescence emission is expected in the 646nm band under 375nm excitation, with average lifetimes longer that $\sim$4ns. This is consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity and average lifetime maps observed in the 646nm band, which show patches of tissue with both measurable fluorescence intensity and lifetime values of $\sim$3-5ns. FAD fluorescence emission is expected in both the 484nm and 553nm bands under 445nm excitation. This is consistent with the $\hat {I}^{C-\text {inter}}_{(\lambda _\text {ex}=375nm)}$ intensity and average lifetime (range: $\sim$2.5-3.5ns) maps observed in both the 484nm and 553nm bands. Some FAD fluorescence emission is expected in the 646nm band under 445nm excitation, which is consistent with the average lifetime map (range: $\sim$2-3ns) observed in the 646nm band.

4.3 Instrumentation

We have implemented a versatile and practical FD-FLIM engine capable of simultaneous multi-wavelength excitation, simultaneous multispectral detection, and sub-nanosecond to nanosecond fluorescence lifetime estimation. The engine was developed using a single FPGA (ZedBoard) programmed to provide all essential functionalities needed for FD-FLIM measurements, including laser modulation, optical scanning control, and time-resolved fluorescence signal digitization (via FMC104). The FPGA also provides real-time processing (i.e., DFT) of the digitized time-resolved fluorescence signal and Ethernet connectivity, thus enabling using the FD-FLIM engine with a low-end desktop computer or laptop computer. Relative to our previous work [16], this updated design offers enhanced capabilities, while significantly reducing FD-FLIM implementation complexity and cost.

In time-domain (TD) FLIM implementations, pulsed light sources are required for excitation. For TD-FLIM implementations based on time-correlated-single-photon-counting (TCSPC), single-photon excitation is achieved using pulsed light sources with low pulse-energy which are available in wavelengths spanning the UV-VI-NIR range. Multi-wavelength excitation TD-FLIM based on TCSPC, however, is not practical due to its innate high implementation cost and slow acquisition speed. Faster TD-FLIM implementations can be achieved using pulsed lasers with higher pulse-energy [19]. These light sources, however, are costly and limited to a few available excitation wavelengths (e.g., 355nm, 532nm, 1064nm), thus, unfitting for multi-wavelength excitation. In FD-FLIM implementation, frequency-modulated CW light sources can be used for excitation. CW light sources are less costly than pulsed light sources and also available in a selection of wavelengths spanning the UV-VI-NIR range (375-1064nm). Multi-wavelength excitation FD-FLIM can thus be readily implemented using CW light sources modulated at frequencies that are not harmonically related, as demonstrated in this work.

Standard light excitation strategies in FD-FLIM are based on analog light modulation, in which the modulation frequency is swept over a frequency range spanning the expected bandwidth of the time-resolved fluorescence signal. Although these strategies enable estimating the entire fluorescence frequency response, they are innately slow. In our FD-FLIM implementation, a simple digital laser pulse modulation approach was adopted to enable faster imaging acquisition speed. The applied digital laser pulse modulation provides simultaneous frequency interrogation at the fundamental frequency and corresponding harmonics. Thus, the fluorescence frequency response is estimated only at a few discrete frequencies over a limited frequency range. Nevertheless, our rigorous validation experiments have demonstrated that fluorescence lifetimes can be estimated over a wide range ($\sim$0.5ns to >10ns) from the fluorescence frequency response measured only at three discrete frequencies.

In TD-FLIM, the time-resolved fluorescence signal is measured with sub-nanosecond temporal resolution, thus requiring high-bandwidth (>1GHz) photodetectors and electronics. In standard FD-FLIM implementations, the bandwidth requirements can be relaxed; however, sub-nanosecond lifetimes are typically measured using modulation frequencies in the 100’s MHz. To further reduce the bandwidth requirements of our FD-FLIM engine, the laser digital modulation was restricted by design to frequencies below 100MHz. This enables implementing cost-effective and practical FD-FLIM systems with simultaneous multispectral detection capabilities using low-cost, limited-bandwidth photodectors and multi-channel digitizers. In spite of the limited bandwidth of our FD-FLIM engine, our rigorous validation experiments have demonstrated that fluorescence lifetimes as short as 0.5ns can be simultaneously measured at multiple emission spectral bands.

Standard FD-FLIM implementations require measuring a reference signal to synchronize or demodulate the excitation and emission signals. This requirement increases the instrumentation complexity, particularly for multi-wavelength excitation, as additional optics, photodecetors and/or digitizer input channels are needed to measure these reference signals. In our FD-FLIM engine, the 250MHz clock signals from the FMC104 digitizer are used to generate the digital modulation signals for the two lasers, and to sample the four spectrally separated fluorescence emission signals in the FMC104 digitizer. The excitation/emission signal synchronization provided by the common clocks simplifies our system instrumentation, as reference signals are not required to be measured. Instead, reference fluorophores with known lifetimes are imaged to measure the system frequency response needed for fluorescence lifetime estimation.

Although the lifetime resolving performance our FD-FLIM implementation was comparable to the dual-excitation system described in [10,11], our implementation has advantages that are targeted towards clinical imaging. The system described in [10,11] required an interferometer and a spinning polygon mirror to generate excitation waveforms. While that approach resulted in superior modulation frequency coverage, its larger optical complexity would make clinical imaging more difficult. By contrast, our system does not require additional optics to measure the fluorescence lifetime of multiple fluorophores excited simultaneously with different excitation wavelengths.

4.4 Future work

Several improvements with respect to our previous FD-FLIM engine design [16] were accomplished in this work. First, the new FD-FLIM engine can be fully implemented in a single FPGA, resulting in a significant reduction in cost (see Table 4) and a compact form factor (as illustrated in Fig. 2). The new engine also enables applying simultaneous dual-wavelength excitation and acquiring time-resolved fluorescence signal at an additional (fourth) spectral emission channel. The DFT implementation was also optimized to calculate more DFTs with fewer FPGA resources while maintaining the same fixed point math precision. This allows increasing the number of modulation frequencies that can be simultaneously used, which enables implementing simultaneous multi-wavelength excitation based on frequency multiplexing. Finally, by performing the DFT calculations on the FPGA, the data throughput is reduced to $\sim$9MiB/s, enabling interfacing the FD-FLIM engine via Ethernet with a low-end PC or laptop computer (an FD-FLIM engine running from a laptop has been installed at Cedars-Sinai Medical Center, in Los Angeles, CA). This versatile, simple, compact, and cost-effective FD-FLIM implementation will facilitate the clinical translation of FLIM imaging and microscopy.

Table 4. Cost summary of a previous implementation of a TD-FLIM system [20], the 1$^\text {st}$ generation of our FD-FLIM system [16], and our 2$^\text {nd}$ generation FD-FLIM system. Costs do not include optical components as those are equivalent across all of the listed point scanning FLIM systems. A detailed list of components for this cost reduction comparison is given in the Supplement 1.

View Table | View all tables in this article

4.5 Limitations

One of the main advantages of the developed FD-FLIM implementation is the use of a simple band-limited digital square wave modulation strategy enabling excitation at simultaneous modulation frequencies. A limitation of this strategy, however, is that the number of simultaneous modulation frequencies is limited by the frequency content of the periodic square wave digital modulation signals used. Alternative digital modulation signals can be further developed to allow for higher number of simultaneous modulation frequencies with more spread power distribution across the 100MHz bandwidth of the system (digital chirp, etc.). The FPGA design already includes logic and block memory for storing and outputting an arbitrary digital waveform for each laser; thus, once this is developed, it can be readily implemented into the FD-FLIM engine.

4.6 Summary

We have presented a novel FD-FLIM instrumentation design capable of simultaneous multi-wavelength excitation, simultaneous multispectral detection, and sub-nanosecond to nanosecond fluorescence lifetime estimation. Laser digital modulation was adopted to enable simultaneous frequency interrogation at the fundamental frequency and corresponding harmonics. Time-resolved fluorescence detection is implemented using low-cost, fixed-gain, narrow bandwidth APDs, thus, enabling cost-effective fluorescence lifetime measurements at multiple emission spectral bands simultaneously. Synchronized laser modulation and fluorescence signal digitization is implemented using a common FPGA. The FPGA also allows implementing real-time processing by performing the DFT of the fluorescence emission. Rigorous validation experiments have demonstrated the capabilities of this novel FD-FLIM implementation to accurately measure fluorescence lifetimes in the range of 0.5-12ns. In vivo endogenous, dual-excitation (375nm/445nm), multispectral (four bands) FD-FLIM imaging of human skin and oral mucosa at 12.5kHz pixel rate and room-light conditions was also successfully demonstrated. This versatile, simple, compact, and cost-effective FD-FLIM implementation is already enabling the development of a number of imaging systems that are being evaluated for different clinical and biomedical applications. These include four FLIM endoscopy imaging systems for label-free biochemical/metabolic imaging of benign, precancerous, and cancerous oral lesions; one handheld FLIM imaging system for label-free biochemical/metabolic imaging of benign and cancerous skin lesions; one benchtop FLIM imaging system for label-free biochemical imaging of surgical margins in resected oral cancer tumors; one benchtop FLIM imaging for label-free biochemical imaging of biobank tissue samples. Current efforts are focused in overcoming the limitations described before (4.5) to further facilitate the clinical translation of FLIM imaging.