1. Introduction

Fluorescence imaging is a powerful tool for the analysis of materials, particularly in the context of biological applications, as many biomolecules exhibit auto-fluorescence upon illumination. This emitted light may be used as a fingerprint of the materials present, as well as of their local environment. Fluorescent Lifetime IMaging (FLIM) differs from traditional fluorescence imaging in that rather than just the intensity or spectra of the emitted light, time resolved detection systems are used to obtain the characteristic fluorescent lifetime. This can have applications in fields as diverse as biomedicine [1–10], plant science [11–13], or chemical sensing [14]. The primary benefit of FLIM over traditional fluorescence intensity imaging is that lifetime is largely independent of the density of fluorescent chromophores and excitation power, giving consistent contrast between regions with differing molecular makeup. A particularly promising avenue for FLIM applications is for surgical guidance and endoscopy [2–8], where FLIM can provide label free contrast between tissue types which may not be apparent when using white light imaging, or fluorescence intensity alone. This is particularly useful when looking at cancer margins, as cancerous tissue has been shown to have a different characteristic fluorescence lifetime compared to surrounding healthy tissue [2–4,10]. Several different biomedical FLIM systems have been demonstrated in the literature, generally all employ a pulsed laser source to induce fluorescence (either endogenous or from labels) which is then collected and relayed down a fibre to an image sensor at the proximal end. The spatial resolution is then either achieved using scanning optics at the proximal end of the system [7], the use of fibre imaging bundles [1,15], or by raster-scanning a fibre acting as a point probe over the object being imaged [6,10,16,17].

FLIM depends upon being able to temporally resolve the fluorescence signal. There are several methods for achieving the necessary time resolution required to perform FLIM, such as time gated optical intensifiers [2] and high speed digitisers [17,18] but one of the most robust and elegant approaches is the use of SPADs where timing electronics for the imaging pixel are integrated at a chip level [8]. Once SPADs are combined into arrays they may become an even more powerful tool. SPAD array line sensors are very well suited to spectrally resolved measurements, allowing for FLIM to be carried out at multiple spectral bands simultaneously [8,19], while 2D SPAD arrays can effectively act as time resolved cameras capable of rapidly performing wide-field FLIM [20,21].

This group has previously demonstrated Endocam, a novel SPAD array specifically designed to perform FLIM in a chip-on-tip fashion, i.e. the SPAD array itself will sit on the distal end of the system with images relayed back to the control unit via a wired data connector [22,23]. This stands in contrast to the proximally mounted sensor of all the FLIM systems described previously and allows a vast simplification of the opto-mechanics required to reconstruct the image. Such systems do not face the same limitations with regards bending radius as fibre systems, and due to the inherent scalability of electronics versus optics may be a lower cost solution [24,25]. Although fluorescence endoscopy systems with an image sensor on the distal end of the probe do exist, these only provide steady state fluorescence intensity rather than FLIM [26,27,25].

For surgical guidance and other diagnostic or analytic applications, it is highly valuable for the FLIM system to be flexible and mobile enough such that it can be operated in a handheld fashion e.g. to image a patient undergoing surgery from different directions without having to move and disturb them. For endoscopy applications, any chip-on-tip FLIM system has to be able to operate at a distance from its control unit. In this work, both of these goals are achieved, clearly demonstrating the potential to integrate time gated CMOS SPAD arrays into an endoscopy system.

To the best of the authors’ knowledge, the system presented here based on the Endocam chip is also the first example of a time resolved SPAD array on the distal end of a handheld FLIM system, and represents an important staging post in the development of chip on tip FLIM endoscopy systems.

2. Methods and materials

2.1 FLIM system

A photograph of the handheld system is shown in Fig. 1(a). The maximum dimensions of the head in each direction are ∼ 4.5 cm x 4.5 cm x 4.5 cm. Figure 1(b) shows the entire system with the motherboard along with the > 1 m cable used for power and data transfer and the optical fibre used to deliver excitation light. A schematic of the optical design of the imager head is shown in Fig. 1(c). The excitation beam, shown by the blue arrow, is generated here by a Hamamatsu Picosecond Light Pulser PLP-10 laser diode head with λ = 483 nm, pulse-width = 80 ps (though replacing with another laser source would be trivial), and is coupled into a commercially available multimode fibre (NA = 0.5, Thorlabs M124L02) which delivers ∼0.3 mW of power. Excitation light is reflected at ∼ 45° by a long pass dichroic (Thorlabs DMLP550R, 550 nm cut on wavelength) out of the head and onto the field of view. The emitted fluorescence (shown by the red arrow) passes through a long pass filter (Thorlabs DMLP490T, 490 nm cut on wavelength) and is collected by an aspheric lens (NA = 0.53, EFL = 4.6 mm, Thorlabs A390TM-B) and an image formed on the Endocam chip. The instrument response function for the Endocam chip when used in conjunction with this laser is 0.55 ± 0.02 ns [23]. The intensity image of a ruler in Fig. 1(d) shows the field of view for the system, ∼2.4 cm at a working distance of approximately 6 cm. Also note that one of the corners of the image is corrupted (bottom right corner of Fig. 1(d)). This is due to a hardware issue which does not affect the rest of the chip, and is described in detail in Ref. [23], along with a full description of the chip design.

 

Fig. 1. (a) Photograph of the handheld FLIM module. (b) Photograph of the entire system showing the handheld FLIM module connected to the mother board via a > 1 m cable running alongside an optical fibre. (c) Schematic diagram of the optoelectronic components of the module. Blue arrows represent the path of the excitation beam, the red arrows represent the path of the fluorescence. (d) A ruler imaged in intensity mode. (e) Schematic diagram of the electronic configuration of the system.

Download Full Size | PDF

Briefly, in its current iteration the Endocam die is mounted in a CPGA68 chip carrier package, though the Endocam die itself is only < 2 mm², which will allow much smaller chip carrier packages to be used in future versions. The chip consists of 120 × 128 pixels each comprising of an individual SPAD, SPAD front-end circuitry, and a 14-bit photon counter. The chip also features a micro-controller unit (MCU) and two 16 bit static random access memory (SRAM) blocks to allow successive frames generated from up to 65535 exposure cycles to be added together on chip (a process we will hereafter refer to as frame additions). Although this increases the footprint of the chip somewhat, the SRAM blocks take up less area per bit than the on-pixel photon counters and also cuts down the need for time consuming data transfer off the chip. Thus they allow for a high bit depth without overly compromising the form factor or frame rate of the system [23]. The chip requires only five wires to run- 1) 18.5 V supply for the SPAD bias, 2) 2.8 V supply for the on-chip power generation network, 3) Ground, 4) Data I/O, 5) Clock. The 18.5 V and 2.8 V power supplies are generated on the motherboard, and a field programmable gate array (FPGA) daughter board (Opal Kelly XEM6310) acts as an interface between the computer and the Endocam chip. The schematic diagram in Fig. 1(e) summarises the electronic configuration of the system.

To allow for its very small form factor, the Endocam chip does not have on chip time correlated single photon counting (TCSPC) timestamping. Instead, the time resolution on chip is achieved through time-gating the on-pixel photon counting electronics. The time-gate widths and positions with respect of the rising edge of the reference clock are in multiples of 379 ps which is the minimum time resolution defined by the on-chip ring oscillator [23]. These time gates are synched to a 20 MHz master clock generated by the FPGA, though using an external clock from e.g. a laser driver as master is also possible.

It should be noted that there are no additional buffer circuits or relays to improve the strength or quality of the signals between the FPGA itself and the chip. Sending clock and data signals over a long distance of up to 1 m away from a motherboard is not a typical use case for the FPGA board; as the cable becomes longer, the scope for interference in the signal from external sources increases, along with the impact of reflections along the transmission line. Initial tests were carried out with short lengths (∼ 10 cm) of solid core jumper wire inserted directly into the motherboard chip socket, and attached to the pins of the chip. Although not capable of running at the 37.5 MHz it could operate at on board [23], this initial iteration of the system was capable of running at 10 MHz, and confirmed that it was possible to operate the chip in time-gated mode remote from its board. However, attempts to increase the length of these solid wires to the ∼ 1 m required resulted in the degradation of the clock signal reaching the chip, such that the chip could not boot correctly. Various other cables were explored, but eventually shielded multicore cable (Alphawire 6305 SL005), which was less susceptible to external electrical interference, was found to be effective and allowed the length of the cable to be increased to ∼ 1 m. In this state the chip was still only running at 10 MHz, which was deemed too low a clock rate for practical applications. This then required multiple rounds of firmware revision to optimise the form of the clock signal such that it could be delivered at a higher frequency while maintaining fidelity. Although a clock rate of 37.5 MHz is achievable with the chip mounted directly on its motherboard [23], the 20 MHz presented here was ultimately the maximum which we could achieve in this configuration. Although lowering the clock rate has a corresponding impact on the frame rate, 20 MHz was deemed suitable for this study as it avoided significant fluorescence wrap around from the longest lived chromophores. Additionally, these firmware upgrades provided the opportunity to add some other additional capabilities to the system as it is described in Refs. [22,23], namely the generation of the necessary voltage levels for chip operation on the motherboard, freeing the system from requiring a bulky external benchtop power supply unit, and the ability to use an externally generated TTL or NIM clock signal (for example from a laser driver) as the master clock for the system. A clock rate of 20 MHz is commonly used as the fixed repetition rate of many super-continuum lasers, which may be employed in future versions of the system, so the ability to run at 20 MHz and use an external master clock was deemed a useful addition.

The Endocam chip is controlled via a custom made Matlab (R2021b Mathworks) graphical user interface (GUI). Via the GUI the system offers users four modes of operation –

2.2 Samples

Ovine (lamb) kidney was purchased from a local butcher and used as received.

Barley (Hordeum vulgare) seeds of the variety Digger were sowed in small 6.5 × 6.5 cm pots and maintained in growth cabinets at 16 h of light (21°C, 150 µE m^-2 s^-1)/ 8 h of darkness in Levington Advance F2 + S soil. After 9 days, plants were removed from the cabinet and left near an open window, enabling wild aphid colonization.

Orange and green fluorescent targets were 3D printed in the shape of the letter “E” using reels of PLA (poly-lactic-acid) doped with fluorescent dye. These were the only fluorescent (as opposed to phosphorescent) filaments we were able to obtain, and the supplier (RepRapWorld) did not disclose the precise dyes used in each PLA reel, so characterisation measurements were taken of each to obtain baseline lifetimes.

2.3 Baseline measurements of PLA fluorescent lifetime

For baseline measurements, a commercially available Horiba FLIMera TCSPC camera was used to obtain a ground truth fluorescent lifetime for each target. The FLIMera was fitted with a fixed focal length imaging lens (Navitar, 16 mm EFL, f/1.4) and synched to the same Hamamatsu PLP-10 laser used in the Endocam system. FLIM images were then taken of each E shaped target, and the pixels for the image summed together to give a single decay curve for each object. The process was repeated with the Endocam system and the results compared. These are shown in supplementary Figs. S1 and S2.

3. Results

Figures 2(a) and 2(b) show intensity and FLIM images of two 3D printed letter E’s interleaved, one has been printed from the orange fluorescent PLA and the other from the green fluorescent PLA. Images were obtained using a gate sweep over 10 bins with 65534 exposure cycles and 100 frame addition counts, giving a total acquisition time of ∼14 s. Note that for the FLIM image shown in Fig. 2(b) and those throughout the rest of the paper, the colour of each pixel corresponds to the lifetime (shown in the colour bar) whereas the brightness is the intensity, controlled via the ‘alpha’ channel for the image. Alongside the images are histograms of intensity counts (Fig. 2(c)) and lifetime (Fig. 2(d)) from the corresponding images (excluding regions where counts are below a threshold intensity value). From the distribution of the intensity histogram in Fig. 2(c) it is hard to discern if there are multiple materials being probed. However, the lifetime histogram in Fig. 2(d) shows a clear bi-modal distribution, with well-defined peaks for each material. This shows both the effectiveness of the system as a FLIM tool, and the value of using FLIM rather than just intensity to provide contrast. The lifetimes of these materials are relatively long compared to those normally expected for endogenous biological chromophores, and the breadth of the lifetime distribution implies a degree of heterogeneity in the lifetime, but due to their stability and size these targets are still useful to compare the system against another baseline system. The Endocam chip has previously been shown to give accurate measurements of lifetime when used as part of a microscopy system [23]. To confirm this is still the case for this handheld FLIM system, Figs. S1 and S2 show the fluorescent decay curves obtained using the handheld system for each of the 3D printed fluorescent targets, along with the same decay obtained using a Horiba FLIMera camera. We see that the decay curve for each of the materials obtained with the handheld system matches well with that obtained using the FLIMera, calculating the lifetime for the green target gives 5.25 ns using the FLIMera and 5.36 ns using the Endocam, and for the orange target 8.47 ns using Endocam and 8.54 ns using FLIMera, giving confirmation that the handheld FLIM system provides accurate lifetime measurements. Furthermore, the combination of laser and sensor used here was previously validated against solutions of fluorescein and rhodamine, and gave accurate lifetime results [23].

 

Fig. 2. (a) Intensity image of two fluorescent targets doped with differing fluorophores, (b) FLIM image of the same scene with colour bar giving lifetime in ns and an alpha channel controlling image brightness, (c) Histogram of photon counts from the image shown in (a), (d) Histogram of lifetime values for the image shown in (b).

Download Full Size | PDF

These initial tests on highly fluorescent PLA samples were valuable for validation, but the real application of FLIM is imaging of biologically active materials where fluorescence lifetime gives contrast between tissue types, or reveals the presence of disease. Figure 3(a) shows the intensity image of an ovine kidney imaged using the handheld FLIM system. The image is centred on the renal pelvis, where the ureter, veins and arteries join the organ. In the fluorescence intensity image there is no clear indication that the tissue in this region is of differing composition from that of the rest of the organ. Figure 3(b) shows the fluorescence lifetime image of the same field of view. Immediately apparent is the contrast throughout this image, with lifetimes ranging from ∼1.2 ns to ∼2 ns. It is notable that regions of very similar intensity show clear differences in lifetime. This is consistent with previously published FLIM images of ovine kidney cross-sections which demonstrate longer lifetimes for the renal pelvis compared to the medulla or cortex [29]. This is further confirmed when regions of interest are highlighted. Taking the 20 × 20 pixel regions approximately represented by the boxes in Fig. 3(b) one may generate histograms, as shown in Fig. 3(c). The FLIM contrast between these two regions is very clear in the histograms, as shown by the two distinct distributions of fluorescence lifetimes. The acquisition time for this image (obtained in RLD mode with alternating frames gates each 5 bins wide, 65534 exposure cycles and 1000 addition counts) was 12 s.

 

Fig. 3. (a) Autofluorescence from an ovine kidney, (b) FLIM image of the same scene with colour bar giving lifetime in ns, (c) Histograms of lifetime values for the regions approximately highlighted with the boxes shown in (b). (d) Autofluorescence from two leaves of barley, the upper leaf was visibly stressed while the lower healthy, (e) FLIM image of the same scene with colour bar giving lifetime in ns, (f) Histograms of lifetime values for the bottom leaf (green) and top leaf (pink).

Download Full Size | PDF

As well as biomedical applications, FLIM is being increasingly explored as a diagnostic tool in plant science and agronomy. Various molecules within plants exhibit fluorescence upon illumination (including chlorophyll [30], anthocyanin [31], and lignin [32]). It has previously been shown that plant auto-fluorescence FLIM may be used to detect the presence of disease or stress due to the effect of virus [11], pesticide [12], water-stress [30] or cold damage [13]. Images of a barley plant undergoing an aphid infestation were obtained and Fig. 3(d) shows the intensity images for the auto-fluorescence from two leaves from this plant – the upper leaf had been substantially damaged whereas the lower leaf was not. The intensity image shows differences between these two leaves, with the lower, undamaged leaf demonstrating a much more homogenous intensity response than the upper, damaged leaf which has bright and dark regions. However, it is possible this difference in intensity could simply be due to e.g. un-even illumination due to shadows. Figure 3(e) shows the corresponding lifetime image. Again, the FLIM modality (obtained with sweeping time gates over 12 bins, 65534 exposure cycles, 100 frame additions, 17 s acquisition time) makes the different status of these two leaves much clearer, with the lifetime histogram of the healthy leaf (shown as green in Fig. 4(f)) demonstrating much shorter lifetimes than that for the stressed leaf, shown in pink. This demonstrates that the system is capable of discerning between healthy and stressed plant tissue via auto-fluorescence FLIM and, given its handheld, mobile configuration, it may be used to scan different leaves or parts of a plant without having to damage or disturb them.

 

Fig. 4. Still from a FLIM video of (a) Ovine kidney and (b) barley leaves.

Download Full Size | PDF

The images shown in Figs. 2 and 3 were obtained with acquisition times of 10 to 20 seconds. This shows the capability of the system in achieving high resolution “snapshots” of static objects, which may be of some value in e.g. diagnostic imaging. However, for other applications such as endoscopy, higher frame rates are more important. Figure 4(a) shows a still taken from a short video of an ovine kidney sample (obtained using the RLD method with gates 4 bins wide, 65534 exposure cycles and 100 addition counts, 1.4 s acquisition time) with a frame rate of 0.7 Hz. The image is significantly noisier than that shown in Fig. 3 (b), but FLIM contrast is still apparent, with the tissue around the renal pelvis (now in the middle of the field of view) showing a much longer fluorescent lifetime. As mentioned in the methods section, it is possible to further increase frame rate by switching from using different gates for successive frames to using different gates for odd and even columns. Figure 4(b) shows a still of a FLIM video of barley leaves taken in this mode, with gates 2 bins wide, 65534 exposure cycles and 10 frame additions, to give a frame rate of 1.3 Hz. Despite the loss of lateral resolution due to using this gating scheme, it is still more than sufficient for 4 different barley leaves to be clearly visible within the field of view.

We include further videos obtained using this gating scheme in the supplementary material (Visualization 1 and Visualization 2), one of a selection of 3D printed targets imaged with 65534 exposure cycles and 1 frame addition, and one of two leaves picked from an evergreen shrub with 65534 exposure cycles and 10 frame additions, where we achieve frame rates of > 2 Hz and 1.7 Hz respectively. In these videos we can clearly see the fluorescent lifetime contrast between the differing 3D printed targets, and more importantly, the lifetime contrast in the auto-fluorescence of the leaves. Despite the reduced spatial resolution in this imaging mode, the different objects being imaged are still clearly discernible.

The fact that Endocam is able to provide FLIM images at > 1 Hz while operating at a distance of ∼1 m from its control board is in itself a significant result, and the first demonstration of a SPAD array operating in this manner. These initial results demonstrate that in its current configuration the handheld FLIM system may be suitable for applications such as the scanning of biopsy material, plants or other biologically relevant samples with contrast clearly visible at frame rates > 1 Hz using only endogenous auto-fluorescence. Although diseased human or mammalian tissue was not available for this study, fluorescence microscopy images obtained using a board mounted Endocam as a sensor have already been shown to be able to detect diseased tissue from lung biopsy material [23]. It should be noted that the 483 nm output of the laser used for these demonstration experiments is not necessarily optimised for plant and tissue auto-fluorescence, with shorter wavelength excitation likely to elicit a stronger fluorescence response, and that the 0.3 mW employed here is relatively modest. The fact that lifetime contrast is obtained with these illumination conditions is highly encouraging, with clear scope to improve the signal to noise ratio and/or frame rates merely by paring the module with a different excitation source. The selection of an optimal excitation source will be a key step to further integrating this system into a biomedical device. The other next step for this system is the further miniaturisation required for in-vivo endoscopy applications. The fundamental limit for the size of the system is the footprint of the Endocam die so, with careful and considered device engineering and encapsulation, a system with a ∼ 2 mm² cross sections should be possible. FLIM endoscopes have been shown to be effective tools in the detection of diseases of the larynx [1], lung [8], bowel [2], and mouth [3] and the dimensions required for these sorts of applications should be achievable with a distally mounted Endocam chip.

4. Conclusion

In this work a novel system which employs a purpose designed miniaturised SPAD array for flexible chip-on-tip fluorescence lifetime imaging is demonstrated. This system is capable of frame rates > 1 Hz, has a form factor small enough that it may be held between the thumb and forefinger of an experimentalist or clinician, and operates over ∼1 m of cable and optical fibre to allow targets to be imaged from different angles or distances without the need to move the patient or object under investigation. Initial demonstrations on plant and animal tissues provide high resolution stills and > 1 Hz frame rate videos. FLIM contrast from this system is capable of showing the difference between different tissue types and damaged and healthy tissues. The authors believe this is the first example of a handheld FLIM system with a distally mounted image sensor, and has achieved the operating range and sensitivity required of a biomedical imaging system. This is thus the first step in developing a miniaturised chip-on-tip endoscopy system capable of in-vivo imaging.