1. Introduction

Flexible fiber-based micro-endoscopes are an effective tool for minimally-invasive deep-tissue imaging. They offer micron-scale resolution with miniature probes having diameters of a few hundred microns, making them suitable for in-vivo imaging of neurons and cells [1–3]. One of the popular implementations for miniature micro-endoscopes relies on multi-core fiber bundles (also known as coherent fiber bundles), where each fiber core serves as one imaging pixel. Conventional imaging using fiber bundles is performed either by placing the bare distal fiber tip in contact or close distance to the target, or by adding a gradient-index (GRIN) lens to the distal tip [4,5]. Both solutions suffer from significant undersampling due to the low fill factor caused by the large spacing between fiber cores, compared to the core size [2] and out-of-focus background.

A common approach to improve the resolution and optical cross-sectioning of fiber-bundle endoscopes is through confocal imaging [4,6–9]. In confocal fiber-bundle-based endoscopy the illumination is scanned over each of the fiber cores, and only the light collected from the illumination core is used to form the image. Thus, each single core serves as an effective confocal pinhole, rejecting out-of-focus light and improving the spatial resolution. Importantly, the implementation of confocal fiber-bundle endoscope does not require the addition of any distal elements since the scan is performed at the proximal fiber end.

Over the years, many techniques have been suggested and demonstrated to improve lensless fluorescence fiber-bundle endomicroscopy. These methods include structured-illumination [10,11], retrieving light-field 3D information from the modal excitation within each core [12], optical fluctuation imaging [13], and recently, advanced wavefront-shaping and image reconstruction approaches [14–19]. However, except for the advanced recent approaches where the fiber bundle effectively serves as the imaging pupil [14–19], all approaches suffer from aliasing/pixelization artifacts due to the undersampling by the fiber cores at the image plane. In addition, lensless confocal fiber-bundle endoscopy suffers from low collection efficiency when the target is not in close proximity to the distal facet. Unfortunately, the low effective fill factor of coherent fiber bundles, which leads to image undersampling, is an inherent design limitation required to minimize the cross-talk between neighboring cores [2].

Here, we demonstrate that both the collection efficiency and spatial sampling frequency can be significantly improved for targets placed at a distance from a lensless fiber bundle by adapting the principles of image scanning microscopy (ISM) [20] (also known as pixel-reassignment microscopy [21,22]) to lensless fiber-bundle endomicroscopy.

ISM has emerged as a microscopy technique that improves the signal-to-noise and resolution of confocal microscopes. ISM utilizes the same raster-scanned illumination as in confocal microscopy but replaces the small detection pinhole with a larger pixelated detector array, allowing the use of all signal photons without loss of resolution. In ISM, the resolution is maintained or improved compared to confocal microscopy by reassigning the detected light from each detector (pixel) to the midpoint between the detected and illumination positions. The final ISM image is the sum of the reassigned signals. Mathematically, the pixel-reassignment process is equivalent to spatially scaling by half (shrinking) each raw image from the detector array (Fig. 1), and summing all scaled raw images. Recently, ISM has been used to improve collection efficiency and resolution in confocal Raman microscopy [23], two-photon microscopy [24], scanning laser ophthalmoscopy [25], transmission-matrix based imaging in multi-mode fibers (MMF) [26], and quantum microscopy [27,28].

 

Fig. 1. Image scanning lensless fiber-bundle fluorescence endomicroscopy. (a) setup: At the proximal tip of a multi-core fiber (MCF) a focused laser sequentially illuminates the MCF cores one by one using a galvo-scanner. The excitation light from each core illuminates a target object placed at a small distance from the distal tip (top inset). The fluorescence signal is collected by both the illuminating core and the neighboring cores (bottom inset). The collected fluorescence pattern for each illumination is imaged on an sCMOS camera through a dichroic mirror (DM) and a bandpass filter (F). (b) A simulated single camera image for an object placed at a distance of $60 {\mathrm{\mu} \mathrm{m}}$ from the distal tip, where the center core is illuminated. A confocal image can be generated by recording the signal only from the illuminating core, ignoring the signals from the neighboring cores (c). White ’x’s mark the cores positions, i.e. the sampling grid. (d) An ISM image is formed by resizing the full camera image (b) by a factor of $0.5$, and summing all captured frames. Thus, utilizing all detected photons and creating a $\times 2$ denser sampling grid with new ’virtual’ cores.

Download Full Size | PDF

Here, we combine the raster-scanned illumination of conventional confocal endoscopy [6] with camera-based detection to implement ISM in a bundle-based lensless microendoscope. Our straightforward approach tackles two main drawbacks of conventional confocal fiber-bundle endoscopy. First, we collect fluorescence signals through all fiber-bundle cores rather than a single core. This proves beneficial for targets placed at a distance from the bundle tip, as the collection efficiency is greatly increased. Second, ISM doubles the spatial sampling frequency of the fiber bundle due to the spatial scaling of the pixel-reassignment process (Fig. 1), addressing the fundamental problem of aliasing and pixelization artifacts of targets located at close distances to the fiber, a result related to very recent reports in ISM microscopy [29].

2. Methods

2.1 Principle

The principle of the image-scanning fiber endoscopy is schematically shown in Fig. 1. Our approach is based on a conventional confocal endoscopy configuration (Fig. 1(a)). A focused beam scans the fiber cores at the proximal tip. The same fiber bundle collects the fluorescence signals. The proximal facet is imaged on a camera through a dichroic beam-splitter (DM) and a band-pass filter (F). At each scan position, the camera captures the fluorescence intensity distribution over the proximal facet (Fig. 1(b)). The sampling grid of the bundle cores is marked by white X’s in Fig. 1(b-d). The camera images allow performing both confocal imaging and ISM. For confocal endoscopy only the collected signal from the illumination core is integrated (Fig. 1(c)). For ISM, the entire camera image is scaled by a factor of $0.5$ [22] and saved (Fig. 1(d)). This reassignment procedure creates ’virtual’ cores at a new detection grid, with a spatial sampling frequency of $2 f_s$ that is two times larger than the original cores grid (Fig. 1(b)). The final ISM image is the summation of all reassigned images, having a denser spatial sampling grid and a higher collection efficiency.

Quantitatively, the point spread function (PSF) of confocal imaging, $h_{conf}(r)$, is the product of the illumination and the detection PSFs [30].

In the case of ISM, the effective PSF, $h_{ISM}$ is the convolution of the illumination and detection PSFs, scaled by a factor of half [20].

In the case of identical Gaussian profiles for illumination and detection PSFs, the final effective PSF of ISM is identical to the confocal. This result assumes an adequately Nyquist sampling of the collected signals. While this is conventionally attained in a conventional microscopy setup, it is not the case in bundle-based endoscopy. As a result, the doubling of the spatial sampling frequency in fiber-bundle ISM improves the final optical transfer function (OTF, the Fourier transform of the PSF), even in the case of identical detection and illumination PSFs since it avoids aliasing of spatial frequencies in the range $[f_s/2 f_s]$ (see Fig. 2).

 

Fig. 2. Simulated OTFs comparisons between confocal endoscopy (blue), ISM endoscopy (red), and confocal imaging without sampling cores (dashed green) (a,b) OTFs crosssection for an object placed at a distance of ${\mathrm{\mu} \mathrm{m}}$ from the distal tip: due to the spatial sampling of the MCF cores, endoscopic OTFs suffer from aliasing artifacts exhibited by the folding of high spatial frequencies content (arrows), compared to ideal confocal imaging. ISM-endoscopy (red) doubles the effective spatial sampling frequency from $f_s$ of confocal endoscopy (blue) to $2f_s$ (see Fig. 1(d), and the folding frequencies (dashed and dotted vertical lines, respectively). This allows correct retrieval of $\times 2$ higher spatial frequency content in the ideal unsampled confocal OTF (dashed green). (c-d) (c-d) Same as (a-b) for an object distance of $150 {\mathrm{\mu} \mathrm{m}}$. At larger object distances, the doubled sampling frequency of ISM endoscopy has a negligible impact due to the lower frequency content of the defocused object image. Nonetheless, the gain in signal collection efficiency of ISM improves the reconstructed images (see Fig. 3 and Supplement 1).

Download Full Size | PDF

3. Results

3.1 Numerical study of the optical transfer function

To study the expected improvement in the normalized OTF due to the increase in spatial sampling frequency of ISM fiber-bundle endoscopy, we numerically simulated the resulting images of a point target placed at distances of $60 {\mathrm{\mu} \mathrm{m}}$ and $150 {\mathrm{\mu} \mathrm{m}}$ for ISM and confocal fiber bundle endoscopy, and compared to the ideal confocal image sampled at a spatial frequency $\sim 6$-times that of the fiber bundle (i.e., without relevant aliasing). The OTF for each case was calculated by Fourier transforming the simulated image. The results of this study are presented in Fig. 2. For each distance, we studied the results when the point target is placed directly in front of a fiber core (Fig. 2(a),(c)) and when the point target is placed between two neighboring cores (Fig. 2(b),(d)).

As expected, the 1-D cross-sections of the OTFs in all cases show that aliased spatial frequencies, which result from undersampling, are present in the confocal images at frequencies higher than $f_s/2$ (blue curves and arrows). In contrast, in the ISM images, the aliased frequencies are only present at frequencies above $f_s$ (red curves). The doubling of the spatial sampling frequency is important at close distances (Fig. 2(a),(b)) since the captured image contains higher frequency content than at larger distances (Fig. 2(c),(d)), where the effective defocus blur due to the propagation low-pass filters the higher frequencies content. Importantly, as we show below (Fig. 3, Supplement 1), ISM endoscopy also improves resolution at longer distances due to the significantly improved collection efficiency and SNR.

 

Fig. 3. Experimental results comparing ISM and confocal endoscopy images of fluorescent beads. (a) Reference ’contact’ image of the fluorescence target when placed in contact with the distal fiber facet ($z_{obj}=0)$. (b) Confocal image of the same target when placed at a distance of $z_{obj}=60 {\mathrm{\mu} \mathrm{m}}$ from the distal tip. (c) ISM image from the same camera frames taken in (b) exhibiting improvements in resolution and SNR. (d) cross-sections of (a-c) over the dashed white line in (a) show the improvement in resolution of ISM endoscopy (red) over confocal endoscopy (blue). (e) Reference transmission microscope image of the target. (f,g) Same as (b,c) when the target is placed at a distance of $z_{obj}=150 {\mathrm{\mu} \mathrm{m}}$ from the distal tip. At such a large distance, the improvement in SNR of ISM endoscopy due to the higher collection efficiency is significant. (i) Experimental (black dots) and numerical (red trace) plots of the improvement in the signal collection as a function of object distance. The improvement is significant at larger distances due to the larger defocus blur in the raw image (Fig. 1(b)). Scale bars: $50 {\mathrm{\mu} \mathrm{m}}.$

Download Full Size | PDF

3.2 Experimental results

Our experimental setup is schematically depicted in Fig. 1(a), and the experimental results are summarized in Fig. 3. In all experiments, a $532\mathrm {nm}$ CW laser (Oxxius LCX-532S) illuminates a pair of scanning mirrors (Thorlabs GVS012), which are imaged on the back focal plane of an objective lens (OL, $20\times$, $0.4$ NA, Olympus) using a 4-f system. The galvanometric mirrors raster-scan the focused beam on the proximal facet of the MCF bundle (Schott P/N 1563385), having an effective imaging area of $0.45 mm$ diameter, containing $\sim 5,600$ cores, with an average core diameter of $4.1 {\mathrm{\mu} \mathrm{m}}$ and an average core-to-core distance of $6.5 {\mathrm{\mu} \mathrm{m}}$. The fluorescence signal is filtered by a dichroic mirror (DM) and band-pass filters (F, Semrock BrightLine FF01-576/10-25, and Thorlabs FEL0550), and the descanned image is formed on an sCMOS camera (Andor Zyla 4.2). The target object is composed of Nile Red fluorescent beads (Spherotech), averaging 6${\mathrm{\mu} \mathrm{m}}$ in diameter, dispersed on a cover glass. A motorized actuator (Thorlabs Z825B) controls the distance between the fiber and the sample.

The ISM endoscopic image is formed in an identical fashion to conventional ISM image formation [20]. Specifically, each captured descanned frame is rescaled by a factor of $1/2$ (Fig. 1(d)), and its center is shifted to the illumination position. The scaled and shifted frames are summed to form the final ISM image.

To demonstrate the spatial resolution and SNR improvements of ISM over confocal microendoscopy, we image the target at different distances from the fiber distal facet to the target. First, we image the sample at a contact distance ($z_{obj}=0 {\mathrm{\mu} \mathrm{m}}$) to establish the ground truth for the experimental measurements (Fig. 3(a)). Next, we position the target at a distance of $60 {\mathrm{\mu} \mathrm{m}}$, where we observe an improvement in spatial resolution for the ISM-endoscopy image over the confocal image (Fig. 3(b),(c)), due to the higher sampling frequency (Fig. 2(a)-(b), Fig. 1(d)). A cross-section of the three images (Fig. 3(d)) shows the improvement in resolution, resolving the beads that compose the target at this cross-section, as validated in a transmission microscope image of the sample (Fig. 3(e)). At a larger distance of $150 {\mathrm{\mu} \mathrm{m}}$ (Fig. 3(f),(g)), the ISM image displays a significant improvement in SNR over the confocal image due to the higher collection efficiency, leading to a resolution improvement that does not originate from the higher spatial sampling frequency (Fig. 2(c),(d), Supplement 1). Figure 3(i) displays the experimental improvement in collection efficiency (black dots) measured by averaging the fiber cores that image sufficiently bright beads as a function of target distance, together with numerically simulated results. The full information on the numerical simulation is given in Supplement 1.

All displayed ISM, and confocal images are Fourier-interpolated using a Hann filter to eliminate the cores pixelization. Since the ISM image has a $\times 2$ denser hexagonal grid of ’virtual’ cores (Fig. 1(d)), the bandwidth of the Hann filter, set by the inverse of the core-to-core pitch, is doubled in the ISM images, compared to the filter used in the confocal image. We note that the hexagonal sampling grid provides superior sampling efficiency compared to the standard rectilinear grid [31].

4. Discussion

In this work, we demonstrated a lensless micro-endoscopy approach that adapts the principles of image-scanning microscopy (ISM) to fiber-bundle endomicroscopy. Our proposed configuration utilizes off-the-shelf components and is based on a conventional confocal fiber-bundle endoscope with a camera replacing the confocal detector. By implementing ISM-endoscopy, we demonstrate an improved collection efficiency of up to a factor of approximately $18$ at an object distance of $150 {\mathrm{\mu} \mathrm{m}}$, and an enhanced spatial resolution compared to confocal endoscopy. We have applied ISM to lensless endoscopy using bare fiber bundles. However, the technique can be directly applied to endoscopes employing imaging distal GRIN lenses, improving the resolution and collection efficiency for objects situated outside the depth of field.

While the improvement of spatial resolution and collection efficiency in ISM has been extensively studied [20,22], we highlight an additional advantage of ISM for spatially undersampled systems such as MCF bundles: doubling of the effective spatial sampling frequency, which doubles the accessible spatial frequency bandwidth of the imaging system.

ISM endoscopy can be directly applied to any fiber bundle endoscope, provided that core-to-core cross-talk is negligible. In addition, the exact doubling of the spatial sampling frequency assumes a periodic ordered cores lattice. Nonetheless, non-periodic MCFs with non-negligible crosstalk can also be directly used at long enough working distances. The potential effects of non-perfectly periodic core arrangements may be interesting for future studies. We note that the exact collection improvement and resolution enhancements are dependent on the MCF numerical apertures (NA) and core-to-core pitch. Importantly, the optimal number of reassigned neighboring cores depends on the object distance and fiber NA, where a larger number of cores collect a useful signal at a larger object distance, and higher NA [32]. Combining ISM MCF-based endoscopy with advanced techniques that rely on wavefront shaping for focus-scanning and wavefront correction of fiber distortion [19,26] may be interesting for future studies.

In our proof-of-principle demonstration of ISM endoscopy, we used a conventional camera as the detector array, which limited the acquisition speed compared to conventional confocal endoscopy employing a single detector. However, this technical limitation can be removed by either utilizing an all-optical realization of ISM [33] that provides a full-field ISM image from a single camera frame. Alternatively, the limitation can be overcome by using a high-speed low-pixel count detector array, currently commercially utilized in ISM microscopy [34]. At larger object distances, it may be interesting to study sparse and/or compressive measurements to reduce the total number of illuminations for creating an image.

Lastly, we note that our ISM-based endoscope may enable label-free reflection endomicroscopy, which is challenging to perform in confocal detection. This is because, in reflection endomicroscopy, the illumination and detection wavelengths are identical. Detecting the weak reflected target signals in the presence of relatively-strong background reflections from the MCF facets is a great challenge [35,36]. However, applying ISM while digitally masking the illumination core allows the reconstruction of target images from the signals collected in the neighboring unilluminated cores.