1. Introduction

Holographic wavefront shaping has allowed the development of multiple imaging techniques through complex media [1–3]. This has given rise to a sizable interest in potentially using these imaging methods for biological applications where there is a need to obtain high resolution light microscopy images at depths of more than few millimeters. A particularly impressive development has been the use of multimode optical fibers as ultra-thin endoscopes [4,5]. Their small footprint has allowed imaging deep within tissue with minimal damage and preserved structural characteristics and function. Furthermore, multiple microcopy modalities have been demonstrated [6–9] opening the road to a wide array of applications.

Particle tracking gives us the ability to locate and follow single molecules and nanoparticles with sub-diffraction limited accuracy [10,11]. It has found a wide range of applications in bio imaging as well as in other fields, such as optical trapping, and has been used to investigate, for example, the movement of molecular motors [12], motion of receptors [13] and uptake pathways for pharmaceutical and viral agents [14]. Tracking can be performed by detecting scattered light or, more commonly, fluorescence from a target. In bio-imaging applications the most common type of tracking is based on fast wide-field fluorescence imaging and relies on fitting a Gaussian function to the diffraction limited image of a single particle to obtain a more exact estimate of the particle’s location [15]. Unfortunately, while wide-field imaging through a MMF has been demonstrated, it has also been shown to be slow [16] and highly inefficient [17], making its use in bio-imaging, especially for tracking, impossible in its current implementation.

Orbital particle tracking (OrPT) allows determining the position of a target, such as a single fluorophore, fluorescent particle or highly scattering particle [18–21], without using wide field imaging. It is a laser-scanning technique that consists of moving a spot in a circle, around the particle (target) located at the imaging plane. Depending on the target chosen, light emitted, transmitted or scattered will be intensity modulated when the target is located off-center, and using a feedback loop this signal can be used to have the orbit track the target. OrPT has been shown to obtain tracking accuracies that range from a few nm to a few tens of nm [22–24]. This method, like other localization techniques, allows pinpointing the location of a particle far beyond the diffraction limit, in principle being limited solely by the signal-to-noise ratio [25,26].

In this work we show, for the first time, a discrete implementation of 2D OrPT at the tip of a MMF. We test the method using a pinhole mounted on a nanopositioning stage as the target and measure in transmission. We achieve a static tracking accuracy (on the order of 20 nm) and a speed that are sufficient for biologically relevant tracking tasks [24].

We discuss the adaptations needed for the method to work through a MMF and to overcome issues intrinsic to MMFs such as the uneven intensity of the foci due to the variation in diffraction efficiency for the gratings on the SLM.

We also note that particle tracking can be used to map out the shape of a space with sub-diffraction limited resolution, e.g. for mapping the extracellular space in the brain [27]. This is potentially even more interesting for MMF imaging, since the maximum NA of commercially available fibers is quite low. Thus, implementing methods that offer improved spatial resolution is of great interest.

2. Methods

Figure 1 shows a simplified schematic diagram of the experimental setup. A CW laser operating at $\lambda = 1064\; \textrm{nm}$ (NKT Koheras Boostik) was used as the light source, the beam was expanded to completely illuminate the active area of a liquid crystal on silicon (LCoS) Meadowlark HSPDM512 spatial light modulator (SLM) set in an off-axis configuration. A calcite beam displacer (Thorlabs BD40) located in the Fourier plane to the SLM enabled us to control two orthogonal polarization states [4]. This Fourier plane is then projected to the proximal fiber facet using a 200 mm lens and a microscope objective (20x Olympus Plan 0.4NA). While initial tests were carried out using a step index MMF, we found that using a graded index (GRIN) MMF was a better choice due to their robust response to bending [28] and a more uniform spot intensity distribution across the fiber facet. The fiber used was 3 cm long and had a NA of 0.3. Full control of the fiber input polarization allows us to maximize the power ratio and is of particular importance when working with graded-index (GRIN) MMF since they do not maintain polarization [29], to achieve this we use a polarizing beam displacer, which allows us to address two orthogonal polarization states as two sets of positions in the plane of the input facet, further details can be found in [4].

 

Fig. 1. a) Experimental setup. PBD-polarizing beam displacer, MO-Microscope objective, HWP-Half wave plate, QWP- Quarter wave plate. Insert: imaging plane 30$\mu m$ in-front of the distal fiber facet with a 1$\mu m$ pinhole on a nanopositioning stage. The pinhole is removed during calibration. Camera 1 is used to observe and quantify the light launched into the fiber, Camera 2 is used for imaging the imaging plane during the calibration procedure and as a bucket detector during experiments. b) Principle of discrete OrPT showing two pinhole positions, off-center (${\boldsymbol{T}} = {\boldsymbol{T}^{\prime}}$) and centered (${\boldsymbol{T}} = {\boldsymbol{O}}$). Three different spot locations along the orbit are shown (${{\boldsymbol{S}}_0},{{\boldsymbol{S}}_1},{{\boldsymbol{S}}_2}$), the plot illustrates the transmitted intensity for each spot.

Download Full Size | PDF

2.1 Calibration

The first step for performing OrPT using a MMF is characterizing light transport through the fiber in a process we refer to as calibration [4,5,30]. During calibration the SLM is used to sequentially project a set of tightly focused spots, arranged in a grid-like pattern, onto the core of the proximal fiber facet. The points used are selected by raster scanning a point across the input facet and selecting those that have a coupling efficiency of over 20%. This procedure is intended to determine the required complex amplitude relationship between these input foci to obtain constructive interference in a single position on the imaging plane. Each of these input points, or modes, corresponds to displaying a blazed grating on the SLM. The speckle pattern seen at the imaging plane, in our case $30\; \mu m$ in front of the output facet of the fiber, is imaged and interfered with a reference beam. Four phase shifts over a range of $2\pi $ are applied to each grating allowing us to determine the output amplitude and phase for each input mode. The procedure is repeated for each orthogonal input polarization. Approximately 1700 individual input positions per input polarization are used with the full procedure taking between 15 to 20 minutes with the SLM used here.

The result of this procedure is a transmission matrix, in which each pixel in the imaging plane becomes an output mode where a near diffraction limited spot [31] can be formed by combining all the input modes with the appropriate phase, so that constructive interference at the desired position results in a diffraction limited spot that contains, in our case, up to 70% of the output power. The spacing between the output modes in our setup was ${\sim} 0.21$ $\mu m$ resulting in over 44,000 output modes. For our 0.3 NA MMF we measured the spot’s $FWHM = 1.7 \pm 0.1\; \mu m$ in the central region of the fiber core (essentially diffraction limited).

2.2 Tracking

Tracking was done using a discrete implementation of OrPT where, instead of displaying a full continuous orbit, we successively displayed a series of n discrete spots, ${{\boldsymbol{S}}_1},{{\boldsymbol{S}}_2}, \ldots ,{{\boldsymbol{S}}_{\boldsymbol{n}}}$, on a circle with radius $\rho $, centered at ${\boldsymbol{O}}$, and with a constant angular separation $\delta \phi $ so that ${{\boldsymbol{S}}_{\boldsymbol{n}}} = \rho \angle ({n \times \delta \phi } )= \rho \angle \phi $. $\delta \phi $ is simply obtained by dividing the circular orbit into the number of spots desired, n, so that $\delta \phi = {\raise0.7ex\hbox{${2\pi }$} \!\mathord{\left/ {\vphantom {{2\pi } n}}\right.}\!\lower0.7ex\hbox{$n$}}$. In our setup we’ve set $\rho = 3.5\; pixels = 0.74\mu m$, which was approximately the standard deviation, $\sigma $, of a Gaussian fit to the spot intensity profile. Other OrPT experiments [22,24] commonly use $\rho = FWHM/2$, 0.85 $\mu m$ in our case, however, we found that in our system tighter orbits resulted in increased spot homogeneity and improved performance despite a decrease in modulation depth. The minimum number of spots needed to acquire the location of the target in a plane is three.

To accurately place each spot, which is critical for reaching a high accuracy in the tracking, and avoid issues that stem from pixelation from the camera used in the calibration procedure, we simultaneously project multiple output modes contained within a radius of $\sigma $ from the intended location, each mode is then weighted using a Gaussian distribution centered at said location. This approach allows us to reach subpixel precision. We measured the accuracy of spot positioning by fitting and locating a Gaussian distribution to images of 150 spots at 3 different angular positions resulting in the standard deviation for $\rho < 20\; nm$ and for $\delta \phi < 0.034\; rad$. Considering that the pixel size is ${\sim} 210\; nm$ this constitutes a significant improvement.

We performed the experiment using a ${\sim} \emptyset 1\; \mu m$ pinhole as our target. The pinhole was made on a microscope slide, using electron-beam lithography and a thin (tens of nm) metalized layer, and was mounted on a nanopositioning stage (Mad City Labs Inc. Nano-LP300). The transmitted intensity for each spot, ${I_n}$, was recorded using the calibration camera as a bucket detector. In a biological experiment the tracked particle would most likely be a fluorescent particle epi-detected through the MMF using a PMT. However, from the perspective of the tracking system the only actual difference would be the signal level, making this a suitable test and development platform.

In the case of a point-like target, ${\boldsymbol{T}}$, and under a Gaussian approximation and the transmitted intensity, ${I_n}$, is

To locate the target, we use the error signal given by:

The error signal will be equal to zero when ${\boldsymbol{T}} = {\boldsymbol{O}}$. So, the position of the pinhole can be tracked by progressively shifting ${\boldsymbol{O}}$ in the direction $\alpha $ and repeating the measurement with the objective of minimizing $\varepsilon $. In our case we have done this by using a fuzzy control loop implemented in LabVIEW 2018.

Two important system parameters are accuracy and tracking speed. In our system they are not independent from each other given that the controller may be tuned for a fast response time at the expense of accuracy and vice versa, allowing us to adapt it to the experimental situation. The tuning of the controller was done empirically by adjusting the input and output gains depending on the desired response. Additionally, the speed at which we can display a full orbit limits the system response time. While the SLM used here is rated by the manufacturer with a response time of approximately 17 ms, even minor ghosting from previous holograms introduces synchronous noise, so the waiting time between holograms was set to 35 ms. Therefore, to maximize the tracking speed, keeping the number of spots per cycle as low as possible is highly desirable, when the absolute minimum of 3 spots is used we can achieve ${\sim} $9.5 orbits per second.

3. Results and discussion

The first test was to determine the systems static accuracy. For this, we first tuned the controller for a slow response time and then we moved the pinhole in 100 nm increments both in the x and y directions, the total number of steps was $m = 9$. For each step, we allowed enough time for the controller to settle and then averaged the retrieved position, ${\hat{{\boldsymbol{r}}}_k}$, for 2 seconds (see Fig. 2). We repeated the same procedure using $n = 3$, $5$ and $7$. Afterwards we calculated the mean absolute error across all the positions,

 

Fig. 2. Static positioning accuracy in x and y. Red diamonds show the 2 second average position given by the tracking system, the error bars represent the standard deviation. The target position is given by the encoder from the nanopositioning stage and is represented by gray open circles.

Download Full Size | PDF

An important aspect to consider is that in order to achieve $|{\boldsymbol{\varepsilon }_{\boldsymbol{T}} = {\boldsymbol{O}}} |= 0$ the intensity of the n spots should be the same. However, the spot intensity across the fiber facet is known to vary [32,33] and while these variations tend to be small, and usually of little consequence to qualitative imaging, in our case they can introduce a systematic error. The variations stem partially from differences in the diffraction efficiency between different SLM holograms so that the total amount of light launched through the fiber varies. To mitigate this in our experiment, the fiber input was monitored using a camera to measure the intensity of the light sent to the input and thus estimate the intensity of light being sent through the fiber. This was then used to compensate the tracking signal intensity prior to calculating $\boldsymbol{\varepsilon }$.

The calculated mean absolute error is shown in Table 1, where we see that the system was able to locate the target with an accuracy of tens of nanometers, a figure comparable to other reports in the literature on particle tracking [34,35].

Table 1. Calculated mean absolute error.

View Table

Another possible error source is the deterioration in spot quality seen near the edges of the fiber core. Tracking in this region is therefore avoided, leaving us with ${\sim} 80\%$ of the field of view were the spot quality has been shown to be optimal [29]. Nonetheless, fluctuations arising from the nature of the light transport through the MMF are unavoidable and form a part of the system noise. Yet, no dependence between position in the imaging plane and accuracy was observed within this region. Additionally, the fiber’s distal end is not held in place but can move freely, effectively making the fiber a cantilever, and consequently, the system can be very sensitive to vibrations. To reduce this sensitivity, during calibration and throughout the static experiments we used a drop of water to ‘bind’ the fiber tip to the coverslip, the surface tension of the water droplet is sufficient to reduce the cantilever effect. However, this is avoided in following experiments where the stage is moved since this would result in the fiber moving as well.

The tracking speed of the system was evaluated by having it follow a spiral trajectory with a constant tangential acceleration. For this dynamic test, the controller was tuned for a fast response and the system was set to follow the pinhole until the error reaches a predefined threshold. By employing a spiral trajectory, rather than a straight one, a continuous change in direction is obtained which results in a more stringent test and allows us to better visualize the moment where tracking has failed. The results can be seen for $n = 3$, $5$ and $7$ spots per cycle in Fig. 3(a), where we can see the trajectory followed by the target and the one reproduced by the tracking system. In Fig. 3(b) $|\boldsymbol{\xi } |$ is plotted against the tangential speed, $|{Vt} |$. $|\boldsymbol{\xi } |= \rho $ was used as the threshold for determining when a target has been lost. Beyond this point the target would be positioned outside the orbit and the performance of the controller begins to decrease.

 

Fig. 3. Tracking of a spiral trajectory. a) Scatter plot of the tracked position (circles) and the target position (colored line), the color scale shows temporal evolution. b) Tangential speed, $|{{V_T}} |$, vs $|\boldsymbol{\xi } |$, for n = 3 (red dots), n = 5 (blue x) and n = 7 (black +). The continuous lines show the moving average for 30 points. The insert show the moving average from 0.1 to 0.3 $\mu m/s$

Download Full Size | PDF

Figure 3 shows that we could obtain a tracking speed of up to ${\sim} 1.2\; \mu m/s$ when using $n = 3$, this is on par with the speed of multiple motor proteins, e.g. cytoplasmic dynein ($1.1\; \mathrm{\mu m}/\textrm{s}$) [36]. In the insert in Fig. 3(b) we once again see the reduction in the error when using a higher n, although it is important to note that this error involves a lag between target and the tracked trajectory, so it will always be higher than the static error.

With the previous results in mind and to test the performance in a more realistic scenario we let the stage move along a trajectory mimicking Brownian motion (Fig. 4(a) and (b)) and hop diffusion (Fig. 4(c)). We produced the trajectories using a random walk algorithm implemented in MATLAB R2018a. For the measurements shown in Fig. 4 a and b the controller was tuned for speed with $n = 3$ and a Brownian trajectory was simulated using a diffusivity, D, of $0.09\mu {m^2}/s$ for Fig. 4(a) and $0.3\mu {m^2}/s$ for Fig. 4(b). We can clearly see that the tracking system was able to replicate the trajectory of the pinhole over tens of micrometers.

 

Fig. 4. Tracking of diffusion trajectories. a) and b) Brownian diffusion with n = 3, c) hop diffusion with n = 5.

Download Full Size | PDF

For the measurements shown in Fig. 4(c), we set $n = 5$ with the controller balanced between precision and speed. Here, we simulated the trajectory of a particle trapped in a well where it is free to diffuse following a Brownian trajectory. The effective diffusivity was of $0.0035\; \mu {m^2}/s$, this low diffusivity value is due to the well confinement. The inter-well distance was 250 nm and the well walls were simulated using a Gaussian profile with a standard deviation of 10 $\textrm{nm}$. Effectively, a jump to a neighboring well occurs every $0.26\; s$ on average. The spread of the tracked position $\hat{{\boldsymbol{r}}}$ within the well is clearly less than the spread in the trajectory positions ${\boldsymbol{r}}$. This is due to the response time of the tracking system [11]; a faster system would be able to resolve the confining walls better. Nonetheless, it is clear that the resulting track can be used to obtain parameters such as inter-well distance, distinguish between Brownian and non-Brownian diffusion and obtaining the effective diffusivity.

Figure 5 shows the mean square displacement for the points ${{\boldsymbol{p}}_k}$ along the trajectories seen in Fig. 4,

 

Fig. 5. Mean squared displacement vs. time interval for the stage encoder and the tracked trajectory, corresponding to the trajectories seen in Fig. 4. The dashed lines represent the best linear least-squares fit of Eq. (6) from $\tau = 0\;to\;\tau = 5\;s$.

Download Full Size | PDF

Hop diffusion will appear as Brownian motion for long time intervals [11] meaning that an effective diffusivity can be obtained from Eq. (6). In Fig. 5(c), we see good agreement between the effective diffusivities calculated for the encoder and for the tracker. Nonetheless, the offset between the two curves is noticeably different, this is once again due to that the reaction time of the tracking system is too slow to reproduce the movements within the well correctly and is to be expected [11].

Although we have here demonstrated 2D tracking, which is sufficient for a number of applications [34], OrPT can be extended to 3D tracking. However, the limited NA of the fiber results in a very long PSF in the direction of beam propagation, z. This makes tracking using changes in intensity at different focal planes impractical. Possible workarounds to this problem would be using 2-photon excitation to improve the sectioning [24] or using a parallax measurement [38] by introducing an angle to the wavefront.

The tracking speed would scale up with faster response times of the phase modulator, and thus, switching from the SLM to a digital micro-mirror device (DMD), with a response time in the order of tens of microseconds, would deliver a tracking speed over an order of magnitude higher, without any changes to the method presented here. The increased sampling speed would also allow us to use a larger number of spots per orbit and to further process the signal to suppress noise and obtain higher accuracy. Nonetheless, LCoS SLMs remain necessary in MMF imaging systems where efficiency or optical dispersion are of concern [39], e.g. when working with fs lasers as the excitation source. Faster LCoS SLMs are available on the market and could also be considered for future implementations.

Furthermore, the possibility of using a square-core MMF to reduce the variation of the spot intensity across the fiber facet could yield an improvement in accuracy, although this would result in a very high sensitivity to bending [32]. The accuracy will also likely improve when using a shorter excitation wavelength, as the smaller orbit at shorter wavelengths result in more even spot intensities.

4. Conclusion

We have shown that the position of a moving target can be tracked using a MMF micro-endoscope with accuracy down to ${\sim} \lambda /50$ using a discrete implementation of OrPT. Our systems tracking accuracy is sufficient for various biological applications such as the study of cellular uptake of nanoparticles [24]. Despite using a slow LCoS SLM, the method can track movement up to a speed of 1.2 um/s, (which is on par with the speed of a number of motor proteins [36]), and we demonstrated that we can accurately measure the diffusion constants in Brownian motion with diffusivity up to 0.3 $\mu {m^2}/s$, opening up for interesting applications such as tracking at depth in brain with minimal damage thanks to the small diameter of the endoscope. Higher tracking speed could be achieved by replacing the SLM with a fast DMD. This technique can be used to complement any of the imaging modalities already available through MMF endoscopes, giving an image of the environment of the tracked particle simultaneously with the tracking.