1. Introduction

Benefiting from its nanometer-scale image resolution, single molecule localization microscopy (SMLM) has been widely used for visualizing molecule details in biological system [1,2]. The high spatial resolution in SMLM is achieved by localizing the randomly and sparsely activated fluorophores in thousands or even tens of thousands of raw images, leading to an intrinsic trade-off between spatial and temporal resolution [3,4]. This trade-off impedes the wide applications of SMLM to volumetric imaging [5–7], as well as high-throughput and high-content screening imaging [8–10], where hundreds of field-of-views (FOVs) need to be imaged under different imaging conditions.

To circumvent this trade-off, researchers had made a significant amount of efforts to optimize the imaging throughput of SMLM, including high-density localization [11–13], fast imaging using scientific complementary metal–oxide–semiconductor (sCMOS) camera [14,15], imaging system automation [16,17] and deep-learning analysis [18–20]. However, these strategies were usually designed without using the “sweet spot” in data acquisition, where sufficient image resolution and high image throughput are carefully balanced. The main reason is that there lacks a real-time image resolution measurement method.

Currently, there are mainly two kinds of methods for measuring the image resolution in SMLM: one is based on a reconstructed super-resolution image (image-based resolution measurement), and the other is based on the localization coordinates used to reconstruct a super-resolution image (coordinate-based resolution measurement). Actually, these methods originate from the last two steps in the entire SMLM process: single molecule localization, super-resolution image reconstruction. Image-based resolution measurement method, including the popular Rayleigh resolution and Fourier ring correlation (FRC) resolution, utilizes existing tools from other microscopy techniques. Specifically, Rayleigh resolution, which is designed to imitate Rayleigh criteria in conventional optical microscopy, is determined by peak signal separation using intensity profile of local fine structure [21]. The FRC resolution [22,23], which is adapted from electron microscopy, calculates the spectral signal-to-noise (SNR) ratio of the cross-correlation of two Fourier transformed super-resolution images and uses a SNR threshold to determine the highest structural frequency that can be resolved in the image. On the other hand, coordinate-based resolution measurement method includes mainly the heuristic resolution [3,21], which calculates the overall image resolution by directly taking the square root of single molecule localization uncertainty (localization precision) and Nyquist resolution (determined by localization density [24]). Compared to image-based resolution measurement, coordinate-based resolution measurement has the benefits of exploiting high precision localization information [25–27].

However, real-time image resolution measurement for SMLM is still challenging, because the existing measurements are determined by many factors that are only accessible after obtaining the whole localization dataset, including underlying sample structural information, sample drift and labeling density [22,28–30]. Rayleigh resolution requires subjective selection of region-of-interest (ROI) based on overall sample information. FRC resolution is affected by structure shape and sample drift during data acquisition [21,23,28]. Moreover, FRC resolution measurement during data acquisition will significantly lag behind camera image acquisition when applied to fast and large FOV imaging due to the time-consuming Fourier transforms of large super-resolution images. Lastly, heuristic resolution can’t be performed in real-time due to the difficulties in real-time calculation of localization density (Supplement 1).

Here, we present a real-time image resolution measurement method for SMLM. This method is termed ROMP for “real-time image resolution measurement based on Poisson random sampling”. ROMP is a coordinate-based image resolution measurement method developed under the framework of heuristic resolution. In this method, real-time localization density calculation is realized through modeling single molecule random activation as a spatial Poisson process. This means that it is possible to estimate the images resolution in real time. After combining a newly developed algorithm for real-time single molecule localization QC-STORM [12], we achieve real-time image resolution measurement and demonstrate this method is effective to guarantee homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

2. Methods

2.1 Poisson modeling of random molecule activation

Nyquist resolution defines how well a structure can be represented in the concept of Nyquist-Shannon sampling theorem. Practically, the Nyquist resolution is usually calculated as ${\rm{2}}\bar r$, where 2 is a constant for satisfying the Nyquist sampling criterion and $\bar r$ is the mean nearest neighboring distance between localization events in the whole localization dataset [24]. Because a typical localization dataset contains hundreds of thousands of localizations, it is computationally expensive to calculate the ground-truth of $\bar r$. Instead, $\bar r$ is usually approximated by ${{\rm{d}}^{\;{\rm{ - }}\frac{{\rm{1}}}{{\rm{D}}}}}$, where D is imaging dimension and d is localization density [24].

In 2D imaging, the localization density d, indicates that there is an average of d molecules per square micrometer. By assuming that the molecule activation is a spatial Poisson process, d is equivalent to the event rate in Poisson process. Then for each molecule, the probability that there are n neighboring molecules presenting in its surrounding area with a specific radius of w is:

Considering the scenario that the distance between a molecule and its nearest neighbor, denoted by r, is larger than w, which is equivalent to that no other molecules appear in its surrounding area with a radius of w, the probability becomes:

Therefore, we can get the probability that the distance r is not more than w and the cumulative probability distribution for r is:

Then, according to the mathematical relationship between distribution function and probability density function, the probability density function for r at w is derived as:

Finally, the mean nearest neighboring distance ($\bar r$) is calculated by:

The distribution of nearest neighboring distances generated from Poisson process shows that only half of molecules have sampling frequency higher than the expected value (Fig. S2(a)), indicating that Nyquist resolution overestimates image resolution. To eliminate this overestimation, we adopted the oversampling approach [21] to enable sufficient sampling frequency. Considering N-fold oversampling, i.e., accumulating N-fold localization density, the percentage of molecules having smaller nearest neighboring distance than the mean value calculated before oversampling is:

With an oversampling factor of 4, the ratio of molecules with sampling frequency higher than the expected value could be significantly increased to 95% (Supplement 2, Fig. S2). With a higher oversampling factor, this ratio would have a limited increase, but at the expense of a significant reduction in the temporal resolution. Thus, in this paper the Nyquist resolution is calculated with an oversampling ratio of 4. Mathematically, applying N-fold oversampling over localization density d is equivalent to increase sampling interval by $\sqrt {\rm{N}}$ times. Therefore, we calculate the oversampled Nyquist resolution as:

In practice, the localizations are not completely random in the whole 2D space. Instead, they will distribute within the underlying structure, except for those freely diffuse or non-specific molecules. This prior knowledge indicates that the dimension D can be fractional, and it is possible to simultaneously estimate both dimension and localization density. To do this, we assume a small number of raw image frames (m) in a sequentially captured time interval, where the accumulated localization density increases linearly with the frame number. The mean nearest neighboring distance as a function of m can be described as (for the detailed derivation process, please refer to Supplement 2):

Then, the dimension and the localization density can be fitted from the mean nearest neighboring distances. By dividing the whole acquisition into short time intervals, the localization density variation among these time intervals can be acquired, and the final Nyquist resolution can be calculated by accumulating the localization densities in these time intervals. In practice, the oversampled Nyquist resolution calculated from Eq. (7) is also refined using integer dimensionality when dense localizations have been accumulated (Supplement 2).

2.2 Calculation of ROMP resolution

Under the framework of heuristic resolution, our ROMP method is readily available by combining the oversampled Nyquist resolution with the localization precision calculated by CRLB [31]. To emphasizing the real-time capacity of our ROMP method, the heuristic resolution calculated by our ROMP method is called ROMP resolution.

The ROMP resolution is calculated by the square root of oversampled Nyquist resolution defined by Eq. (7) and the localization precision described by full width half maximum (FWHM) of single molecule localization uncertainty:

The method we proposed before, called QC-STORM, was used for real-time molecule localization [12]. During the data acquisition process, the localization precision of each localization is calculated as 2.35 times the CRLB [31].

2.3 Calculating FRC resolution

FRC resolution was calculated using the codes in the original paper [22]. Before calculation, the localizations were drift-corrected and filtered by the same method used in ROMP resolution calculation. The computation times were measured on a Dell precision T7610 workstation, which has two Intel Xeon E5-2630 v2@ 2.6 GHz CPU with 56 GB memory, and one NVIDIA TITAN Xp graphics card with 12 GB memory.

2.4 Imaging system

SMLM experiments were performed on an Olympus IX73 inverted optical microscope. A 640 nm and a 405 nm laser were combined and coupled into a customized fiber combiner [32], and the fiber output was collimated and focused onto the sample using a high-numerical-aperture (NA) oil-immersion objective (100×, NA1.4, Olympus) to form homogeneous flat-field illumination [32]. For mosaic localization microscopy, the rounded square output fiber (WF 200 × 200/252/440N, NA= 0.22, Ceramoptec, Latvia) of the fiber combiner was replaced by a non-rounded square fiber (WF 200 × 200/230 × 230/440/620/1100N, NA = 0.22, Ceramoptec, Latvia) to achieve a square illumination light field. Alexa Fluor 647 or CF 680 labeled samples were soaked with standard STORM buffer (50 mM Tris, pH 8.0,10 mM NaCl, 10% glucose, 100 mM mercaptoethylamine, 500 µg/mL glucose oxidase, 40 µg/mL catalase) and imaged with the 640 nm laser at an intensity of 6 kW·cm^-2. Images were acquired with an sCMOS camera (Flash 4.0 v3, Hamamatsu Photonics) at a frame rate of 50 Hz and a pixel size of 108 nm.

The real time analysis function of our QC-STORM plug-in was further used to provide hardware feedback control. Firstly, the 405 nm activation laser was controlled according to a predetermined localization density. QC-STORM provides two different approaches for calculating localization density: the first one is calculated from each time interval using random sampling and fractional dimension, and the second is calculated by the percentage of molecules with no neighbor molecule within 1 µm radius (Supplement 2). The second approach was used to control the laser intensity because it can be fast updated with only few images.

The localization density feedback was based on PID control, which calculates the error between the measured value and the predetermined value. The laser power regulating valve was determined by proportional, integral and derivative coefficients. For simplicity, the derivative coefficient was set to 0, and the proportional and integral coefficients were acquired by preliminary experiments to enable both stable and quick response. To avoid oscillations of localization density, the laser power only be regulated when the measured density is lower than predetermined value.

A closed-loop focus lock system using the signal of near-infrared laser totally reflected by the coverslip was adopted to maintain the focus during single-FOV imaging [3]. In mosaic SMLM, this method was also used to correct the z drift after FOV movement. Then, the mean SNR from real-time molecule localization was used to locate the best focal plane of the new FOV automatically. An automatic XYZ piezo stage with closed-loop resolution of less than 1 nm (P-545.3C7, Physik Instrumente) was utilized for active focus stabilization.

2.5 Sample preparation

U-2 OS cells were grown in McCoy's 5A (Modified) Medium (Gibco) supplemented with 10% fetal bovine serum (FBS, Gibco) and 1% penicillin (10,000 IU/mL)/streptomycin (10,000 µg/mL) at 37 °C with 5% CO2. Before imaging, cells were seeded on 35-mm glass-bottom dishes overnight. For microtubule labeling, cells were washed three times with PBS prewarmed to 37 °C and fixed with 3% paraformaldehyde, 0.05% glutaraldehyde and 0.2% Triton X-100 diluted in PBS for 15 min. Then cells were washed with PBS three times, permeabilized and blocked with blocking buffer (3% BSA (Jackson) and 0.05% Triton X-100 in PBS) for 1 h with gentle rocking. Cells were incubated with mouse monoclonal anti–α-tubulin antibody (Sigma T5168, 1:500 diluted in blocking buffer) at room temperature for 1.5 h. After washed with blocking buffer for three times, cells were incubated with Alexa Fluor 647 goat anti-mouse IgG (Invitrogen A-21236) at a concentration of approximately 10 µg/mL for 1 h. Samples were washed three times with PBS and stored in PBS at 4 °C until imaging. For NPCs labeling, rabbit anti-NUP133 primary antibody (PA5-63774, Invitrogen) and CF 680 labeled donkey anti-rabbit IgG second antibody (20820, Biotium) were used.

3. Results

3.1 Estimating localization density from the spatial random Poisson process

To validate the accuracy of our Poisson random sampling method in estimating localization density, we compare the estimated $\bar r$ with the ground-truth. As shown in Fig. 1, for an experimental microtubule dataset, the estimated $\bar r$ from our method fully agrees with the underlying ground-truth, while the conventional method (dividing localization number by structure area) cannot completely follow the underlying ground-truth.

 

Fig. 1. Localization density measurement of experimental microtubule imaging. (a) 2D imaging of immunofluorescent labeled microtubule of U-2 OS cell. (b) The relationship between frame number and the mean nearest neighbor distances calculated from different approaches: the whole localization dataset (ground-truth), our Poisson random sampling method, and the traditional localization/area dividing method. An intensity threshold of 100 photons is set for the localization/area dividing method.

Download Full Size | PDF

Then, the performance of our method in anti-background is measured with the open tubulin I dataset [33]. Localizations obtained from 200 consecutive raw image frames were used for simulation. And, the background localizations were randomly scattered in the whole FOV. As shown in Fig. 2, the nonspecific or diffused background localizations were filtered and thus do not affect our calculation of $\bar r$.

 

Fig. 2. The effect of background filtering on the calculation of mean nearest neighboring distance. (a-b) Different number of simulated background localizations (red dots) were added to the open Tubulin I dataset (blue lines). (c) Mean nearest neighboring distances under different levels of background noises. Poisson random sampling was used.

Download Full Size | PDF

3.2 Real-time image resolution measurement by ROMP

We measure the capability of our ROMP method in accurately measuring image resolution in real-time. For immunofluorescent labeled microtubule structure of U-2 OS cell (Fig. 1(a)), ROMP resolution increases with accumulated images and gives a final resolution of 27.9 nm, while FRC resolution gives a final resolution value of 35.6 nm (Fig. 3(a)). It’s not surprising to see this difference in resolution measurement, because ROMP resolution and FRC resolution are conceptually different: the former describes the resolution limit defined by localization precision and sampling rate; the latter is a spatial frequency with sufficiently high signal-to-noise ratio. The Rayleigh resolution given by the relative distribution of tubulin in microtubule cross-section is around 30 nm (Fig. 3(b)), which evidences that ROMP resolution is capable of accurately measuring image resolution. Then, the applicability of our ROMP method on different structures is verified. As shown in Fig. 4, similar results from ROMP resolution quantifications on nuclear pore complex and mitochondria were obtained. More importantly, our metric is independent of the overall underlying sample structure (Fig. S5) and sample drift (Fig. S6), while FRC resolution, on the contrary, is significantly affected by these factors.

 

Fig. 3. ROMP resolution measurement of experimental microtubule imaging. (a) The resulted ROMP resolution and FRC resolution as a function of time. (b) Zoomed images of two boxed region in Fig. 1(a). The relative distribution of tubulin revealed by intensity plots showing the resolving capability of the system is around 30 nm. (c) Computation time of the FRC and ROMP resolution under different image sizes for a time interval of 50 frames. The yellow dash line indicates real-time processing threshold of the sCMOS camera used in this study at the fastest frame rate. The images with various sizes are cropped from a microtubule 2D imaging with 106×106 µm2 FOV (Fig. S4).

Download Full Size | PDF

 

Fig. 4. 2D ROMP resolution from NPC and mitochondria experiments. (a, d) Super-resolution images, (b, e) ROMP resolution and FRC resolution as a function of frame number, (c, f) Enlarged images (upper) from the boxed area in (a, d) and line intensity distribution (lower, the line is showed in the upper image). (a-c) are for NPC. (d-f) are for mitochondria.

Download Full Size | PDF

The computational speed of ROMP resolution depends on the number of localizations within the chosen time interval. Using a short time interval for calculation (Fig. S7) results in a computation time of only 10-100 ms for various sizes of FOV, which is even faster than the data acquisition time at fastest sCMOS camera frame rate (Fig. 3(c)). This speed is two-order of magnitude faster than that from FRC for small FOV, and this advantage enlarges to three-order of magnitude when FOV is larger than 100×100 µm².

3.3 Optimizing imaging throughput

In SMLM, the number of emitters in each frame usually reduces with time due to photobleaching or image buffer pH changes [34,35]. Real-time localization density calculation enables stabilization of localization density by providing feedback control to the activation laser, and thus optimizes the imaging throughput. Using localization density for feedback control is more advantageous comparing with previously used method that relies on the number of localizations in each raw image frame [17,36] because the localization density is a relative variable while the number of localization is an absolute variable that requires prior knowledge about the underlying sample structure. Note that the intensity of the activation laser directly controls the number of activated molecules (activation density), and activated molecules are subsequently localized by single molecule localization. Therefore, we use localization density to optimize the data acquisition process.

To quantify the benefit of throughput optimization from localization density stabilization, we acquired datasets of microtubule with and without real-time localization density stabilization (Fig. 5(a)). We show that our method is capable of achieving a stable localization density (Fig. 5(c, d)). By setting the localization density to be 0.16 µm^-2 in the feedback controlled dataset, a total number of 3300 frames (66 s for 20 ms exposure time) is enough for achieving 40 nm resolution (Fig. 5(b)). Meanwhile, when there is no feedback control, the emitter density decreases with acquisition time. As a consequence, a total number of 10000 frames is required to achieve the same 40 nm spatial resolution. These results demonstrate that the imaging throughput increases by three folds after using real-time localization density stabilization. We note that our method is compatible with other data acquisition strategies designed for optimizing imaging throughput, for example, high-density activation and localization [13,37]. Actually, for images containing high-density emitters, multi-emitter fitting algorithm can be combined with our method to improve localization density and image resolution (Fig. S9).

 

Fig. 5. Localization density stabilization improves image throughput. (a) The localization density and (b) ROMP resolution as a function of frame number for microtubule 2D imaging with and without localization density stabilization. The 2D localization density is acquired by mean nearest neighboring distance of each frame (Supplement 2). (c) Raw images from acquisitions with and without localization density stabilization at different data acquisition time points. The number in the upper right corner indicates the molecule number in that frame. (d) Enlarged super-resolution images from the first and the last 3000 raw image frames. The number in the upper right corner indicates the number of localizations in the image. The arrows point out significant structure changes due to different sampling rates in the first and the last 3000 raw image frames.

Download Full Size | PDF

3.4 Achieving homogeneous image quality for mosaic localization microscopy

In high throughput or high content screening imaging, tens or hundreds of FOVs are usually needed to be imaged [9,10]. With real-time image resolution measurement, it is possible to achieve mosaic localization microscopy with optimized imaging throughput and homogeneous image quality. To realize this goal, we developed a Micro-Manager plug-in, named QC-STORM (here QC stands for quality control) that provides real-time molecule localization, image reconstruction, and resolution measurement, along with active imaging system feedback control (Fig. S10 & S11).

As a proof-of-principle, we demonstrated the possibility of mosaic localization microscopy using multi-FOV imaging of U-2 OS microtubules (Fig. 6(a)). In the experiments, 9 adjacent FOVs with a size of 106×106 µm² was acquired. For each FOV, we set the same desired localization density (0.2 µm^-2) and ROMP resolution (55 nm). The number of raw image frames for each FOV is automatically determined during the data acquisition according to the resolution setting. As a result, we achieved a stable localization density and a homogenous image resolution (Fig. 6(b, c)). Meanwhile, a fixed number of 6000 raw image frames for each FOV results in large variance in the image quality (Fig. 6(b, c)). In the experiments with localization density stabilization, the number of acquired raw image frames for each FOV ranges between 3010 and 3660, exhibiting 1.6∼2 times of throughput improvement compared with the experiments without localization density stabilization. It should be noted that the localization density (Fig. 6(c)) was calculated by Eq. (5), which mainly depends on the labeling efficiency and the number of detected emitters. However, there is no linear relationship between the localization density and labeling in this method, and the localization density would be affected by a number of experimental factors, including but not limited to sample properties and imaging conditions. Therefore, it is not surprised to see that the nine fields of acquicition2 have different localization densities.

 

Fig. 6. Achieving homogenous image quality with real-time ROMP resolution measurement. (a) Multi-FOV imaging of microtubules of U-2 OS cells using mosaic localization microscopy, with (left) or without (right) real-time localization density stabilization and ROMP resolution control. For acquisition 1, the 405 nm activation laser was controlled to achieve a temporally homogenous localization density of 0.2 µm^-2 and the stage automatically moved to the next FOV once the desired ROMP resolution was obtained. While in acquisition 2, a fixed number of 6000 frames was acquired for all FOVs without any hardware feedback control. Note that each FOV was excited to drive most of the molecules to a dark state before acquisition. (b) The zoomed views of the corresponding boxed regions in (a). (c) The maps of localization density, the number of raw image frames and ROMP resolution for the 9 FOVs in both (a) and (b).

Download Full Size | PDF

4. Conclusion

We present a new method called ROMP for real-time image resolution measurement. This method realistically models the random molecule activation process and structural dimensionality, and thus enables hyper-parameter free calculation of localization density and accurate calculation of mean nearest neighboring distance. Because only a short time interval is required for accumulating sufficient localization events, ROMP resolution measurement is real-time and sample-drift insensitive. With these capabilities, our ROMP method is a useful tool for optimizing imaging throughput and achieving homogeneous image resolution for mosaic localization microscopy. By combing 3D Nyquist resolution and localization precision calculation [3,21], ROMP resolution measurement is readily extended to 3D imaging, as demonstrated by open microtubule and nuclear pore complex 3D data sets (Fig. S8) [38,39].

Image resolution is an informative guide for setting the ending point of data acquisition. We integrated the real-time ROMP resolution measurement into a user-friendly and open-source Micro-Manager plug-in, QC-STORM. This plug-in is capable of providing plentiful statistical information, such as localization density, photon number, localization precision, PSF width and SNR, etc., which are useful for quality control and optimization during data acquisition process. For example, SNR can be used to find the best imaging focal plane. With this plug-in, we can readily find the data acquisition sweet spot that both temporal and spatial resolution are optimized under different imaging conditions.