1. Introduction

Super resolution (SR) fluorescence microscopy has become a powerful tool for biologists to study the microcosm [1–4]. Structured Illumination Microscopy (SIM) [5–9], as a wide-field SR imaging technology with high imaging speed, low phototoxicity and good compatibility of various fluorescence probes, is most suitable for observing live cells. In recent years, SIM has developed rapidly and has been widely used in biological research.

Increasing imaging speed is a promising research direction, because higher-speed SIM can reduce the phototoxic effect and help reveal previously unseen dynamic processes of subcellular structures. For example, with a spatiotemporal resolution of 88 nm and 188 Hz, Hessian-SIM can observe the structural dynamics of mitochondrial cristae and structures [7]. With a 97 nm resolution and a speed of 266 frames/s, Grazing Incidence SIM (GI-SIM) enables observation of the basal cell cortex. Conventional SIM reconstruction algorithms, including Hessian-SIM and GI-SIM, use 9 raw images to reconstruct an SR image. From the perspective of information theory, these images contain redundant spatial frequency information, and fewer images are sufficient for resolution doubling [10–15]. Hence, reducing the number of raw images has emerged as the most effective approach to improve the temporal resolution of SIM. For simplicity, we hereinafter use 9SIM and RSIM to denote the SIM reconstruction algorithms using 9 raw images and reduced number of raw images, respectively.

For the research of live-cells, it is important not only to have high spatial-temporal resolution, but also to capture multi-dimensional information, such as spectrum and polarization [16]. For instance, the spectrum information can be used to distinguish the spatial overlapped subcellular structures, and help study cell dynamics, such as neuron growth and clathrin-mediated endocytosis [17,18]. The polarization behavior of fluorescence probes can be used to study the orientation of nucleoporins and green fluorescent protein (GFP), and further help revel the underlying arrangement and organization of nuclear pore complex and septin [19–21]. Fortunately, since SIM has good compatibility with fluorescence probes, it is easy to extend to a multi-color imaging technique [17,18,22], or even a hyperspectral imaging technique [23]. Furthermore, SIM is a natural polarized fluorescence microscopy. Exploiting the polarization states of structured illumination co-rotate with the illumination orientation [24], PSIM can simultaneously obtain the SR spatial information and dipole polarization information in the spatio-angular hyperspace [25]. In summary, information in different dimensions such as intensity (I), spectrum ($\lambda $) and polarization (P) can each provide a unique and useful optical perspective for observing cells, so that capturing multiple dimensional information with high spatial-temporal resolution can help reveal more complex mechanisms of biological process [26]. Therefore, there is a great need for a RSIM scheme that can simultaneously recover multi-dimensional information (I, p, $\lambda $) at high spatial-temporal resolution.

However, the reported RSIM reconstruction frameworks have shortcomings in handling polarization information, so it cannot support multidimensional super-resolution imaging. Specifically, the reported RSIM reconstruction frameworks use an incomplete forward imaging model that does not consider the polarization-dependent absorption efficiency. Since RSIM needs to jointly use the raw SIM images of all illumination orientations, SR reconstruction using this incomplete imaging model is prone to severe artifacts. Therefore, the reported RSIMs so far cannot be applied to fluorescent polarized samples. In addition, since different types of fluorescent probes are usually used in multicolor SR imaging, including those with polarization properties, the reported RSIMs cannot restore the information of all spectral channels.

In this work, we report a new SIM reconstruction framework called PRSIM, which enables recovering multi-dimensional information (I, p, $\lambda$) of sample using a reduced number of images. Compared to other super-resolution imaging techniques such as stimulated emission depletion (STED) [27], and stochastic optical reconstruction microscopy (STORM) [28,29], the most unique quality of PRSIM is its ability to capture multidimensional information of a sample at ultra-high speed with very low photon budgets. Therefore, PRSIM is much suitable for live-cell multiplexed imaging applications. PRSIM adopts a complete forward imaging model that considers the polarization-dependent absorption efficiency so that it is versatile for normal and polarized samples. We also develop a new frequency-domain iterative reconstruction algorithm based on gradient descent in PRSIM, which can perform artifact-free SR reconstruction using reduced number of SIM images. For normal samples, only 4 SIM images are required for SR reconstruction. For polarized sample, 6 SIM images are required to simultaneously obtain the SR spatial structure and dipole orientation. Verified by simulated and experimental data, our scheme not only exhibits the performance comparable to 9SIM, but also has less computational complexity and higher reconstruction speed than the state-of-the-art RSIM, making it more suitable for large field-of-view and high-throughput imaging. This work opens up a new route for the development of ultra-high-speed multicolor SR imaging tools.

2. Methods

To the best of our knowledge, PRSIM is the first RSIM reconstruction framework that considers the polarization modulation. By incorporating the polarization-dependent polarization efficiency into the forward imaging model, PRSIM is versatile for both normal and polarized sample, which is important to obtain multi-dimensional information. Since PRSIM requires more imaging parameters such as polarization-dependent absorption efficiency, modulation factor to construct precise forward imaging model, we also propose some new posteriori parameter estimation algorithms to extract parameters from reduced number of SIM images. To enable faster reconstruction without artifacts, we further design a new frequency-domain iterative reconstruction algorithm based on gradient descent. Overall, PRSIM contains three main steps: parameter estimation, iterative reconstruction and information fusion, as illustrated in Fig. 1, and the entire framework is described in detail below.

 

Fig. 1. Schematic of PRSIM. PRSIM mainly includes three steps: parameter estimation, iterative reconstruction and information fusion. The input is 6 frames of raw SIM images, including three illumination directions, and two original SIM images with complementary phases in each illumination direction. The first step is to estimate the reconstruction parameters from the input raw SIM images. The second step is to iteratively update the SR reconstruction image by minimizing the difference between the measured and estimated Fourier spectra of SIM images, where the estimated Fourier spectrum of SIM images is obtained based on the estimated parameters in the first step and the imaging model. The third step is to fuse polarization information and spatial intensity information in a pseudo-color PSIM image.

Download Full Size | PDF

The first step is to estimate the reconstruction parameters. All reconstruction parameters, such as the frequency spectra of raw SIM images ${D_{nm}}({\boldsymbol k} )$, the absorption efficiency of sample ${A_n}({\boldsymbol r} )$, the illumination patterns ${I_{nm}}({\boldsymbol k} )$, the wide-field image and polarization direction of dipoles $P({\boldsymbol r} )$, are extracted from the input raw SIM images, where ${\boldsymbol \; k}$ is the spatial frequency variable in Fourier domain, ${\boldsymbol r}$ is the nominal 2D coordinate of sample, and n and m indicate the ${n^{th}}$ illumination orientation and ${m^{th}}$ illumination phase, respectively. These processes are indicated by the green arrows in Fig. 1. For the input raw SIM images, the phase difference in one illumination orientation should be 2π/M (M is the number of phases), so that the sum of raw SIM images in each illumination orientation would be a wide-field image modulated by polarization-dependent absorption efficiency. On this basis, we developed a new approach to determine the polarization orientation of dipoles and polarization-dependent absorption efficiency by analyzing the modulated wide-field images in all three illumination orientations, as described in Eqs. (13)-(18) of Appendix B. To obtain the modulated wide-field images at minimum cost, it is desirable to acquire two raw SIM images with complementary phases in each illumination orientation, leading to a total of 6 raw SIM images for three illumination orientations. Different from the conventional illumination pattern estimation in 9SIM that pre-separates spectrum components from 3 SIM raw images and calculates frequency-weighted cross-correlation, PRSIM extracts the illumination parameters from each raw SIM image directly. Similar to the work of Lal et.al [30], the auto-correlation method was used to determine the spatial frequency, as shown in Eq.(8) in Appendix B. The non-iterative auto-correlation algorithm was used to retrieve the illumination pattern phases [31], as shown in Eq.(9) of Appendix B.

The second step is to iteratively reconstruct the SR image, as shown by the orange arrows in Fig. 1. The average sum of all raw SIM images is used as the initial value of the SR reconstruction image .n each forward pass, the raw SIM images are estimated according to the imaging model:

The third step is to fuse the polarization information $P({\boldsymbol r} )$ and the spatial intensity information ${S_{SR}}({\boldsymbol r} )$ in a pseudo-color PSIM image [25]. Since the polarization direction image is full of random noises in the sample-free region, it is difficult to visually observe useful information. In our scheme, the polarization direction image and the reconstructed SR image are taken as the Hue component and the Value component of PSIM image, respectively. The Saturation of PSIM image is set to 0.6. The Saturation temporarily does not characterize any information so that the principle of saturation selection is to be able to clearly express the change of Hue. The fusion of polarization direction image and the reconstructed SR image can effectively filter out random noises, and clearly reveal the relationship between polarization information and spatial intensity information.

Through the above three steps, PRSIM can restore both the spatial intensity and polarization information, and perform high speed reconstruction while effectively suppressing artifacts. Moreover, for samples labeled with ordinary fluorescent probes, the absorption efficiency ${A_n}({\boldsymbol r} )$ determined by PRSIM will be approximated to 1. Therefore, the PSIM imaging model will degrade to the conventional SIM imaging model, and PRSIM can also successfully reconstruct SR images. The mathematical formula along with the details of the reconstruction algorithms are provided in Appendix A–B.

In the following section, we will validate the proposed PRSIM through simulation and three experiments.

3. Results

3.1 Simulation results

To verify our theory and demonstrate the artifact-free reconstruction capability, we first generated 6 frames of raw SIM images (see Materials and methods) based on the PSIM imaging model. An image with rich high-frequency information was chosen as the test object depicted in Fig. 2(a). It is noted that the selected image is only used to generate simulated data and is not related to the imaging technique. Any image full of rich high frequency details can be used to generate raw SIM images for testing. For comparison, a deconvoluted wide-field image of the test object was also generated as shown in Fig. 2(b). The fluorescent dipole orientation $P({\boldsymbol r} )$ was set in the shape of a spiral phase plate as shown in Fig. 2(c). We performed 6-frame SR reconstructions with and without considering the polarization-dependent absorption efficiency. The same iterative reconstruction scheme proposed in Section 2.1 is adopted but with different imaging models. The first reconstruction uses the conventional imaging model without considering polarization-dependent absorption efficiency:

 

Fig. 2. (a) An image representing the fluorophore distribution of sample. (b) The wide-field deconvolution image of sample. (c) An image of fluorescent dipole orientation ${\boldsymbol P}({\boldsymbol r} )$ (d) The SR reconstruction result using conventional imaging model, the center and edges of which contains artifacts. (e) The artifact-free SR reconstruction result using PRSIM. (f) Estimated fluorescent dipole orientation.

Download Full Size | PDF

3.2 Artifact-free SR polarization imaging

We imaged phalloidin-labeled F-actin filaments in bovine pulmonary artery endothelial (BPAE) cells using Nikon N-SIM. The phalloidin-conjugated fluorophore labels the targeting structure with a strong covalent binging, which makes it exhibit polarization dependence on absorption. We chose 6 raw SIM images and performed the 6-frame reconstructions with and without considering the polarization-dependent absorption efficiency on the experimental data, respectively. The details of the experimental data are provided in the SIM imaging section in Materials and methods. Similar to the simulation experiment, these two reconstructions utilize the same iterative reconstruction scheme proposed in section 2.1 and differ only in the forward model. The reconstruction results are displayed in Fig. 3(b), and a color wheel indicates the relationship between the color and the orientation. The upper left part of Fig. 3(b) shows the 6-frame reconstruction result obtained by the conventional 6SIM imaging model, which is filled with periodic hexagonal artifacts. In the frequency domain, these hexagonal artifacts are manifested as the bright residual bright peaks in Fig. 3(c). While using PRSIM, such artifacts were effectively eliminated (The lower right part of Fig. 3(b)), and the corresponding residual peaks in the frequency domain were also suppressed, as shown in Fig. 3(d). This kind of artifacts caused by the polarization characteristics of fluorescent probes is analyzed in detail in Appendix D. Compared with the wide-field (WF) image deconvolved by applying a wiener filter (Fig. 3(a)), the reconstruction result of PRSIM not only restored the polarization information, but also resolved more fine structures of F-actins. Several actins in close proximity that are difficult to resolve in the WF image (Fig. 3(e)) can be easily distinguished by PRSIM (Fig. 3(g)). The reconstruction result of PRSIM is comparable to that of 9-frame PSIM (Fig. 3(h)), which is further validated by the intensity profiles (Fig. 3(i)). It is worth noting that the intensity profiles are plotted using raw data and have not been interpolated or fitted. This plot provides a realistic depiction of the intensity values for each pixel on the intensity profile, which is beneficial for subsequent analysis.

 

Fig. 3. SR Polarization imaging results of F-actin in BPAE cells. (a) Wiener deconvolved wide-field (WF) image. (b) 6-frame reconstructed SR images using conventional SIM imaging model and PRSIM imaging model, respectively. The upper left part is the reconstruction result using conventional SIM imaging model and the lower right part is the reconstruction result of PRSIM. The color wheel in the bottom right indicates the relationship between the pseudo-color and dipole orientation. (c), (d) The Fourier spectra of 6-frame SIM reconstruction result and 6-frame PRSIM reconstruction result in b, respectively. (e) The magnified views of the dashed box regions in (a). (f), (g) The magnified views of same region in 6-frame SIM and PRSIM reconstruction results respectively. (h) The magnified views of same region in 9-frame SIM reconstruction results. (i) Intensity profiles on the selected line in (e)-(h).

Download Full Size | PDF

To further quantitatively analyze the performance of our proposed iterative reconstruction algorithm, SR reconstructions were also performed using 4, 5, and 7 frames of raw SIM images. The polarization-dependent absorption efficiency ${A_n}({\boldsymbol r} )$ was obtained in advance using 9 raw SIM images. The raw SIM images used for each reconstruction are listed in Table 3. All SR reconstructions using a reduced number of raw SIM images were successfully performed and achieved results comparable to 9SIM reconstructions, as shown in Fig. 9 in Appendix F. For quantitative analysis, the SSIM and PSNR indexes are computed for the wide-field image and the PRSIM images (Figs. 9(b-e)), and listed in Table 1. Because the ground truth of samples is unknown, the SR reconstruction result of 9SIM is used as a reference for calculating the SSIM and PSNR indexes. All indexes are greater than those of the wide-field images.

Table 1. The imaging quality comparison between 4-frame, 5-frame, 6-frame and 7-frame reconstructions using quantitative indexes.

View Table | View all tables in this article

As expected, the more the raw SIM images used for reconstruction, the better the reconstruction quality obtained. Although the indexes are found to increase sequentially as the number of raw SIM images increase, there is no significant difference in the quality of reconstruction results. This indicates that the iterative reconstruction algorithm of PRSIM can use as few as 4 SIM images for SR reconstruction with comparable performance.

3.3 Multi-dimensional SR imaging using a reduced number of images

In biological applications, cells are usually labeled with multiple fluorescent probes. Since each subcellular structure is labeled with a specific fluorescent probe, spatially overlapping subcellular structures can be clearly distinguished in the spectral domain, allowing efficient investigation of dynamic interaction between subcellular structures. But the existing RSIMs are only suitable for ordinary fluorescent probes without polarization properties, which limits their applications in multi-dimensional imaging. In PRSIM, the three-step reconstruction algorithm is applicable for different kinds of fluorescent probes, enabling simultaneous polarization and multicolor SR imaging using a reduced number of raw images. To demonstrate this, BPAE cells stained with a combination of fluorescent dyes were imaged by Nikon N-SIM. The F-actin is labeled with Alexa Fluor 488 phalloidin which behaves like an electronic dipole. Although only 6 raw SIM images were used, PRSIM successfully performed multicolor SR reconstructions with artifact suppressed, as shown in Fig. 4(a). In the region of interest (ROI) of the green channel, PRSIM distinguished two F-actins in a close proximity, which would be mistaken for a single thick F-actin in the wide-field imaging results (Fig. 4(c)). Similarly, in the ROI of the red channel, the blurred cristae of the mitochondria in the wide-field imaging result was also resolved by PRSIM (Fig. 4(d)).

 

Fig. 4. (a) The multicolor SR imaging result of PRSIM on BPAE cell. Mitochondria were labeled with red-fluorescent MitoTracker Red CMXRos, F-actin was stained using green-fluorescent Alexa Fluor 488 phalloidin, and blue-fluorescent DAPI was used to label the nuclei. (b) The PRSIM image of green channel that contains polarization information. (c) PRSIM image and wide-field image (WF) of F-actin in ROI marked by the dashed white box. (d) PRSIM image and wide-field image of Mitochondria in the ROI marked by the white box. (e) Intensity profiles on the selected line in (c), interval between the two peaks on the PRSIM profile was 160 nm (32 nm per pixel). (f) Intensity profiles on the selected line in (d), the interval between the two peaks on the PRSIM profile is five pixels. Scale bars: (a) 3$\mathrm{\mu}$m, (c) 1$\mathrm{\mu}$m, (d) 1$\mathrm{\mu}$m.

Download Full Size | PDF

Unlike conventional RSIM, PRSIM can perform multicolor SR imaging for a variety of fluorescent probes. In addition, PRSIM can restore the polarization information of polarized sample, as shown in Fig. 4(b). In a word, PRSIM provides a strong support in software for realizing high-speed multidimensional imaging.

3.4 Comparison with the state-of-the-art RSIM scheme

The scheme proposed by Lal et.al [13] is the state-of-the-art non-iterative RSIM based on the conventional SIM imaging model. We conducted two experiments to compare our scheme with Lal’s scheme to demonstrate the superiority of PRSIM over non-iterative methods. The first one is to reconstruct SR images of F-actins, which are labeled by fluorescent probes with polarization properties, and the imaging parameters are described in the SIM imaging section in Materials and methods. PRSIM’s frequency domain iterative reconstruction algorithm reduces the spatial complexity of the computation, allowing PRSIM to process the raw data of the entire field of view (Fig. 5(a), field of view: 32.77${\times}$ 32.77 $\mathrm{\mu}{\textrm{m}^2})$. Whereas Lal’s scheme requires the construction of a huge matrix equation, so it cannot handle large-size images. Hence, the raw SIM images are cropped before being used for reconstruction, and the imaging field of view is limited to the white-box region of Fig. 5(a) (field of view: 12.93${\times}$ 12.93 $\mathrm{\mu}{\textrm{m}^2}$). Further, PRSIM can effectively avoid a large number of invalid computations for sparse matrix equations, so the overall computational time is less than that of Lal’s scheme. The computational times of PRSIM and Lal’s scheme are listed in Table 2, which demonstrates that the reconstruction speed of PRSIM is much faster than Lal’s scheme.

 

Fig. 5. (a) SR reconstruction results of PRSIM with full field of view. The color wheel in the bottom right indicates the relationship between the pseudo-color and dipole orientation. (b) 5-frame SR reconstruction results using Lal's scheme, whose field of view is restricted to the white-box region of (a). (c) The cropped wide-field image. (d) The cropped SR reconstruction result of conventional 9SIM. (e) The cropped PRSIM image. (f) From left to right are the magnified view of orange box regions in b-e respectively. The reconstruction results of PRSIM and 9SIM can distinguish the fine structures of F-actin, while the reconstruction result of Lal's could not duo to the inference of artifacts. (g) Intensity profiles on the selected line in (f). The intensity profile of PRSIM is consistent with that of 9SIM, having five peaks which are corresponding to five F-actins. While the intensity profiles of WF and Lal only have two big peaks. Scale bar: a 3$\mathrm{\mu}$m, b 1$\mathrm{\mu}$m, d 500nm.

Download Full Size | PDF

Table 2. Comparison of computational time between our scheme and the state-of-the-art scheme.

View Table | View all tables in this article

Because the cropped region contains fluorescent dipoles with different polarization directions, the lack of polarization information would affect the reconstruction quality. Lal’s scheme does not consider the polarization properties of fluorescent probes, and its reconstruction image is grayscale and full of artifacts (Fig. 5(b)). For comparison, the grayscale images of the same region are cropped from the wide-field image, 9SIM reconstructed image and PRSIM reconstructed image, respectively (Figs. 5(c)-(e)). It can be clearly seen that the reconstruction result of PRSIM is artifact-free, which is due to the effective processing of the polarization information as discussed above. The PRSIM reconstruction result is consistent with that of 9SIM, and both can resolve some F-actins in sub-diffraction proximity (Figs. 5(f), (g)).

We further performed another comparative experiment on Microtubules of the COS7 cells, which were labeled with ordinary fluorescent probes. The experimental results show that the SR reconstruction results of both PRSIM and Lal’s scheme can better distinguish fine structures of samples compared with the wide-field deconvolution image. Our scheme achieves performance comparable to the conventional RSIM. For ordinary fluorescent probe, the absorption efficiency ${A_n}({\boldsymbol r} )$ does not vary with illumination direction, and can be approximated as 1. Therefore, the PRSIM imaging model will degenerate to the conventional RSIM imaging model, resulting in similar results for the two methods. Overall, our proposed PRSIM is versatile for a wider variety of fluorescent samples with superior comprehensive performance than the conventional RSIM.

4. Materials and methods

4.1 Simulation data generation

We choose an image with rich high-frequency information as test object and the image is depicted in Fig. 2(a). The image is cropped from a larger image ‘12084’ that included in dataset B100. The imaging parameters are assumed as follows: numerical aperture NA = 0.9, excitation wavelength and emission wavelength both are 650 nm, calibration = 45 nm/pixel. OTF is calculated using the theoretical formula. The illumination pattern frequency is $\left| {{{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}} \right| = 0.8\,{k_c}$ (${k_c}$ is the cut off frequency of OTF support). The fluorescent dipole orientation $\,P(r )$ is set as the shape of spiral phase plate as shown in Fig. 2(c). All the simulation raw images are generated based on Eq. (3) and then corrupted by additive Gaussian noise (signal-to-noise ratio is set to 10). The 6 frames of raw SIM images are generated according to the imaging model Eq. (3) (n = 3, m = 2). The illumination orientations and phases ($\mathrm{\theta },\mathrm{\varphi }$) of these raw SIM images are (${15^\circ }$, ${45^\circ }$), (${15^\circ }$, ${225^\circ }$), (${75^\circ }$, ${45^\circ }$), (${75^\circ }$, ${225^\circ }$), (${135^\circ }$, ${45^\circ }$), (${135^\circ }$, ${225^\circ }$) respectively. The deconvolved wide-field image of this test object is also generated as shown in Fig. 2(b). It is noted that the values of ($\mathrm{\theta },\mathrm{\varphi }$) used for the generation are not utilized as a prior information for SR reconstruction. These parameters are determined from the raw SIM images using posteriori algorithms in Appendix b.1–b.3.

4.2 SIM imaging

The sample is the FluoCells prepared slide #1 (F36924) which contains bovine pulmonary artery endothelial (BPAE) cells stained with a combination of fluorescent dyes. F-actin was stained using green-fluorescent Alexa Fluor 488 phalloidin, of which the absorption efficiency is polarization dependent. The optical system we used is Nikon N-SIM and the imaging parameters are as follows: numerical aperture NA = 1.49 (oil immersed), excitation wavelength is 488 nm, emission wavelength is 515 nm, calibration = 60 nm/pixel. The OTF $H({\boldsymbol k} )$ was estimated according to imaging theory. A set of nine raw SIM images ${\textrm{D}_{nm}}({\boldsymbol r} )$ for three different values of illumination orientations (${\theta _1}$, ${\theta _2}$, and ${\theta _3}$), each with three different values of phase ${\varphi _{nm}}$ (${\varphi _{n1}}$, ${\varphi _{n2}}$, and ${\varphi _{n3}}$) were acquired (image size: 512${\times}$ 512. field of view: 30.72$\ast$ 30.72$\mathrm{\mu}{\textrm{m}^2}$). The difference between any two $\theta $ is about ${60^\circ }$, whereas the phase difference in the same illumination orientation is about ${120^\circ }$. All the nine raw SIM images were normalized and used for extracting the polarization information of fluorescent probes. 4-frame, 5-frame, 6-frame and 7-frame reconstruction are carried out using a subset of the 9 raw SIM images, and individual raw SIM images that are used for reconstructions are listed in Table 3 in Appendix F. The 9SIM results are obtained by further applying notch filter on the reconstruction results of fairSIM [32]. For Lal’s scheme, using more than 4 raw SIM images can effectively avoid leaving any blind regions of the sample. The blind regions are spatial regions not effectively illuminated by all illumination patterns [13]. Therefore, for a fair comparison, 5 frames of raw SIM images are used for reconstruction in both Lal’s scheme and PRSIM.

5. Discussion

In summary, we proposed a SIM reconstruction framework termed PRSIM to help achieve high-speed multi-dimensional SR imaging. Previously, although reducing the number of raw images is the most effective approach to improve the temporal resolution of SIM (RSIM), the reported reconstruction algorithms all neglect the polarization information, which not only result in RSIM to be prone to artifacts, but also cannot recover multi-dimensional information for the research of live-cells. By incorporating the polarization-dependent absorption efficiency into the imaging model, we developed a frequency domain iterative reconstruction algorithm based on gradient descent, which enables PRSIM is versatile for samples labeled with different kinds of fluorescent probes and can perform SR multi-dimensional imaging using a reduced number of SIM images. In addition, PRSIM has less spatial computational complexity than the state-of-the-art non-iterative RSIM, making it more suitable for large field-of-view imaging. We also developed posteriori parameter estimation algorithms to automatically extract the all the reconstruction parameters including polarization-dependent absorption efficiency from reduced number of images, making PRSIM an easy-to-use SIM reconstruction algorithm that eliminates tedious instrument calibration.

Our work reveals that incorporating the properties of fluorescent probes is essential for artifact-free computational fluorescence imaging, especially when the input raw images lack information redundancy. The paradigm of PRSIM can be applied to other types of computational imaging, such as SIM using two-dimensional pattern, non-linear SIM [20,21], 3D-SIM [22] and Fourier ptychographic microscopy [23] to accelerate reconstruction and suppress artifacts.

Since the degradation functions of PRSIM reconstruction results are unknown, the direct use of OTF for Wiener deconvolution may be prone to artifacts, and the reconstruction results of PRSIM are not deconvolved. The weight of high frequency details is relatively low, and the visual effects of PRSIM reconstructions are not as sharp as those of 9SIM. Recently. Gang Wen et al. developed a post-processing algorithm called HiFi-SIM to suppress sidelobe artifacts [33]. The HiFi-SIM scheme that transforms the effective point spread function (PSF) into an ideal form can be combined with PRSIM to further improve the fidelity of the reconstructions.

Hessian-SIM and GI-SIM are currently the fastest SR imaging technologies. Both of them improve the temporal resolution by reducing the exposure time, while PRSIM improves the temporal resolution by reducing the number of raw images required for reconstruction. These two types of technologies are compatible. Therefore, combining PRSIM with Hessian-SIM or GI-SIM is expected to further improve SIM imaging speed. Different from the interleaved reconstruction strategy [34] used in Hessian-SIM and GI-SIM that only updates spatial SR information in one direction at each time step, PRSIM not only provides resolution enhancement in all directions, but also restores polarization information. With the help of customized GPU-acceleration algorithms, our PRSIM has a great potential to develop into an ultra-fast multicolor imaging technology.

Recently, machine learning (ML) has achieved amazing success in RSIM. In the work by Jin et al. [15], Unet was applied to restore super-resolution result at low SNR level using only 5 or even 3 raw SIM images. Ling et al. utilized a more training-efficient neural network, CycleGAN, to extract the correlation between high-frequency information in each direction of the spectrum, and realized high-precision super-resolution reconstruction using 3 raw images in one direction [14]. Christensen et al. used ideal simulated image pairs to train a deep residual network to prevent the model from being exposed to traditional reconstruction artifacts, making the trained model becomes robust to noise in the raw SIM images, and the reconstruction using only 3 or 6 frames still performed well [35]. However, the current ML-based approach cannot completely outperform the model-based approach. The ML-based approach and the model-based approach have their own advantages and suitable application scenarios, and complement each other. Qiao Chang et al. [36]. also made a similar point. They proposed the concept of priority region, and showed that the priority regions of ML-based methods are relatively small and concentrate on zones of low fluorescence and low structural complexity, which rarely overlap with the applicable regions of conventional SIM methods. In the future development of RSIM, ML-based methods and model-based methods will certainly interact and learn from each other, giving rise to more novel ideas. How to establish a suitable evaluation index and from what perspective to compare these two methods are very important research topics. And, how to hybrid these two types of techniques is also an interesting research direction. For instance, neural state machine is a possible approach to cooperate model-based PRSIM with multiple neural networks to accomplish complex tasks [37].

Appendix A. Formation

Let the fluorophore density distribution within the sample be $S({\boldsymbol r} )$, where ${\boldsymbol r}$ is the nominal 2D coordinate of sample. For a conventional 2D SIM, the ideal sinusoidal pattern used for illumination can be written as:

(4)$${\textrm{I}_{nm}}\left(r \right)= 1 + {a_n}cos\left({{\boldsymbol{k}_{{\theta_n}}}\boldsymbol{r} + {\varphi_{nm}}} \right)$$

where ${\theta _n}$ is the ${n_{th}}$ orientation of the illumination pattern, and ${{\boldsymbol k}_{{\theta _n}}}$ = $({k\textrm{cos}{\theta_n},k\textrm{sin}{\theta_n}} )$ is the corresponding spatial frequency vector. ${a_n}$ is the modulation factor of structured illumination whose orientation is ${\theta _n}$ and ${\varphi _{nm}}$ is the ${m_{th}}$ phase of the illumination pattern at orientation ${\theta _n}$ ($n = 1,2,3$). Generally, in order to perform structured illumination with maximum contrast, the polarization orientation is set perpendicular to the pattern directions. Meanwhile, most fluorescent probes behave like electric dipoles that show polarization dependence on absorption. If we let the orientation of fluorescent dipole be $P({\boldsymbol r} )$, the absorption efficiency of fluorescent probes can be obtained as:

(5)$${A_n}(\boldsymbol{r} )= 1 - Bcos({2{\theta_n} - 2P(\boldsymbol{r} )} )$$

where B is the polarization factor. The fluorescent emission of sample is modeled by the multiplication of the sample $S({\boldsymbol r} )$, the illumination ${I_{nm}}({\boldsymbol r} )$ and the absorption efficiency ${A_n}({\boldsymbol r} )$. Finally, the emission fluorescence is collected by microscope, and a raw SIM image is generated:

(6)$${D_{nm}}(\boldsymbol{r} )= [{{I_{nm}}(\boldsymbol{r} ){A_n}(\boldsymbol{r} )S(\boldsymbol{r} )} ]\otimes h(\boldsymbol{r} )$$

where $h({\boldsymbol r} )$ is the optical system’s point spread function (PSF), and for simplicity, the magnification of microscope is neglected. In our scheme, the SR reconstruction is considered as an inverse problem to jointly use SIM images of all the three orientations. Specifically, the problem is formulated by minimizing the difference between the measured and estimated Fourier transform of SIM images:

(7)$${S_{SR}}(\boldsymbol{r}) = \mathop {\arg \min }\limits_{{S_{SR}}(\boldsymbol{r})} {\sum\limits_{n,m} {\|{{D_{nm}}(\boldsymbol{k} )- \mathscr{F}\left[{{I_{nm}}(\boldsymbol{r} ){A_n}(\boldsymbol{r} )S(\boldsymbol{r} )} \right]H(\boldsymbol{k})} \|} ^2}$$

where $F[\cdot ]$ denotes Fourier transform, ${\boldsymbol k}$ is the spatial frequency variable in Fourier domain. ${D_{nm}}({\boldsymbol k} )$ and $H({\boldsymbol k} )$ are the Fourier transforms of ${D_{nm}}({\boldsymbol r} )$ and $h({\boldsymbol r} )$, respectively. $H({\boldsymbol k} )$ is also the system’s optical transfer function (OTF). The frequency domain iterative approach not only offers greater chance to inherit benefits of the frequency domain algorithms developed since the advent of SIM, but also avoids the occurrence of edge artifacts when reconstruction is carried out directly in the spatial domain.

Appendix B. Reconstruction algorithms

In RSIM, the number of raw SIM images are reduced, so the imaging parameters such as spatial frequency ${{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}$, phase ${\varphi _{nm}}$ and modulation factor ${a_n}$ cannot be determined using the same way as in 9SIM. Further, pre-defined and calibrated imaging parameters are not suitable for moving object. Therefore, how to obtain the imaging parameters from reduced number of SIM raw image is a big problem. We propose posterior algorithms to obtain the modulation factor ${a_n}$ and the polarization-dependent absorption efficiency ${A_{\boldsymbol n}}({\boldsymbol r} )$. Combining with the existing algorithms that can calculate the spatial frequency ${{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}$ and phase ${\varphi _{nm}}$ from single raw SIM image, all the imaging parameters can be automatically determined. Further, using our proposed frequency domain iterative reconstruction algorithm, the SR image can be restored quickly without artifacts. The details are discussed below.

B.1. Determination of illumination spatial frequency ${{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}$

According to the work of Lal et.al [30], to generate an artifact-free result using four SIM images, the $\left| {{{\boldsymbol k}_{{\theta _n}}}} \right|$ should be set less than 0.86${\times}$ cutoff frequency ${k_c}$ (within the optical transfer function (OTF) support). Under this condition, the auto-correlation of ${D_{nm}}({\boldsymbol k} )$ will have a significant frequency peak in frequency domain, and the spatial frequency ${{\boldsymbol k}_{{\theta _n}}}$ can be calculated by locating this frequency peak with subpixel accuracy.

(8)$$\textrm{A}{\textrm{C}_{nm}}(\boldsymbol{k} )= {D_{nm}} \otimes {D_{nm}}(\boldsymbol{k} )$$

B.2. Determination of illumination pattern phase ${\varphi _{nm}}$

To retrieve the illumination pattern phases, here we use a fast non-iterative algorithm which determines each pattern phase from an auto-correlation of the respective Fourier image [31]. First, the Fourier transformed images are weighted with the complex conjugated OTF: ${D'_{nm}}({\boldsymbol k} )= \,{D_{nm}}({\boldsymbol k} ){H^\textrm{*}}({\boldsymbol k} )$. Then, using the estimated value of illumination spatial frequency ${{\boldsymbol k}_{{\theta _n}}}$ above, phase ${\varphi _{nm}}$ can be obtained by:

(9)$${\varphi _{nm}} ={-} \arg \left\{{[{{D^\prime }_{nm} \otimes {D^\prime }_{nm}} ]({{\boldsymbol{k}_{{\theta_n}}}} )} \right\}$$

where $\textrm{arg}\{\cdot \}$ represents the argument (or angle) of the complex valued.

B.3. Determination of illumination modulation factor ${{\boldsymbol a}_{\boldsymbol n}}$

In our approach, the illumination modulation factor ${a_n}$ is also extracted in the Fourier domain. For convenience of analysis, the polarization-dependent absorption efficiency ${A_n}(r )$ is absorbed into the fluorophore distribution of sample $S({\boldsymbol r} )$, and $S({\boldsymbol r} )$ is rewritten as ${S_n}({\boldsymbol r} )$. The Fourier transform of the SIM image can be given by:

(10)$${D_{nm}}(\boldsymbol{k} )= \left[ {{S_n}(\boldsymbol{k} )+ \frac{{{a_n}}}{2}{S_n}({\boldsymbol{k} - {\boldsymbol{k}_{{\theta_n}}}} ){e^{i{\varphi_{nm}}}} + \frac{{{a_n}}}{2}{S_n}({\boldsymbol{k} + {\boldsymbol{k}_{{\theta_n}}}} ){e^{ - i{\varphi_{nm}}}}} \right]H(\boldsymbol{k} )$$

Then the value of the ${D_{nm}}({\boldsymbol k} )$ at ${{\boldsymbol k}_{{\theta _n}}}\,$ can be written as:

(11)$${D_{nm}}({{\boldsymbol{k}_{{\theta_n}}}} )= \left[ {{S_n}({{\boldsymbol{k}_{{\theta_n}}}} )+ \frac{{{a_n}}}{2}{S_n}(0 ){e^{i{\varphi_{nm}}}} + \frac{{{a_n}}}{2}{S_n}({2{\boldsymbol{k}_{{\theta_n}}}} ){e^{ - i{\varphi_{nm}}}}} \right]H({{\boldsymbol{k}_{{\theta_n}}}} )$$

In the above equation, the first term ${S_n}({{{\boldsymbol k}_{{\theta_n}}}} )$ and the third term $\frac{{{a_n}}}{2}{S_n}({2{{\boldsymbol k}_{{\theta_n}}}} ){e^{ - i{\varphi _{nm}}}}$ is much smaller than the second term $\frac{{{a_n}}}{2}{S_n}(0 ){e^{i{\varphi _{nm}}}}$ for a natural object. Hence the value of ${D_{nm}}({{{\boldsymbol k}_{{\theta_n}}}} )$ is approximated as $\frac{{{a_n}}}{2}{S_n}(0 )H({{{\boldsymbol k}_{{\theta_n}}}} ){e^{i{\varphi _{nm}}}}$. Similarly, ${D_{nm}}(0 )$ can be approximated as ${S_n}(0 )\,H(0 )$, and the modulation factor is given by:

(12)$${a_n} = abs\left( {\frac{{2{D_{nm}}({{\boldsymbol{k}_{{\boldsymbol{\theta }_{\boldsymbol{n}}}}}} )H(0 )}}{{{D_{nm}}(0 )H({{\boldsymbol{k}_{{\boldsymbol{\theta }_{\boldsymbol{n}}}}}} )}}} \right)$$

Using the illumination parameters calculated by algorithms b.1-b.3, the structured illumination patterns can be estimated.

B.4. Determination of polarization-dependent absorption efficiency ${{\boldsymbol A}_{\boldsymbol n}}({\boldsymbol r} )$

Generally, the phase difference between sinusoidal illumination patterns is 2π/M (M is the number of phases in one illumination orientation), and the summation of these patterns is a constant for uniform modulation. Thus, averaging the SIM images with same pattern orientation, the wide-field images that modulated by different absorption efficiencies can be obtained:

(13)$${W_n}(\boldsymbol{r}) = \sum\limits_m {{D_{nm}}(\boldsymbol{r})} = [{{A_n}(\boldsymbol{r})S(\boldsymbol{r})} ]\otimes h(\boldsymbol{r} )$$

If M $\ge $ 3, a more robust approach is to use traditional 9SIM algorithm to decouple the three frequency components in Fourier domain [30] and the inverse Fourier transform of decoupled 0th order components are the modulated wide -field images. Note that the modulation of polarization dependent absorption efficiency is similar as the modulation of structured illumination, so the Fourier transform of ${W_n}({\boldsymbol r} )$ also contains three frequency components:

(14)$${W_n}(\boldsymbol{k} )= \left\{ {S(\boldsymbol{k} )- \frac{B}{2}F\left[{S(\boldsymbol{r} ){e^{ - i2P(\boldsymbol{r} )}}} \right]{e^{i2{\theta_n}}} - \frac{B}{2}F\left[{S(\boldsymbol{r} ){e^{i2P(\boldsymbol{r} )}}} \right]{e^{ - i2{\theta_n}}}} \right\}H(\boldsymbol{k} )$$

These three frequency components can be extracted from modulated wide-field images using the following equation:

(15)$$\left[ {\begin{array}{c} {S(\boldsymbol{k} )H(\boldsymbol{k} )}\\ {\frac{B}{2}F[{S(\boldsymbol{r} ){e^{ - i2P(\boldsymbol{r} )}}} ]H(\boldsymbol{k} )}\\ {\frac{B}{2}F[{S(\boldsymbol{r} ){e^{i2P(\boldsymbol{r} )}}} ]H(\boldsymbol{k} )} \end{array}} \right]\textrm{ = }{\left[ {\begin{array}{ccc} 1&{\textrm{ - }{e^{i2{\theta_1}}}}&{{e^{\textrm{ - }i2{\theta_1}}}}\\ 1&{\textrm{ - }{e^{i2{\theta_2}}}}&{{e^{\textrm{ - }i2{\theta_2}}}}\\ 1&{\textrm{ - }{e^{i2{\theta_3}}}}&{{e^{\textrm{ - }i2{\theta_3}}}} \end{array}} \right]^{\textrm{ - }1}}\left[ {\begin{array}{c} {{W_1}(\boldsymbol{k} )}\\ {{W_2}(\boldsymbol{k} )}\\ {{W_3}(\boldsymbol{k} )} \end{array}} \right]$$

Among these three frequency components, the 0th order component $S({\boldsymbol k} )H({\boldsymbol k} )$ is the Fourier transform of the unmodulated wide-field image:

(16)$${W_0}(\boldsymbol{k} )= S(\boldsymbol{k} )H(\boldsymbol{k} )= F[{S(\boldsymbol{r} )\otimes h(\boldsymbol{r} )} ]$$

By employing Wiener Filter, the deconvolution of wide-field images ${W_n}({\boldsymbol r} )$ (n = 0,1,2,3) can be obtained:

(17)$${W^\prime }_n(\boldsymbol{r} )= {F^{ - 1}}\left[ {\frac{{{W_n}(\boldsymbol{k} ){H^ \ast }(\boldsymbol{k} )}}{{{{|{H(\boldsymbol{k} )} |}^2} + \varepsilon }}} \right]$$

where $\varepsilon $ is the Wiener parameter, $\ast$ represents conjugate, and ${F^{ - 1}}\{\cdot \}$ denotes inverse Fourier transform. The 0th Tyler approximation of ${W'_0}({\boldsymbol r} )$ and ${W'_n}({\boldsymbol r} )$ (n = 1,2,3) are $\textrm{S}({\boldsymbol r} )$ and ${A_n}({\boldsymbol r} )\textrm{S}({\boldsymbol r} )$ respectively, so the polarization dependent absorption efficiency can be approximated as:

(18)$${A_n}(\boldsymbol{r} )= \frac{{{W^\prime }_n(\boldsymbol{r} )}}{{{W^\prime }_0(\boldsymbol{r} )}}$$

Even though the 0th Tyler approximation is coarse-grained, it is sufficient to eliminate the artifact introduced by neglecting the absorption efficiency. Figure 6 shows an example of the estimated wide-field images and the corresponding absorption efficiency images.

 

Fig. 6. (a, b & c) Wide-field images of F-actin modulated by different absorption efficiencies. (d, e & f) Estimated absorption efficiency images corresponding to the modulated wide-field images (a, b & c), respectively

Download Full Size | PDF

B.5. Determination of fluorescent dipole orientation ${\boldsymbol P}({\boldsymbol r} )$

By solving the Eq.(15), the +1st order component is also obtained:

(19)$$G(\boldsymbol{k} )= \frac{B}{2}F[{S(\boldsymbol{r} ){e^{i2P(\boldsymbol{r} )}}} ]H(\boldsymbol{k} )$$

Further employing Wiener Filter and inverse Fourier transform, the undegraded estimation of the +1st order component on spatial domain is computed as:

(20)$$G(\boldsymbol{r} )= \;{F^{ - 1}}\left[ {\frac{{G(\boldsymbol{k} ){H^ \ast }(\boldsymbol{k} )}}{{{{\left|{H(\boldsymbol{k} )} \right|}^2} + \varepsilon }}} \right]$$

Under the same assumption that the change of dipole orientation is small in the range of PSF, the approximate estimation of $\,P({\boldsymbol r} )$ is given by:

(21)$$P(\boldsymbol{r} )= arg\{{G(\boldsymbol{r} )} \}$$

It is noted that the illumination beams are rotated during SIM imaging, which may introduce intensity nonuniformity among the three orientations. In this case, calibration of the intensity nonuniformity is needed to achieve accurate measurement of the dipole orientations.

B.6. Iterative reconstruction based on Pytorch

The target SR image is set as a learnable tensor ${\textrm{S}_{\textrm{SR}}}({\boldsymbol r} )$ and initialized with average of all the input SIM images. If the sampling frequency of SIM image doesn’t exceed 4 times the cutoff frequency of imaging system, the size of SR image will be up-sampled by a factor of 2, so that the Nyquist sampling theory is still be fulfilled after the resolution of SR image is doubled. The Adam optimizer is used in our scheme. This optimizer is a first-order gradient-based optimizer using adaptive estimates of lower-order moments [38]. The number of iterations and the learning rate are in an approximately reciprocal relationship. We can roughly determine these parameters by observing the curve of loss. In our implementation, the learning rate is set to 0.005 and the number of iterations is set to 400. In every forward pass, the Fourier transform of raw SIM images are estimated according to the forward model:

(22)$${D_{n{m_ - }est}}(\boldsymbol{k} )= F\left[{{I_{nm}}(\boldsymbol{r} ){A_n}(\boldsymbol{r} ){S_{SR}}(\boldsymbol{r} )} \right]H(\boldsymbol{k} )$$

The L2-norm is used to measure the difference between the estimated and measured Fourier spectra of SIM image, and constrain the parameter of learnable tensor ${\textrm{S}_{\textrm{SR}}}({\boldsymbol r} )$.

(23)$$loss = {\sum\limits_{n,m} {\|{{D_{nm}}(\boldsymbol{k} )- {D_{nm\_est}}(\boldsymbol{k})} \|} ^2}\textrm{ + }\varepsilon {\|{{S_{SR}}(\boldsymbol{r})} \|^2}$$

In updating pass, the gradient of loss is calculated by the autograd of Pytorch, and then the learnable tensor is updated using gradient descent. By minimizing loss between the estimated and measured Fourier spectra of SIM images, the learnable tensor will approximate the SR result we want. In order to show both the SR intensity information and dipole orientation information of sample in one pseudo-color image $\textrm{PSIM}({\boldsymbol r}$), the dipole orientation $P({\boldsymbol r} )$ is set as the hue of $\textrm{PSIM}({\boldsymbol r}$), the ${\textrm{S}_{\textrm{SR}}}({\boldsymbol r} )$ is set as the value of $\textrm{PSIM}({\boldsymbol r}$), and the saturation of $\textrm{PSIM}({\boldsymbol r}$) is set to 0.6. The whole reconstruction algorithms (as described in section b.1–b.6) are coded into a series of Python-based functions.

Appendix C. Effect of polarization dependent absorption on 9SIM

In traditional 9SIM, three phase shifts (0, 2π/3, 4π/3) are needed for each orientation of the sinusoidal pattern to separate frequency components, and 3 different orientations are needed to extend the optical transfer function (OTF) support isotopically. The SR information are independently solved in each pattern orientation, and illumination nonuniformity and polarization dependent absorption efficiency is unchanged in each pattern orientation. Thereby, this parameter can be absorbed into fluorophore distribution of sample $S({\boldsymbol r} )$, and $S({\boldsymbol r} )$ is rewritten as ${S_n}({\boldsymbol r} )$. In Fourier frequency domain, this image formation is:

(24)$${D_{nm}}(\boldsymbol{k} )= \left[ {{S_n}(\boldsymbol{k} )+ \frac{{{a_n}}}{2}{S_n}({\boldsymbol{k} - {\boldsymbol{k}_{{\theta_n}}}} ){e^{i{\varphi_{nm}}}} + \frac{{{a_n}}}{2}{S_n}({\boldsymbol{k} + {\boldsymbol{k}_{{\theta_n}}}} ){e^{ - i{\varphi_{nm}}}}} \right]H(\boldsymbol{k} )$$

where ${\boldsymbol k}$ is the spatial frequency variable in Fourier domain. ${D_{nm}}({\boldsymbol k} )$, ${\textrm{S}_\textrm{n}}({\boldsymbol k} )$ and $\textrm{H}({\boldsymbol k} )$ are the Fourier transforms of ${D_{nm}}({\boldsymbol r} )$, ${\textrm{S}_\textrm{n}}({\boldsymbol r} )$ and $h({\boldsymbol r} )$. $\textrm{H}({\boldsymbol k} )$ is the system’s optical transfer function (OTF). Due to the structured illumination, SR information of the sample is downmodulated into two frequency components, ${\textrm{S}_\textrm{n}}({{\boldsymbol k} - {{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}} )\textrm{H}({\boldsymbol k} )$ and ${\textrm{S}_n}({{\boldsymbol k} + {{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}} )\textrm{H}({\boldsymbol k} )$. If ${\textrm{S}_\textrm{n}}({\boldsymbol k} )\textrm{H}({\boldsymbol k} )$, ${\textrm{S}_\textrm{n}}({{\boldsymbol k} - {{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}} )\textrm{H}({\boldsymbol k} )$ and ${\textrm{S}_\textrm{n}}({{\boldsymbol k} + {{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}} )\textrm{H}({\boldsymbol k} )$ are considered as independent unknown frequency components, Eq.(24) is a simple ternary linear equation. In order to solve these unknown frequency components in one direction, three SIM images: ${D_{n1}}({\boldsymbol r} )$, ${D_{n2}}({\boldsymbol r} )$ and ${D_{n3}}({\boldsymbol r} )$ with a phase difference of $2\pi /3$ arc degree are acquired, and the matrix equation in frequency domain can be given by:

(25)$$\left[ {\begin{array}{c} {{D_{n1}}(\boldsymbol{k} )}\\ {{D_{n2}}(\boldsymbol{k} )}\\ {{D_{n3}}(\boldsymbol{k} )} \end{array}} \right] = \left[ {\begin{array}{cccc} 1&{\frac{{{a_n}}}{2}{e^{i{\varphi_{n1}}}}}&{\frac{{{a_n}}}{2}{e^{\textrm{ - }i{\varphi_{n1}}}}}\\ 1&{\frac{{{a_n}}}{2}{e^{i{\varphi_{n2}}}}}&{\frac{{{a_n}}}{2}{e^{\textrm{ - }i{\varphi_{n2}}}}}\\ 1&{\frac{{{a_n}}}{2}{e^{i{\varphi_{n3}}}}}&{\frac{{{a_n}}}{2}{e^{\textrm{ - }i{\varphi_{n3}}}}} \end{array}} \right]\left[ {\begin{array}{c} {{S_n}(\boldsymbol{k} )H(\boldsymbol{k} )}\\ {{S_n}({\boldsymbol{k} - {\boldsymbol{k}_{{\theta_n}}}} )H(\boldsymbol{k} )}\\ {{S_n}({\boldsymbol{k} + {\boldsymbol{k}_{{\theta_n}}}} )H(\boldsymbol{k} )} \end{array}} \right]$$

Then relying on the knowledge of the relative phase between SIM images, the estimation of three scaled frequency components can be computed as:

(26)$$\left[ {\begin{array}{c} {{C_{0n}}(\boldsymbol{k} )}\\ {{C_{ + 1n}}(\boldsymbol{k} )}\\ {{C_{ - 1n}}(\boldsymbol{k} )} \end{array}} \right]\textrm{ = }\left[ {\begin{array}{c} {{S_n}(\boldsymbol{k} )H(\boldsymbol{k} )}\\ {\frac{{{a_n}}}{2}{e^{i{\varphi_{n1}}}}{S_n}({\boldsymbol{k} - {\boldsymbol{k}_{{\theta_n}}}} )H(\boldsymbol{k} )}\\ {\frac{{{a_n}}}{2}{e^{\textrm{ - }i{\varphi_{n1}}}}{S_n}({\boldsymbol{k} + {\boldsymbol{k}_{{\theta_n}}}} )H(\boldsymbol{k} )} \end{array}} \right] = {\left[ {\begin{array}{ccc} 1&{{e^0}}&{{e^0}}\\ 1&{{e^{\frac{{2\pi i}}{3}}}}&{{e^{ - \frac{{2\pi i}}{3}}}}\\ 1&{{e^{\frac{{4\pi i}}{3}}}}&{{e^{ - \frac{{4\pi i}}{3}}}} \end{array}} \right]^{ - 1}}\left[ {\begin{array}{c} {{D_{n1}}(\boldsymbol{k} )}\\ {{D_{n2}}(\boldsymbol{k} )}\\ {{D_{n3}}(\boldsymbol{k} )} \end{array}} \right]$$

where the $\frac{{{a_n}}}{2}{\textrm{e}^{i{\varphi _{n1}}}}$ and $\frac{{{a_n}}}{2}{\textrm{e}^{ - i{\varphi _{n1}}}}$ are scaling factors. It is worth noting that when ${\textrm{C}_{ {\pm} 1\textrm{n}}}({\boldsymbol k} )$ are shifted to their original place, the shifted frequency components ${\textrm{C}_{ {\pm} 1\textrm{n}}}({{\boldsymbol k} \pm {{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}} )$ contain significant overlapped zone with ${\textrm{C}_{0\textrm{n}}}({\boldsymbol k} )$. Therefore, the spatial frequency vector ${{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}$ can be estimated by iteratively optimizing the frequency-weighted cross-correlation at zero frequency between the ${\textrm{C}_{0\textrm{n}}}({\boldsymbol k} )$ and the ${\textrm{C}_{ {\pm} 1\textrm{n}}}({{\boldsymbol k} \pm {{\boldsymbol k}_\textrm{s}}} )$ by the shifted vector ${{\boldsymbol k}_\textrm{s}}$. When the cross-correlation coefficient is maximized, the spatial frequency vector is determined as $\,{{\boldsymbol k}_\textrm{s}}$ = ${{\boldsymbol k}_{{{\boldsymbol \theta }_{\boldsymbol n}}}}$. Then the modulation depth and initial phase of the illumination pattern (scaling factors) can also be calculated by the complex linear regression of $\textrm{H}({\boldsymbol k} ){\textrm{C}_{ {\pm} 1\textrm{n}}}({{\boldsymbol k} \pm {{\boldsymbol k}_\textrm{s}}} )$ against $\textrm{H}({{\boldsymbol k} \pm {{\boldsymbol k}_s}} ){\textrm{C}_{0\textrm{n}}}({\boldsymbol k} )$. By separating the scaling factors of ${\textrm{C}_{qn}}({\boldsymbol k} )$ (q = -1, 0, +1), the three frequency components are obtained as ${\textrm{E}_{\textrm{qn}}}({\boldsymbol k} )$. Same as the process procedure for SIM data of one direction, all illumination parameters and SR frequency components of three directions can be calculated. Finally, all frequency components are combined using weighted averaging with a generalized Wiener filter [39]:

(27)$${I_{SR}}(\boldsymbol{k}) = \frac{{\sum\limits_{q ={-} 1}^1 {\sum\limits_{n = 1}^3 {\frac{2}{{{a_n}}}} } {e^{ - i{\varphi _{n1}}}}{E_{qn}}(\boldsymbol{k} + q{\boldsymbol{k}_{{\theta _n}}}){H^ \ast }(\boldsymbol{k} + q{\boldsymbol{k}_{{\theta _n}}})}}{{\sum\limits_{q ={-} 1}^1 {\sum\limits_{n = 1}^3 {{{\left|{H(\boldsymbol{k} + q{\boldsymbol{k}_{{\theta_n}}})} \right|}^2} + {\eta ^2}} } }}G(\boldsymbol{k)}$$

where $\ast$ is conjugate, $\eta$ is the Wiener parameter, and $\textrm{G}({\boldsymbol k} )$ is the apodization function. Although resolved SR image ${S_\textrm{n}}({\boldsymbol r} )$ contains different information of illumination nonuniformity and polarization dependent absorption efficiency, these differences will be averaged in the step of generalized Wiener filter. Overall, due to the special algorithm flow of 9SIM, the polarization dependent absorption efficiency has little impact on resolving SR frequency information. In a word, 9SIM is versatile to all kinds of fluorescent probes.

Appendix D. Analysis of artifact caused by the polarization characteristics of fluorescent probes

This section is to show that for sample labeled with fluorescent probes that behave like electric dipoles, PRSIM is indispensable to achieve artifact-free SR imaging. We perform the 6-frame RSIM reconstructions with and without considering the polarization-dependent absorption efficiency on experimental data, respectively. These two reconstructions utilize the same iterative reconstruction scheme proposed at b.6, the only difference is the forward model they use. As displayed in Fig. 7(a), the reconstruction result using conventional incomplete forward model Eq.(28) is filled with the periodic hexagonal artifacts.

(28)$${D_{nm}}(\boldsymbol{r} )= \left[{{I_{nm}}(\boldsymbol{r} )S(\boldsymbol{r} )} \right]\otimes h(\boldsymbol{r} )$$

In frequency domain, these hexagonal artifacts are manifested as the residual bright peaks in Fig. 7(e). While another reconstruction result using complete forward model Eq.(1) is artifact-free, as depicted in Fig. 7 (c).Although its frequency spectrum also contains residual peaks, as shown in Fig. 7(g), these peaks are small and weak, and has little effect on result image. All these residual peaks in frequency spectra are located in the vicinity of frequency vector ${{\boldsymbol k}_{ij}}$

(29)$${\boldsymbol{k}_{ij}} = {\boldsymbol{k}_{{\theta _i}}} + {\boldsymbol{k}_{{\theta _j}}}({i,j ={-} 3, - 2, - 1,1,2,3,i \ge j\;{\& }\;i \ne - j} )$$

where ${{\boldsymbol k}_{{\theta _{ - i}}}} = - {{\boldsymbol k}_{{\theta _i}}}$. We believe that the residual peaks in Fig. 7(e) are mainly due to the use of wrong forward model that dose not consider polarization-dependent absorption efficiency, and these residual peaks are hard to remove using some heuristic methods such as notch filtering. Whereas, the residual peaks in Fig. 7(g) are similar to those reported in [40], and are due to strong frequency shifted signals arising from greater out-of-focus background. These residual peaks introduced by out-of-focus background cause artifacts only at low SNR regions, and can be easily suppressed by applying notch filters. To verify our idea, we designed a notch filter confined to frequencies around ${{\boldsymbol k}_{ij}}$ and apply the notch filter on both Fig. 7(e) and Fig. 7(g) respectively.

(30)$$N({\boldsymbol k} )= \prod {\left({1 - {e^{{{\textrm{ - }{{\left|{{\boldsymbol k} - {{\boldsymbol k}_{ij}}} \right|}^2}} / {{\sigma^2}}}}}} \right)}$$

where $\textrm{N}({\boldsymbol k} )$ is our designed notch filter, and $\sigma $ is a parameter to adjust the cutoff frequency of filter. The $\sigma $ is set small, because we want to sacrifice only a little amount of frequency information to suppress the artifacts. After applying notch filter, the residual peaks in Fig. 7(e) are not suppressed completely, as displayed in Fig. 7(f). Accordingly, the periodic hexagonal artifacts in spatial domain are not eliminated completely, as shown in Fig. 7(b). Whereas, the residual peaks in Fig. 7 (g) are removed cleanly after applying the notch filter, as shown in Fig. 7(h). Figures. 7(i-l) are magnified views of the dashed box regions in Figs. 7(a-d) respectively. Figures 7(i, j) clearly reveal the artifacts in Figs. 7(a, b) and Figs. 7(k, l) show the artifact-free reconstruction results using our proposed forward model. The experimental results verify our idea that the artifacts due to the use of wrong forward model are different from the artifacts introduced by out-of-focus background. These artifacts introduced by the use of wrong forward model are hard to remove using some heuristic methods at small cost, so our scheme is indispensable for solving this problem.

 

Fig. 7. (a) SR PSIM image reconstructed using conventional model. The SR PSIM image has periodical hexagonal artifacts, which severely degrade the imaging quality. (b) Resulting image after applying the notch filter on the Fourier spectrum of (a), and the artifacts are not eliminated completely. (c) SR PSIM image reconstructed using our proposed model, is artifact-free. (d) Resulting image after applying the notch filter on the Fourier spectrum of (c). (e-h) The Fourier spectra of (a-d) respectively. (i-l) The magnified views of the dashed box regions in (a-d).

Download Full Size | PDF

Appendix E. Quantitative tests of parameter estimation algorithms

In order to verify the accuracy of these parameter estimation algorithms in our reconstructed framework, we also conduct quantitative tests of the parameter estimation algorithms on the simulation data. The experimental parameters used for the simulation were: excitation wavelength = 650 nm; emission wavelength = 650 nm; numerical aperture NA = 0.9; image size = 512. The pixel size in the simulated raw image corresponds to 45 nm in sample space. We simulated noisy images for 51 different signal-to-noise levels, using Poisson noise and an expectation value of ${10^{n/10}}$, n = 0.50, in the brightest pixel. For each signal-to-noise (SNR) level, we generated 500 raw SIM data and the objects are images from super-resolution dataset ‘B100’. Each raw SIM data is generated with random structured illumination pattern:

(31)$$I = 1 + cos\;\left[{\left|\boldsymbol{f} \right|(xcos\theta \; + ysin\theta )\; + \varphi } \right]\;$$

where the spatial frequency $\left| {\boldsymbol f} \right|$ is randomly distributed between 0.5 and 0.9 times the cutoff frequency of the OTF support, and the direction angle $\mathrm{\theta }$ and the initial phase $\mathrm{\varphi }$ are randomly distributed between 0 and $\mathrm{\pi }$. We calculated the mean and standard deviation of the datasets’ spatial frequency and phase errors for all 500 images at each SNR level. Figure 8 shows the result of this quantitative analysis.

 

Fig. 8. The parameter estimation error at different SNR levels.

Download Full Size | PDF

The blue curves show the average error of our parameter estimation approach, and the error bars mark the standard deviation. For these extremely low SNR situations, the parameter estimation errors are too high to guarantee artefact-free SR reconstruction. For realistic photon levels (photon number > 100), the estimation approach yields accurate results when the average location error of spatial frequency vector is less than 0.02 pixel and the average phase error is less than 1°. Furthermore, the lower standard deviation indicates a much stronger robustness of our parameter estimation algorithms, so our parameter estimation algorithms do not negatively affect the reconstruction quality. Furthermore, our estimation algorithms only use a single raw image to extract its illumination parameters. Therefore, the parameter estimation of each raw SIM image is independent, and reducing number of SIM frames for reconstruction dose not affect the accuracy of parameter estimation.

Appendix F. Reconstruction results using a different number of raw images

 

Fig. 9. Pseudo-colored imaging results of F-actin in BAPE cells. (a) Wiener deconvolved wide-field image. The color wheel in the bottom right indicates the relationship between the pseudo-color and dipole orientation. (b-e) Reconstructed SR images using 4, 5, 6 and 7 raw SIM images, respectively. (f) Reconstructed SR image using the conventional 9SIM algorithms. (g-l) Magnified views of the dashed box regions in the reconstructed images depicted, respectively, in (a-f).

Download Full Size | PDF

Table 3. SIM images used by 4-frame, 5-frame, 6-frame and 7-frame reconstructions.

View Table | View all tables in this article

Funding

National Natural Science Foundation of China (61836004, 62088102); National Key Research and Development Program of China (2021ZD0200300, 2018YFE0200200); CETC Haikang Group-Brain Inspired Computing Joint Research Center.

Acknowledgements

The authors want to thank Mingkun Xu and Faqiang Liu for stimulating discussions on the use of deep neural networks.

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef]  

2. T. A. Klar, S. Jakobs, M. Dyba, A. Egner, and S. W. Hell, “Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission,” Proc. Natl. Acad. Sci. U. S. A. 97(15), 8206–8210 (2000). [CrossRef]  

3. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods 3(10), 793–796 (2006). [CrossRef]  

4. E. Nehme, L. E. Weiss, T. Michaeli, and Y. Shechtman, “Deep-STORM: super-resolution single-molecule microscopy by deep learning,” Optica 5(4), 458–464 (2018). [CrossRef]  

5. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. (Oxford, U. K.) 198(2), 82–87 (2000). [CrossRef]  

6. M. G. L. Gustafsson, D. A. Agard, and J. W. Sedat, “Doubling the lateral resolution of wide-field fluorescence microscopy using structured illumination,” Three-Dimensional and Multidimensional Microscopy: Image Acquisition Processing Vii 3919, 141–150 (2000). [CrossRef]  

7. X. Huang, J. Fan, L. Li, H. Liu, R. Wu, Y. Wu, L. Wei, H. Mao, A. Lal, P. Xi, L. Tang, Y. Zhang, Y. Liu, S. Tan, and L. Chen, “Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy,” Nat. Biotechnol. 36(5), 451–459 (2018). [CrossRef]  

8. Y. Guo, D. Li, S. Zhang, Y. Yang, J.-J. Liu, X. Wang, C. Liu, D. E. Milkie, R. P. Moore, U. S. Tulu, D. P. Kiehart, J. Hu, J. Lippincott-Schwartz, E. Betzig, and D. Li, “Visualizing Intracellular Organelle and Cytoskeletal Interactions at Nanoscale Resolution on Millisecond Timescales,” Cell 175(5), 1 (2018). [CrossRef]  

9. K. Samanta, S. Sarkar, S. Acuña, J. Joseph, B. S. Ahluwalia, and K. Agarwal, “Blind Super-Resolution Approach for Exploiting Illumination Variety in Optical-Lattice Illumination Microscopy,” ACS Photonics 8(9), 2626–2634 (2021). [CrossRef]  

10. F. Orieux, E. Sepulveda, V. Loriette, B. Dubertret, and J.-C. Olivo-Marin, “Bayesian Estimation for Optimized Structured Illumination Microscopy,” IEEE Trans. Image Process. 21(2), 601–614 (2012). [CrossRef]  

11. F. Ströhl and C. F. Kaminski, “Speed limits of structured illumination microscopy,” Opt. Lett. 42(13), 2511–2514 (2017). [CrossRef]  

12. S. Dong, J. Liao, K. Guo, L. Bian, J. Suo, and G. Zheng, “Resolution doubling with a reduced number of image acquisitions,” Biomed. Opt. Express 6(8), 2946–2952 (2015). [CrossRef]  

13. A. Lal, C. Shan, K. Zhao, W. Liu, X. Huang, W. Zong, L. Chen, and P. Xi, “A frequency domain SIM reconstruction algorithm using reduced number of images,” IEEE Trans Image Process 27(9), 455–4570 (2018). [CrossRef]  

14. C. Ling, C. Zhang, M. Wang, F. Meng, L. Du, and X. Yuan, “Fast structured illumination microscopy via deep learning,” Photonics Res. 8(8), 1350–1359 (2020). [CrossRef]  

15. L. Jin, B. Liu, F. Zhao, S. Hahn, B. Dong, R. Song, T. C. Elston, Y. Xu, and K. M. Hahn, “Deep learning enables structured illumination microscopy with low light levels and enhanced speed,” Nat. Commun. 11(1), 1934 (2020). [CrossRef]  

16. C. He, H. He, J. Chang, B. Chen, H. Ma, and Martin, “Polarisation optics for biomedical and clinical applications: a review,” arXiv pre-print server (2021).

17. D. Li, L. Shao, B.-C. Chen, X. Zhang, M. Zhang, B. Moses, D. E. Milkie, J. R. Beach, J. A. Hammer III, M. Pasham, T. Kirchhausen, M. A. Baird, M. W. Davidson, P. Xu, and E. Betzig, “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics,” Science 349, 6251 (2015). [CrossRef]  

18. R. Fiolka, L. Shao, E. H. Rego, M. W. Davidson, and M. G. L. Gustafsson, “Time-lapse two-color 3D imaging of live cells with doubled resolution using structured illumination,” Proc. Natl. Acad. Sci. U. S. A. 109(14), 5311–5315 (2012). [CrossRef]  

19. M. Kampmann, C. E. Atkinson, A. L. Mattheyses, and S. M. Simon, “Mapping the orientation of nuclear pore proteins in living cells with polarized fluorescence microscopy,” Nat. Struct. Mol. Biol. 18(6), 643–649 (2011). [CrossRef]  

20. B. S. DeMay, X. B. Bai, L. Howard, P. Occhipinti, R. A. Meseroll, E. T. Spiliotis, R. Oldenbourg, and A. S. Gladfelter, “Septin filaments exhibit a dynamic, paired organization that is conserved from yeast to mammals,” J. Cell Biol. 193(6), 1065–1081 (2011). [CrossRef]  

21. B. S. DeMay, N. Noda, A. S. Gladfelter, and R. Oldenbourg, “Rapid and Quantitative Imaging of Excitation Polarized Fluorescence Reveals Ordered Septin Dynamics in Live Yeast,” Biophys. J. 101(4), 985–994 (2011). [CrossRef]  

22. L. J. Young, F. Strohl, and C. F. Kaminski, “A Guide to Structured Illumination TIRF Microscopy at High Speed with Multiple Colors,” Journal of visualized experiments : JoVE 30(111), 53988 (2016). [CrossRef]  

23. G. Liu, H. Yang, H. Zhao, Y. Zhang, S. Zhang, X. Zhang, and G. Jin, “Combination of Structured Illumination Microscopy with Hyperspectral Imaging for Cell Analysis,” Anal. Chem. 93(29), 10056–10064 (2021). [CrossRef]  

24. K. O’Holleran and M. Shaw, “Polarization effects on contrast in structured illumination microscopy,” Opt. Lett. 37(22), 4603–4605 (2012). [CrossRef]  

25. K. Zhanghao, X. Chen, W. Liu, M. Li, Y. Liu, Y. Wang, S. Luo, X. Wang, C. Shan, H. Xie, J. Gao, X. Chen, D. Jin, X. Li, Y. Zhang, Q. Dai, and P. Xi, “Super-resolution imaging of fluorescent dipoles via polarized structured illumination microscopy,” Nat. Commun. 10(1), 4694 (2019). [CrossRef]  

26. K. Zhanghao, W. Liu, M. Li, Z. Wu, X. Wang, X. Chen, C. Shan, H. Wang, X. Chen, Q. Dai, P. Xi, and D. Jin, “High-dimensional super-resolution imaging reveals heterogeneity and dynamics of subcellular lipid membranes,” Nat. Commun. 11(1), 5890 (2020). [CrossRef]  

27. G. Vicidomini, P. Bianchini, and A. Diaspro, “STED super-resolved microscopy,” Nat. Methods 15(3), 173–182 (2018). [CrossRef]  

28. E. Nehme, D. Freedman, R. Gordon, B. Ferdman, L. E. Weiss, O. Alalouf, T. Naor, R. Orange, T. Michaeli, and Y. Shechtman, “DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning,” Nat. Methods 17(7), 734–740 (2020). [CrossRef]  

29. B. Diederich, P. Then, A. Jügler, R. Förster, and R. Heintzmann, “cellSTORM—Cost-effective super-resolution on a cellphone using dSTORM,” PLoS One 14, e0209827 (2019). [CrossRef]  

30. A. Lal, X. S. Huang, and X. Peng, “A Frequency Domain Reconstruction of SIM Image using four Raw Images,” in Optics in Health Care and Biomedical Optics Vii, Q. Luo, X. Li, Y. Gu, Y. Tang, and D. Zhu, eds. (Spie-Int Soc Optical Engineering, Bellingham, 2017).

31. K. Wicker, “Non-iterative determination of pattern phase in structured illumination microscopy using auto-correlations in Fourier space,” Opt. Express 21(21), 24692–24701 (2013). [CrossRef]  

32. M. Mueller, V. Moenkemoeller, S. Hennig, W. Huebner, and T. Huser, “Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ,” Nat. Commun. 7(1), 10980 (2016). [CrossRef]  

33. G. Wen, S. Li, L. Wang, X. Chen, Z. Sun, Y. Liang, X. Jin, Y. Xing, Y. Jiu, Y. Tang, and H. Li, “High-fidelity structured illumination microscopy by point-spread-function engineering,” Light: Sci. Appl. 10(1), 70 (2021). [CrossRef]  

34. Y. Ma, D. Li, Z. J. Smith, D. Li, and K. Q. Chu, “Structured illumination microscopy with interleaved reconstruction (SIMILR),” J. Biophotonics 11, 9 (2018). [CrossRef]  

35. C. N. Christensen, E. N. Ward, M. Lu, P. Lio, and C. F. Kaminski, “ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning,” Biomed. Opt. Express 12(5), 2720–2733 (2021). [CrossRef]  

36. C. Qiao, D. Li, Y. Guo, C. Liu, T. Jiang, Q. Dai, and D. Li, “Evaluation and development of deep neural networks for image super-resolution in optical microscopy,” Nat. Methods 18(2), 194–202 (2021). [CrossRef]  

37. L. Tian, Z. Wu, S. Wu, and L. Shi, “Hybrid neural state machine for neural network,” Science China Info. Sci. 64(3), 132202 (2021). [CrossRef]  

38. . Diederik and Ba, “Adam: A Method for Stochastic Optimization,” arXiv pre-print server (2017).

39. K. Wicker, O. Mandula, G. Best, R. Fiolka, and R. Heintzmann, “Phase optimisation for structured illumination microscopy,” Opt. Express 21(2), 2032–2049 (2013). [CrossRef]  

40. D. Li and E. Betzig, “Response to Comment on “Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics”,” Science 352, 522–527 (2016). [CrossRef]