Structured illumination microscopy (SIM) [1] stands out among the optical superresolution techniques due to its modest illumination intensities combined with high acquisition frame rates. Furthermore, it is compatible with the full gamut of chemical and biological dyes, which makes it one of the most versatile of all nanoscopies. Widefield SIM can image at almost real-time speeds reaching 11 Hz with a resolution approaching 100 nm [2]. The more light intense confocal SIM method can perform at over 100 Hz, but at a lower resolution of about 150 nm [3]. Although the resolution improvement in widefield and confocal SIM modalities exploits the same physical principle [4], their implementations and reconstruction procedures differ substantially [5–7]. Here we focus on widefield SIM, as illumination intensities are inherently low, which renders it the most gentle and useful superresolution technique for live subcellular imaging. The same applies for longer-term imaging of living systems: SIM makes better use of the fluorophores’ photon budget compared to alternative superresolution methods [2], and imaging over hundreds of time-points is possible without damaging the specimen through phototoxicity or fluorophore bleaching. This is especially true when combined with selective plane illumination microscopy (SPIM) [8]. In general, SIM increases resolution by encoding high spatial frequencies of the sample in beat patterns. These are transmittable through widefield microscopes, but need to be un-mixed from the raw data in order to increase resolution. Conventional 2D-SIM requires three images to be acquired with translated illumination patterns to un-mix spatial frequencies along a single orientation. According to the well-known SIM image formation equation,

 

Fig. 1. Principle of SIM reconstruction from (a) 9 or (b) 7 images. In the top right-hand corner, the widefield support is shown. Colors are used to indicate different orientations of illumination patterns. Using all orientations concurrently avoids redundancies in the reconstruction process (see the text). (c) 4SIM: here each SI frequency component is decomposed into “superresolution” information and widefield information. Segmenting each SI spectrum into multiple components yields additional equations. Adding a widefield image permits unambiguous SIM reconstruction (see the text).

Download Full Size | PDF

For example, 7SIM works with just 2 SI images per orientation and a single widefield image [gray circle, Fig. 1(b)]. Redundancies also exist between SI images of the same orientation. In fact, each SI frequency component contains just a certain amount of “superresolution” information; the rest is information from shifted widefield spectra. A simplification of this observation is illustrated in Fig. 1(c). The left-hand side shows the sample spectrum represented with the same color coding as before. One sees considerable spectral overlap between widefield (gray) and superresolution (colored) spectral components. To illustrate how redundancies arise in overlapping regions, we sketch raw image spectra in rectangular, rather than circular format. This can be exploited to obtain a solvable equation system for each orientation without requirements for phase stepping during the acquisition procedure and 4 image acquisitions are enough to obtain superresolution (4SIM = 3SI + WF). A rigorous analysis shows that this argument holds also for circular OTFs. To see this, let us first consider the complete SIM spectrum to be decomposed into the 66 unique spectral segments (see left-hand side of Fig. 2). Sixty-six unknowns have to be determined. Each SI spectrum provides 18 equations consisting of three unknowns, representing unique spectral segments (for example, segments [6,52,65] in Fig. 2). The widefield image provides 18 equations with direct access to 18 unknowns, shown as grayed out areas in the top of Fig. 2, while the three SI images (one per orientation) consist of superimposed frequency information, which must be unmixed. Each of the overlay spectra contain 18 regions, i.e., a total of $3\times 18=56$ equations, where each equation has three unknowns from the whole set of 66 unknowns.

 

Fig. 2. SIM spectrum can be divided into 66 segments (left). SIM imaging yields spectra that contain spectral information overlaps. For example, ${\mathrm{SI}}_3$ shows a superposition of segments [6,52,65], which are highlighted as an example. Hence, each SI spectrum gives rise to 18 equations totaling to $3\times 18=56$ equations. This equation system is underdetermined as there are 66 distinct spectral elements in the problem, but a maximum likelihood estimation can nevertheless yield resolution improvements comparable to 9SIM. If a widefield spectrum (WF) is recorded and added to the three SI spectra, a unique reconstruction is possible (illustrated by the dotted line). The segmentation is illustrated in further detail in Visualization 1.

Download Full Size | PDF

Therefore, in 4SIM, a total of 72 equations is available to retrieve the 66 unknown segments, representing an overdetermined equation system, which can be inverted uniquely. Note that all equations are mutually independent, as can be proven by calculating the rank of the $66\times 72$ mixing matrix $\mathbf{M}$. The number of raw images can be further reduced by leaving out the widefield component, which we term 3SIM. In this case, however, the system is now underdetermined, as only 56 equations are available for 66 unknowns, and $\mathbf{M}$ is no longer uniquely invertible. However, using a maximum likelihood approach, we can approximate the optimal separation of segments to reconstruct SIM images from only three SI images to achieve similar resolution as in 9SIM, but with a three-fold raw data reduction:

 

Fig. 3. (a) Simulations of widefield imaging and SIM with 25 beads of 80 nm diameter. The widefield image was deconvolved using jRL to allow comparison with 3SIM and 9SIM data. The simulation parameters are listed in the supplementary information. The line profiles show that both 3SIM and 9SIM achieve similar resolution and are beyond the Sparrow limit, while the widefield and deconvolved versions are not. (b) Radially averaged spectra of (a) demonstrate that only 3SIM and 9SIM offer true superresolution and restore frequencies of the ground truth spectrum up to twice the Abbe limit. The scale bars are 0.5 μm.

Download Full Size | PDF

However, a closer inspection shows performance differences: for adjoining beads [see, for example, the inlays in Fig. 3(a)], a two-point separation below the Sparrow limit is only achieved by the SIM modalities, but not by deconvolution alone. This becomes even clearer on analysis of respective Fourier transforms. Figure 3(b) shows a plot of the radially averaged spectra. The hard edge of the simulated beads produces circular ripples in the ground truth spectrum, which can only be restored via SIM superresolution [arrow head in Fig. 3(b)]; pure deconvolution boosts high-frequency content, but fails to reconstruct the first ripple. Encouraged by these results, we imaged fluorescent beads (Tetraspeck, Thermo Fisher Scientific) of a nominal diameter of 100 nm on a custom-built SIM system [15]. The microscope was equipped with a 1.49 NA objective (Olympus), and the SI pattern frequency was set to 70% of the maximal spatial frequency of the 488 nm excitation light. An sCMOS camera (ORCA Flash v4.0, Hamamatsu) was used, with each pixel representing 65 nm in the sample plane. Figure 4 compares an acquired widefield image and its deconvolved version to both 3SIM and 9SIM. To enable quantitative comparison, all four images were generated from the same data set: nine raw SI images were first acquired and summed to form a widefield image, which was subsequently deconvolved. The 3SIM image was reconstructed from three out of the nine raw frames, the 9SIM image from all nine raw frames. jRL was used in all cases. Again, the reconstructed bead sizes of the deconvolved image are similar to both SIM reconstructions. Real resolution improvements can be observed in the enlarged panels of neighboring beads. Only the 3SIM and 9SIM algorithms lead to point separation beyond the widefield Sparrow limit, although slight distortions are apparent in the 3SIM example.

 

Fig. 4. Experimental comparison if different imaging modes. For all methods, nine raw SI images were used as the basis for processing. (a) Widefield image was generated by summing all raw images. (b) Widefield image deconvolved with the jRL algorithm. (c) 3SIM image was reconstructed from three of the nine SI raw images. (a)–(d) Inlay panels show two neighboring beads. Both 3SIM and 9SIM distinguish the beads and, thus, achieve resolution beyond the Sparrow limit. The scale bars are 1 μm and 100 nm in the enlarged panels.

Download Full Size | PDF

To quantify the performance of #SIM on dynamic samples, we simulated a moving target. (The full simulation parameters are listed in Appendix B.) The target featured a double-line structure spaced 180 nm apart and moving from left to the right at constant velocity $v$. After each full acquisition sequence, the velocity was increased (Fig. 5), and the simulation repeated. Widefield and SIM imaging were simulated at a 100 Hz raw frame rate, and the same line profiles was taken for every sample velocity. Figure 5 contrasts these “velocity-dependent” line profiles, starting at 0 nm/ms at the top and finishing at 5 nm/ms at the bottom. Shown are the ground truth, widefield, and deconvolved widefield cases, as well as four different #SIM implementations. The simulation was repeated 20 times for each parameter setting, and the results were averaged. Already for a stationary target the double-line feature remains unresolved in the widefield image, while the deconvolution recovers it according to the Sparrow criterion. All SIM versions resolve the feature clearly. As the velocity increases, a sample shift becomes significant as the number of raw frames increases, and the SIM modalities begin to fail at various points. We find that 9SIM reaches its limits at target velocities of about 1.5 nm/ms for the sample modeled. This is close to the speed at which, for example, membrane-bound vesicles traffic along microtubules [16]. The presented results provide experimental and numerical validation for the proposed SIM framework. They demonstrate that SIM imaging with only three raw frames is possible.

 

Fig. 5. Double-line target with 180 nm spacing generated in silico, as shown in the upper left-hand corner, was modeled to move in a direction perpendicular to the line pattern at different velocities $v$. The image recording was simulated at 100 Hz. The line profiles through the target are plotted for different $v$. At velocities up to 1 nm/ms, all SIM modalities were able to resolve the lines clearly. The images of faster moving targets deteriorated as the underlying structure varied significantly during the acquisition sequence. The modalities for which fewer raw images are required permit imaging of faster moving samples. Above 3 nm/ms, the optimal modality is deconvolution microscopy for the target shown.

Download Full Size | PDF

Faster events can only be imaged reliably in SIM reconstructions from fewer raw frames. For sample velocities above 3 nm/ms, deconvolution microscopy (implementing energy conservation, positivity, support constraints, etc.) turns out to be superior to SIM and, therefore, should be the method of choice. We also find that the standard deviation of the reconstructions in the SIM modalities increases with increasing target velocity (not shown). Note that although the acquisition speed is tripled, 3SIM can only provide a robust reconstruction performance up to frame rates of about twice that of 9SIM. Generally, 3SIM offers potential for new combinations of SIM with light sheet microscopy [17] over those demonstrated so far [8]. As optical sectioning is intrinsic to SPIM, the raw frame signal-to-noise ratio (SNR) is improved, which favors 3SIM. An extension of the #SIM principle to 3D-SIM is also thinkable. On the other hand, the fact that 3SIM requires the minimal number of raw images to achieve superresolution makes excellent use of the photon budget, allowing for benign imaging conditions. In addition, 3SIM eliminates the need for pattern shifting, significantly simplifying the complexity of SIM-SPIM hybrid systems. On the downside, imaging samples with a low SNR remain a challenge for SIM. Furthermore, the overall illumination is nonuniform, which results in spatially dependent photobleaching and locally varying SNR. Potential software solutions to this problem include stricter regularization with total variation approaches [18], using the concepts of mutual information and Kullback–Leibler divergence as an image proximity metric [19], and the penalization of frequencies associated with illumination frequency spikes [20]. Furthermore, we found that beyond a certain sample speed, deconvolution outperforms SIM methods. Hence, within the parameter space between speed, gentleness, and resolution, we conclude that 3SIM has its place in the center.