1. Introduction

Since the invention of Stimulated Emission Depletion (STED) Microscopy by Stefan Hell at the beginning of the nineties of the last century [1, 2], the field of super-resolution fluorescence microscopy (microscopy beyond Abbe’s classical resolution limit) has seen an dramatic development with the invention and refinement of a plethora of new techniques such as Photoactivated Localization Microscopy (PALM) [3] or Stochastic Optical Reconstruction Microscopy (STORM) [4], for a recent review see e.g. [5]. One of the latest additions to the family of super-resolution methods is Stochastic Optical Fluctuation Imaging or SOFI [6]. In this method, one employs the temporal stochastic intensity fluctuations of emitters for enhancing the spatial resolution of an image. SOFI uses a conventional wide-field microscope and does not require any change in hardware or setting. The only requirement is that the used microscope has to be able to rapidly record images with high sensitivity. Only after a stack of images is recorded, SOFI evaluates the temporal fluctuations in each pixel of the images and calculates a super-resolved final image. SOFI will work with any labeling where the used labels exhibit stochastic, statistically independent intensity fluctuations. SOFI has been used in conjunction with quantum dots (QDs), photoswitching organic dyes [7–9], or photo-switchable fluorescent proteins [10, 11]. In [12], an particularly clever approach to SOFI was presented where the authors used donor molecules for labeling structures of interest and then used the FRET-induced blinking of by-diffusing acceptor molecules for performing SOFI. This approach is reminiscent of point accumulation for imaging in nanoscale topography (PAINT) in the context of PALM/STORM. Another particular variant of labeling was recently presented in [13] by Zheng et al. where the authors used multi-color quantum dot labeling for significantly boosting the label density and acquisition speed of SOFI. Besides enhancing the spatial resolution of imaging, SOFI provides also a very efficient suppression of scattering background and autofluorescence [6], and it endows a wide-field microscope with optical sectioning capability, thus allowing to use a wide-field microscope for obtaining true three-dimensional images of a sample [9, 14].

An important issue of SOFI is the impact of the finite pixel size of the wide-field detector (e.g. emCCD) which records the images. SOFI allows, in principle, to achieve arbitrarily high spatial resolution which is only limited the measurement time required for obtaining sufficiently well statistics (see also next section). However, the raw material on which the SOFI algorithm is applied to are frames with pixels of finite size. How can one thus achieve a spatial resolution of e.g. 50 nm in a final SOFI image if the raw images with which one starts have a pixelation of e.g. 100 nm? A solution to this problem was first proposed by Dertinger at al. in [15], and later perfected by Geissbuehler et al. in [8, 9, 16]. The core idea is to generate virtual pixels interspersed between the physical pixels of the recording camera by cross-correlating signals from the original pixel grid. This approach allows indeed to generate SOFI images with arbitrarily small pixel size, which is not limited by the pixel size of the recorded raw images. However, there are several disadvantages connected with this idea. First, the SOFI brightness values of the generated virtual pixels in the final SOFI image are different form the SOFI brightness values of the primary pixels, and even between different types of virtual pixels, the brightness values are different. Thus, for obtaining an unbiased final SOFI image, one has to apply sophisticated brightness re-calibration procedures, which heavily depend on the exact knowledge of the PSF of the imaging microscope. Second, when calculating higher order SOFI images, the number of possible high-order cross-correlations between several pixels for obtaining one desired virtual pixel at a given position is increasing exponentially, and there is no unambiguous way of how to chose which cross-correlation geometry will be optimal.

Here, we present a much simpler solution which is straightforward and exact, and which is based on the fact that the Optical Transfer Function (OTF) of a microscope has only a finite support. Thus, for sufficiently large spatial frequencies, a microscope does not transmit any further information, which is also the origin of the classical resolution limit. Thus, when performing a Fourier transform on an image, the Fourier amplitudes will drop to zero at large values of the Fourier vector. Padding the Fourier-transformed image with zeros will does not change or alter its information content, but after back-transforming such a padded image into real space, one obtains an image with increased pixel number of reduced size. This procedure thus represents a method of exact interpolation of the original image, without introduction of artifacts. Using this interpolation scheme, the pixel size of the original frames of a recorded movie can be adapted to the desired spatial resolution delivered by SOFI. We call this combination of SOFI with Fourier-transform based interpolation Fourier-SOFI or fSOFI, and in what follows, we give a detailed description of its principles and exemplify the method on imaging fluorescently labeled neurons.

2. Theoretical background of SOFI

We start by briefly recalling the core idea of SOFI. When imaging a sample with a wide-field microscope, each emitter in the sample contributes to the final image on the detector (typically a CCD) with some intensity distribution U(r − r′) which is called the Point Spread Function (PSF) of the microscope, where r denotes the position on the camera and r′ the position of an emitter in the sample. The PSF is fully determined by the optical properties of the microscope (in particular by the numerical aperture or N.A. of the used objective), and its size defines the resolving power of the microscope: the smaller the diameter of the PSF, the smaller details can be resolved in a sample, because contributions from closely spaced emitters start to overlap, so that any information of the fine details of the emitter distribution is lost. In PALM and STORM, one circumvents this problem by having, in each recorded image, only a very small sub-population of the labeling molecules in a fluorescent state, such that each emitter can be individually identified in the image, and its center position determined with an accuracy which is much superior to the width of the PSF. SOFI does not require this sparsity of fluorescent emitters and works even at large densities of simultaneously fluorescing emitters. To understand how it works, consider a sample which is labeled with N fluorescing molecules at positions ${\mathbf{r}}_j^{\prime }$ and time-dependent brightness s_j(t), 1 ≤ j ≤ N. The image which is recorded at any moment in time t is then given by the sum

The just described simple concept of second-order SOFI can be easily generalized to higher orders by using higher order cumulants, see [6].

In principle, SOFI is straightforward to implement and use. First, one records a temporal stack of images F(r,t) (movie), where the only requirement is that the image acquisition speed is faster than the typical time scale of the intensity fluctuations of the emitters in the studied sample. For calculating the e.g. second-order SOFI image, one would then have to calculate the cumulant images using Eq. (3) for all possible correlation times {τ}, and then to sum the results over all values of {τ} with 1 ≤ τ ≤ τ_max, where τ_max is some arbitrary maximum correlation time which has to be chosen large enough so that all intensity-fluctuation correlations of an emitter have decayed to zero. Values with τ = 0 should be excluded because shot noise is not suppressed in this case. However, the just described approach is computationally prohibitive, in particular when computing higher-order SOFI image using higher-order cumulants. One can adopt a simplified computational scheme which is taken from fluorescence correlation spectroscopy (FCS) where one often encounters the task to compute a correlation curve from asynchronous single-photon counting data [17]. For computing the second-order SOFI image, one starts with calculating the second order cumulant image for τ = 1,

In a next step, one generates a new stack of images F⁽²⁾(r,t) by binning every two subsequent frames into a single frame, i.e.

With this new stack of frames with coarse-grained temporal resolution, one again calculates a cumulant image for τ = 1,

This procedure is repeated k times until the coarse-grained time, 2^k times the frame time, is by ca. an order of magnitude larger than the typical correlation time of the intensity fluctuations, so that all the information contained in these fluctuations is captured. The final SOFI image is then calculated as the simple sum

The whole procedure is visualized in Fig. 1. Of course, the coarse-graining algorithm will not capture all possible correlation information. However, comparisons between both approaches on real data have shown that the difference in SOFI image quality between both approaches is in most cases imperceptible. Even more, in most cases the calculation of just $C_2^{(1)}(\mathbf{r})$ is already enough for obtaining a high-quality SOFI image, which tremendously reduces the numerical load and accelerates computation speed.

 

Fig. 1 Hierarchical SOFI algorithm: For each window w with N_w frames multiple cumulant images are computed in a hierarchical way, where the time resolution is coarsened by factor 2 in each level. The sum of the cumulants of all levels approximates the integral over τ of $C_n^w$ with exponentially growing bin size.

Download Full Size | PDF

Last but not least, let us finish this section by briefly addressing photo-bleaching, which is a common problem in fluorescence microscopy. Because SOFI relies on the stationarity of the fluorescence fluctuations of the sample, it is sensitive to photo-bleaching artifacts. To prevent that photo-bleaching affects a final SOFI image, it is recommended to divide the full stack of recorded images into sub-stacks of N_w frames, see Fig. 1, which has to cover a time span longer than the typical correlation time of the emitters’ intensity fluctuations, but has to be much shorter than the characteristic photo-bleaching time. The SOFI algorithm should then be applied to each sub-stack of N_w frames separately, so that photo-bleaching will not skew the SOFI analysis. Finally, the SOFI images of all sub-stacks can be added together to obtain the final SOFI image. This assures that the cumulant analysis of each sub-stack is unaffected by photo-bleaching.

3. Fourier SOFI

As stated in the introduction, one technical challenge encountered by SOFI in the past is the finite pixel size of the imaging detector. From equations (3) and (4) one can see that the final pixel size of an SOFI image is equal to the initial pixel size of the images on which the SOFI algorithm is applied. To circumvent this limitation and to adopt the final pixel size of an SOFI image to the spatial resolution provided by SOFI, Dertinger et al. [15] proposed to calculate temporal cumulants not only on one and the same pixel, but to also cross-correlate between pixels for generating new intermediate pixels with smaller effective size. Although this method gives good results, it is prone to grid-like artifacts because the SOFI brightness of cross-correlated virtual pixels is affected by PSF dependent weighting factors, see also [8,9,16]. Usually one focal plane (i.e. one PSF shape) has to be assumed for all emitters during this calibration, potentially leaving residual artifacts in the corrected images (see Fig. 6e). Furthermore, the method becomes computationally prohibitive and complex for higher-order SOFI.

There exists a much simpler and exact solution to this problem. The core idea is to recalculate the recorded images F(r,t) on a finer grid with reduced pixel size before calculating the SOFI images. This can be done exactly by employing the fact that Fourier transform of the PSF, which is called the Optical Transfer Function (OTF), has only a finite support. With other word, for sufficiently large absolute values of the Fourier vectors, the OTF drops to zero. This can be used to devise an interpolation scheme for re-calculating the recorded images on a finer pixel grid without introducing any artifacts. Let us assume that the images F(r,t) are recorded with a camera having N × N pixels of size d × d and co-ordinates r_ij, where 1 ≤ i, j ≤ N. Thus, when directly applying the SOFI algorithm, one obtains SOFI images with the same number of pixels and pixel size as the original images. Also, the discrete two-dimensional Fourier-transformed images $\tilde{F}(\mathbf{k},t)$ which can be calculated by a fast Discrete Fourier Transform (DFT) have the same N × N number of sampling points k_ij, with a maximum length of the spatial Fourier vector of 2π⌈(N − 1)/2⌉/d along both the x- and y-direction, where ⌈⌉ is the ceiling function. If the camera pixel size is chosen sufficiently small, smaller than the resolution limit of the microscope, then the Fourier-transformed images $\tilde{F}(\mathbf{k},t)$ will become zero for large absolute values |k_ij| of the Fourier vector. Thus, padding the Fourier-transformed images with zeros will not change at all the information content of the images. However, when back-transforming the zero-padded Fourier images to real space, this results in images with smaller pixel size and larger pixel number, without changing, in any way, the real image content. By using this general idea, as also schematically shown in Fig. 2, one can recalculate the original images on a finer grid with arbitrarily small pixel size without introducing any artifacts. In particular, when padding the Fourier images with Δ sampling points on all sides, the pixel size of the back-transformed images will be

 

Fig. 2 Algorithm of Fourier interpolation: Each frame (example: single emitter represented by an airy disc) is first Fourier-transformed. For sufficiently small detector pixel sizes, the Fourier transform is zero on the borders due to finite support of the OTF. The Fourier transform can thus be padded with zeros without changing the frequency information, here separated by a dashed white line from the original Fourier transform. Transforming back into real space gives an artifact-free image with more pixels, where each “virtual” pixel corresponds to a smaller area than that of the original detector pixels.

Download Full Size | PDF

It should be stressed here that the SOFI algorithm is applied to the raw images after they have been interpolated to smaller pixel size using Fourier interpolation. The core idea is that Fourier interpolation of the original movie makes higher frequencies numerically accessible for the subsequent SOFI calculation which yields a superresolved image with smaller pixel size. The combination of Fourier interpolation with SOFI is called Fourier-SOFI or fSOFI, and it allows for obtaining artifact-free SOFI images on an arbitrarily refined pixel grid. In practice, one should chose a final pixel size d_new which is at least two times smaller than the resolution limit of the highest order SOFI one wants to calculate (Nyquist sampling theorem). Because n^th order SOFI theoretically achieves a resolution that is n times better than that of the original images (after post-processing [15]), d_new should be at least 2n times smaller than the resolution of the used microscope when calculating n^th order SOFI.

For experimental data the Fourier-transformed images generally does not fall to zero for higher spatial frequencies due to noise. This leads to artifacts in the upsampled images, which are, however, canceled during SOFI computation (due to absent temporal correlation of the noise), leaving the SOFI image unaffected. Another issue is that a DFT intrinsically assumes an image which repeats itself infinitely often along all directions (double-periodic image). This may lead to artifacts in the upsampled image, because a real image is typically discontinuous across borders when stacking it periodically. A rigorous and exact solution to this problem is the procedure which is visualized in Fig. 3: Instead of directly upsampling the original image, it is first padded with mirror-symmetric half-copies of itself (see Fig. 3), and only this enlarged image is then upsampled and finally cropped back to the original size. Although this method increases the numerical load, it is exact and totally free of any artifacts.

 

Fig. 3 Example of periodic padding of an image for preventing boundary artifacts upon Fourier upsampling: Instead of Fouier-upsampling the original image, one first extends the image with mirror symmetric half-copies to obtain a larger and continuously periodic image (periodic padding), which is then Fourier-upsampled, and then cropped back to the original size. The comparison between both results (unpadded Fourier upsampling, right bottom, versus padded Fourier upsampling, right top) clearly shows that the padding procedure leads to a perfectly artifact-free upsampling of the original image.

Download Full Size | PDF

A comparison of simple interpolation and fSOFI is shown in Fig. 4 on a simulation of two very close emitters. As can be clearly seen, although the basic SOFI algorithm certainly improves spatial resolution when applied to the raw images, the pixel size of the raw images does not match the Nyquist criterion for the resolution achieved by SOFI, which leads to strongly “pixelated” SOFI images. By applying fSOFI, this pixelation can be avoided and the pixel size of the fSOFI image matches the improved spatial resolution. The improved image fidelity as well as subpixel positioning is presented on two simulated examples in Fig. 5, one of a ringlike structure of emitters, and one two closely positioned single emitters For comparison the raw cross-cumulant SOFI image is shown, displaying the typical artifacts inherent to this method.

 

Fig. 4 Comparison between fSOFI and interpolation of SOFI image using a simulation of two close emitters. a) Average of movie. b) 4^th order SOFI image. c) Linear interpolation of b. d) 4^th order fSOFI image. The coarseness of the pixel grid prevents standard SOFI to resolve the emitters. In contrast to interpolation, the fSOFI image captures true information and correctly resolves the emitters.

Download Full Size | PDF

 

Fig. 5 Two examples from a simulated SOFI experiment illustrating the presented Fourier interpolation in combination with SOFI. Top panels: 10 emitters in a ring. Bottom panels: Two emitters with sub-pixel shift (1.75 px, 0.25 px). a) Time average of all frames. b) 4^th order SOFI. c) 4^th order SOFI from 4x-super-sampled frames. d) Raw 4^th order cross-correlation SOFI. It is easy to see that the Fourier interpolation improves image quality and exposes the sub-pixel positioning of the emitters without producing artifacts visible in the raw cross-correlation approach.

Download Full Size | PDF

 

Fig. 6 Fourier interpolation on a images of blinking quantum dots. Rat hippocampal neuron with neurotransmitter receptor subunit GABA_BR1 immunostained with commercial quantum dots QD₅₂₅ (Invitrogen). The raw stack of images contains 3000 frames recorded at 20 Hz frame rate. Fluorescence was excited at 401 nm wavelength and about 20 W/cm² using an laser (Cube401, 100 mW, Coherent). The microscope used was a commercial epi-fluorescence microscope (IX-71, Olympus) equipped with an 1.4 oil-immersion objective (UPlanSApo, Olympus), and an emCCD (DU-897-CS0-BV, Andor). Magnification was chosen in such a way that the effective pixel size of the recorded images was 100 nm. Fluorescent light was filtered from the excitation light using a dichroic beam splitter (FF444/520/590, Semrock). (a) Time average of original images. (b) 2^nd order SOFI. (c) 2^nd order SOFI with 3x Fourier interpolation. The last two images show the raw 2^nd cross-correlation SOFI (d) and the artifact corrected image (e) calculated using publicly available software [20].

Download Full Size | PDF

As an experimental proof of principle, we have imaged the distribution of the R1-subunit of the B-type of the γ-aminobutyric acid (GABA) receptor in hippocampal neurons. GABA is one of the most important inhibitory neurotransmitters in the central nervous system, typically reduces the probability of action potential generation and thus prevents excess stimulation. Of the two types of GABA receptors, the ionotropic GABA_A receptor is a ligand-gated chlorine channel, while the metabotropic GABA_B receptor mediates further signal transduction by G-protein activation upon ligand binding. The GABA_B receptor is a heteromer consisting of the subunits GABA_BR1 and GABA_BR2 [19]. Both are expressed separately in the soma of neurons and initially reside in the somatic endoplasmic reticulum (ER) membrane. From there, they are transported to their sites of insertion into the plasma membrane of a dendrite. Fig. 6 shows the distribution of GABA_BR1 labeled with QD625 quantum dots in a hippocampal neuron. Comparison between the raw image (a) and the simple SOFI image (b) (without Fourier pixel interpolation) clearly shows the increase in resolution, but also the efficient suppression of background, which is due to the ability of SOFI to cancel the contribution of any non-fluctuating signal such as autofluorescence or scattering but also to the z-sectioning capability of SOFI. However, panel (b) also clearly shows strong pixelation because the pixel does not match the increased resolution provided by SOFI. This pixelation is removed by applying fSOFI, as shown in panel (c). There, all structural details become clearly visible, without the introduction of any artifacts such as the infamous ringing (ringlike shadow images around structures) so often appearing when applying sophisticated deconvolution algorithms in microscopy image processing. This is due to the fact that fSOFI does not change in any way the information content of the image, which is only contained within the finite support of the OTF where fSOFI does not change anything. Also shown are the raw and corrected cross-cumulant SOFI image. Note that the corrected image still shows artifacts, as they can not be completely corrected, probably because of the one-focal-plane assumption (see beginning of this section).

4. Conclusion

In the present paper, we have presented an easy and straightforward way ho to solve the “pixelation” problem of SOFI. The presented Fourier interpolation scheme is conceptually simple, easy to implement, and completely artifact-free. One of the open challenges of fSOFI is the computational demand: with an n-fold reduced pixel size of the input images processed by the SOFI algorithm, the computational load increases by n². For the future, we envision the development of hardware-based approaches for the rapid calculation of the image cumulants of the Fourier-interpolated raw images. Due to the simplicity and fundamental immutability of the fSOFI algorithm, this will be straightforward to achieve by using dedicated hardware such as a FPGA or a graphics cards. This can speed up calculations by orders of magnitude which could enable real-time fSOFI calculations.