1. Introduction

Optical microscopes have been widely used in the field of biology and life science for the past 400 years. However, the magnification and resolution cannot be enhanced to an unlimited level because of the nature diffraction of light. According to the Abbe diffraction limit, a sharp pointed object will be blurred into a finite-sized Airy disk through the optical microscope [1], which means the traditional microscope is difficult to observe the sub-wavelength structure (see Supplement 1 for more information). Therefore, advanced techniques such as atomic force microscope (AFM), and scanning electron microscope (SEM) have been proposed [2]. However, these methods are generally not suitable for real-time imaging [3]. Recently, microspheres imaging has been demonstrated as an important and potential technology for breaking through the diffraction limit [4–6]. This technology is simply based on optical imaging, using microspheres to generate nanojet that can simultaneously achieve wide-field imaging and sub-diffraction-limit resolution. Different from stimulated emission depletion (STED) and stochastic optical reconstruction microscopy (STORM), microsphere-assisted nanoscopy does not require complex fluorescent labeling procedures [7,8] and can be easily integrated with conventional optical microscopes [9,10]. Under the effect of photonic nanojets, transparent microspheres can generate a field with sub-diffraction width energy concentration [11–13]. And the magnified imaging is presented under the microspheres, which is similar to the virtual image of magnifying lens [14].

However, the quality and efficiency of microspheres imaging are not ideal because of the tiny imaging field-of-view (FOV) and spherical aberration [15]. The microsphere-assisted nanoscopy has not thus been promoted and used ulteriorly. To observe more of the imaging area, some researchers have used optical fibers [16] and AFM cantilever as support to fix and manipulate the microspheres to enhance the maneuverability of the microspheres, thereby realizing the sample scanning [17,18]. Other researchers directly combined the microspheres intending to make the “superlensing microscope” and perform the nanostructure measurement like normal microscopes [19–21]. Optical tweezers as a non-contact technology have been widely used to manipulate and observe transparent micro-particle in biological research [22], thus are more suitable for precise manipulation and imaging of microspheres. Li et al. have successfully created a single-cell bio-magnifier by utilizing optical fiber tweezers [23,24]. Wen et al. achieved super-resolution imaging with contactless microspheres under a conventional optical tweezers system [25,26]. These schemes indeed effectively increase the observation area of the microspheres, but the mosaic-like effect of the stitched images cannot be eliminated easily due to the uneven brightness and distortion of the microspheres imaging [27]. The imaging of microspheres can be modeled in a similar way to the working principle of ordinary cameras and projectors. This means that the magnification, imaging sharpness, distortion, and light intensity are directly related to the gap between the sample plane and microspheres [28]. These issues have rarely been mentioned in previous studies as the imaging mechanism of microspheres remains challenging [6].

Here, we establish a microsphere-assisted nanoscope modified from an optical tweezer system to investigate the mechanism of microsphere imaging. Relying on the bifocal plane of the laser and illumination module, we achieve precise control of the Z-axis and summarize the imaging property of the microspheres, and finally determine its optimal imaging plane. Our results verify that nanojets focus cannot simply be used as the imaging focal point of the microspheres; irradiance is the direct factor that affects imaging quality and magnification. This is consistent with the theory that microspheres do not have a single accurate focus plane. Besides, we also find that the tiny imaging view scope limits the equivalent aperture of microspheres, thus doubling its depth of field (DOF). To avoid spherical aberration and extend the imaging view scope of microspheres, we calculated the imaging distortion rate for different scales of scope and obtain the acceptable area of the magnified imaging through simulation and theoretical analysis. Then we scan some typical nano-structures, such as Blu-ray discs and standard through-silicon-via arrays, with microspheres under the action of optical traps. Finally, the large scope of 16 × 12 µm² imaging of microspheres is achieved within 3 seconds by precise manipulation and fast image stitching. These results demonstrate the reliability of the proposed extension scheme for microspheres imaging, which may be a crucial solution for facilitating the application of microsphere-assisted nanoscope and super-resolution algorithms.

2. Theory and simulation

2.1 Imaging mechanism of microspheres

Essentially in microsphere-assisted nanoscopy, the microsphere is treated as a micro-lens, which can discern the shape of a pattern with a minimum feature size of ∼λ/7 [29], where λ is the illumination wavelength. Although many studies have shown that the photonic nanojets effect of microspheres can focus light down to sub-diffraction-limited dimensions, it is still hard to directly adopt these models in system-level simulations [30]. To validate the imaging mechanism of microspheres, we start with geometric optics theory, using OpticStudio and investigating the focusing properties of the microspheres based on ray tracing. Considering there is a negative correlation between magnification and spherical aberration during the microsphere magnification imaging process [31], we select polystyrene (PS) microspheres as the micro-lenses to obtain a larger magnification and less distorted imaging simultaneously in the following experiments. The key simulation parameters of microspheres are set as follows: the diameter is 5 µm, the refractive index n is 1.592 for microspheres, and the wavelength of illumination is 450 nm. To be closer to the actual working conditions of our optical tweezers, water is selected as the environment medium in simulations, and its n is set to 1.333.

Figure 1(a) shows the simulation result, the gray lines demonstrate the scattered focus of the microspheres, and the focusing will converge as the diameter of the incident beam decreases, as the blue lines show. The final focus position after convergence is consistent with the derivation result of the paraxial formula [32]. This phenomenon is caused by spherical aberration, which is analyzed in the following section. However, for camera and eye imaging, irradiance is the decisive factor affecting microsphere imaging. In ordinary optical imaging systems, the imaging of a focused object may still be overwhelmed by strong stray light from elsewhere, unless adequate illumination of the object can be guaranteed [33,34]. Therefore, physical optical propagation analysis is performed at different locations in the ray tracing simulation, and it is found that the maximum irradiance is at 5.5∼6.5 µm distance from the center of the microsphere, as Fig. 1(b-c) shows. This is because the microspheres cannot perfectly focus the light as Fig. 1(a) shows. As the incident light is farther away from the optical axis, the luminous flux of the incident light will be raising, Meanwhile, the surface curvature of the microsphere at the incident point will also increase rapidly, making the light convergence point closer to the microsphere. Due to the polarization of light, its incident efficiency and refracted angle may also be affected by the incident angle. Ultimately, the light will focus at different positions along the central axis of the microsphere under the action of spherical aberration, manifesting as a change in irradiance. The maximum irradiance means most of the light is concentrated here, not the paraxial focus. Hence, the focal point of the microsphere for imaging should be set by the maximum irradiance position. The magnification of the microsphere is deduced to be around 1.63∼1.83× when the microspheres are close to the sample.

 

Fig. 1. The simulation results for the ray tracing simulation and FDTD solutions. (a) is the light propagation path based on ray optics simulation. It is clear that with the different incident positions of the light, the final convergence positions are also different. (b) is the physical optical propagation analysis in OpticStudio. The vertical axis represents the irradiance variation along the central axis of microspheres. (c) is the 2D planar distribution of irradiance perpendicular to the central axis of the microsphere. (d) is the standard 3D emulation of FDTD. A plane wave with BFAST mode is chosen to ensure the uniformity of the light field and the accuracy of the simulation. (e) is the light intensity energy change along the horizontal central axis of (d), and (f) is the intensity distribution in another dimension at the strongest energy density position. (g) shows the side-lobes formed after the ray passes through the microsphere, which may introduce artifacts in reality.

Download Full Size | PDF

Finite difference time domain (FDTD) solutions as a commonly used computational electrodynamics method in the study of photonic nanojets. The energy and power density changes of the light before and after passing through the microspheres can be finely reflected as Fig. 1(d-g) show. The ambient light field is set to be a uniform plane wave. Because only the effect of the microspheres on the light is considered, all boundary conditions are set to “PML” and “standard” mode, and the mesh step is set to 0.25 nm. The other key parameters are consistent with the OpticStudio, 2.5 µm for sphere radius, 1.592 for its refractive index, and 1.333 for background index. The simulation results are similar to the distribution of irradiance in Fig. 1(b) with more mesh accuracy. Slicing along the longitudinal axis at the center shows that there are still multiple energy-concentrating regions on the data curve and close to each other. Besides, it can be seen from (g) that the light forms many side-lobes after passing through the microspheres, not only on the central axis of the microspheres. In some conditions, the side-lobes are even higher than the main-lobe [35]. This may cause distortions, poor contrast and even artifacts in wide-field imaging.

However, the energy peaks are in different positions compared with the irradiance distribution. For this phenomenon, we think in OpticStudio, the ray trace simulation is incoherent, while the FDTD simulation is coherent. This means that the theory of photonic nanojets can reflect the energy distribution after light passes through the microspheres, but cannot directly act as the focus and explain the imaging mechanism of the microspheres [35]. For the optical microscope module, irradiance remains the key element in dominating imaging performance. As long as the irradiance intensity of the sample passing into the camera is not overwhelmed by surrounding stray light, the microspheres imaging can still be acquired. The range around the maximum irradiance point could be acted as the true focus of the optical imaging for microspheres, which is also confirmed by the magnification measurement results in Results.

As for the phenomenon of multiple convergence points after the light passes through the microspheres, this may be because of the refraction shiver caused by phase or polarization properties at different curvatures of the microsphere, which is another difficult problem that needs to be studied in the future.

2.2 Spherical aberration

In optics, spherical aberration is a key factor affecting the imaging quality of a lens, and it is certainly unavoidable for microsphere imaging. Under the influence of spherical aberration, the focusing of light is not perfect, as analyzed in the previous section. The imaging of microspheres will not only warp with deformation, but also loss of image sharpness. Thus, the ray tracing simulation is run again to model the propagation of rays through the microsphere and analyze the effect of spherical aberration on the imaging quality. Figure 2 shows the distortion simulation results of microsphere imaging and similar actual imaging. In the image simulation, the width of the microsphere imaging reaches 9.3 µm, which is much larger than the diameter of the microsphere. This phenomenon also exists in the actual experiment, but due to the excessive attenuation of light at the boundary, this part of imaging is extremely susceptible to ambient light. And through grid distortion analysis shown in Fig. 2(a and c), it can be seen that the largest distortion amount is also at the imaging boundary, close to 28%, which is unacceptable for microspheres imaging.

 

Fig. 2. The distortion simulation in OpticStudio and similar imaging for a square stage array. (a) is the imaging simulation of a 5 µm PS microsphere in water. The parameters remain the same as in previous simulations (see Visualization 1). (b) shows a confirmatory experiment in which a 5 µm PS microsphere is utilized to image a square stage array with the 2 µm side length of a single stage (see Visualization 2). The corners of the imaging area are not sharp because the stage is difficult to make a perfect square at the nano-scale level, as Figure S2 shows in the Supplement 1. (c) is the grid analysis extraction of (a). It is clear that as the distance from the center becomes farther away, the distortion of imaging also changes significantly. The largest distortion is located in the four corners, and the distortion rate is 27.82%. (d) is the display of the distortion rate at different positions. In the square area, the distortion rate is less than 4.4%.

Download Full Size | PDF

Whereas the central part of the microsphere imaging could ensure low distortion and strong irradiance intensity at the same time. As shown in Fig. 2(d), in the square area with a 3.5 µm length at the center, the average distortion is only 2.24% and the maximum distortion is 4.4%, which is acceptable in optical imaging. For microspheres, this acceptable area is truly small, only 37.4% of the full magnified imaging. It also is a key reason for limiting the popularization and application of microsphere-assisted nanoscopy.

To tackle this issue, we proposed a stitching scheme to expand this low distortion field. In this scheme, we simply get and use the low distortion imaging field of the microsphere, and achieve its expansion through a precise scanning process. During the image acquisition, the area of acceptable distortion only needs to be compared with the cross-sectional size of the microspheres. Because the imaging of the microsphere boundary and beyond the boundary contains a large amount of interference from the microsphere itself and its diffraction rings. Under this condition, up to 70% of imaging utilization efficiency can be achieved, which can guarantee the precision and efficiency of the image stitching process. We also modified an optical tweezers system to realize the scanning process of microspheres together with magnified imaging observations at the same time. Profit by the precise control capability of optical trap, the accuracy of the microsphere scanning process can also be guaranteed.

For peripheral imaging, it is still possible to increase the contrast by eliminating ambient light and the boundaries imaging of the microspheres themselves [36], which will be achieved in our other studies.

3. Methods

3.1 Optical tweezers-based microsphere manipulation system

The optical tweezer system used for microsphere-assisted nanoscopy adopts a kind of vertical structure, which utilize only one objective for optical trap generation and sample chamber illumination simultaneously. A high numerical-aperture objective (ZEISS, NA:1.2, 63×) is chosen to provide a stable optical trapping force for capturing and manipulating microspheres. Piezoelectric mirror (PM) can tilt in the microsecond range, which is advantageous in smooth rotating laser beams and moving the captured microsphere. A 3-dimensions piezo (PI) stage acts as a smooth motion device with nanometer level, to hold and drive the sample chamber. Some necessary clamping mechanisms are used to ensure the stability of the sample chamber and prevent unwanted jitter, which could provide a reliable basis for the microsphere-assisted imaging system. In order to achieve the longitudinal separation of the imaging plane and the altitude of the optical trap, two cameras are used in this system with independent lenses for separate imaging of the sample and optical trap. The main video camera (Manta G-507, monochrome) is mounted on a linear guide. The auxiliary camera is installed at another location in the optical path, solely for monitoring the horizontal position of the optical trap. Kohler illumination is used to provide a bright and even lighting environment in the sample chamber, and avoid artifacts such as glare and shadowing in the camera image. In addition, a 450nm bandpass filter is used to decrease the stray light and avoid chromatic dispersion, further improving the sharpness of imaging [37]. Many dichroic mirrors and beam splitters are placed to combine or separate the laser and illumination light. The rest devices of the optical tweezers system are installed on the horizontal base, including the PM mount, 1064 nm laser, and necessary beam expanders [38]. To obtain the maximum luminous flux and capture efficiency, both the illuminating light and the laser are precisely adjusted to cover the complete aperture of the objective.

3.2 Imaging setup

The vertical optical tweezer essentially contains a modern compound optical microscope with an infinity correction objective [39]. Benefiting from the simplified structure, this system can horizontally place the sample like the compound optical microscopes, not only can observe the transmission and reflection samples at the same time but also has very few restrictions on the material, shape, and weight of the sample. Thus, the microspheres captured by optical traps can be flexibly integrated within the imaging module to form a microsphere-assisted nanoscope. The linear guide and the telephoto lens (F = 350 mm) are the main difference from the conventional optical tweezers systems. Since the microsphere is captured by the optical trap, its imaging plane is actually below the optical trap. Therefore, it is necessary to adjust the focal plane of the camera to obtain a virtual image as shown in 3(b). The linear guide can provide a flexible movement for the imaging camera, expediently changing its imaging focus plane and thus realizing the bifocal surface. The use of telephoto lenses is also for more precise separation of laser and imaging planes. Since the imaging focal plane may exceed the optimal working distance during the adjustment process, the high numerical aperture (NA) objective is necessary to install, which can not only enhance the intensity gradient of the focused laser and achieve a more powerful traps force, but also maintain the resolution and sharpness of the microscope module.

In addition, the main camera is used for microsphere-assisted imaging of microspheres with a resolution of 2464 × 2056. While the camera supports the 12-bit grayscale depth in recording, the working mode is still set to 8-bit most of the time, since the image capture should be fast during the scanning process. The 12-bit mode is only used to capture static images with higher precision for data analysis. Figure 3(c) exhibits actual imaging of a standard resolution target (Ready Optics, Group 11 Element 6) by using 5 µm PS microspheres in the water environment. The imaging comparison before and after using the microspheres is shown in Figure S3 in Supplement. According to the datasheet, the bar and space width of the target are both 137 nm, while the diffraction limit of the imaging module without microspheres is 187.5 nm (see Supplement 1 for the complete calculation). This confirms that the modified optical tweezer system can obtain the smaller features below the resolution under the action of microspheres [40,41].

 

Fig. 3. Schematic view of the microsphere-assisted nanoscope and the sample chamber. (a) represents the schematic diagram of the vertical microscope module. (b) displays the imaging principle of microspheres captured by optical tweezers (OT). (c) exhibits a well-magnified imaging of a standard target dimension under 5 µm PS microspheres. The blue circle indicates the position of the captured microsphere. (d) shows the structure of the prepared sample chamber. All experiments are conducted in a water medium. The rays in (b) and (d) only indicate that the focus positions of illuminating light and the laser are separated.

Download Full Size | PDF

As a typical observation sample to verify the magnified imaging, Blu-ray discs have large-area nanoscale structures with uniform and constant periods, which is a common sample for the calibration of microsphere magnification. To prevent interference or diffraction caused by the data tracks, the Blu-ray discs are pre-recorded with data, further producing the tiny characteristics that offer the advantages of the best imaging surface selection and image stitching calibration. The protective film on the surface of the Blu-ray disc is peeled off after the data-burning process to provide direct contact between microspheres and the samples. Then, the disc is cut to an appropriate size as a base, and a coverslip and necessary support fabrics are used to form the chamber on this base (as shown in Supplement 1, Figure S4). Finally, PS microspheres are diluted with ultrapure water and injected into the sample chamber. And the magnified imaging can be easily reflected under the microspheres by a conventional focusing operation, as shown in Fig. 3(d). Driven by optical tweezers, the movement of microspheres is unlimited.

4. Results

4.1 Imaging plane and magnification

In the optical lens imaging system, the imaging quality and imaging magnification have significant relevance to the distance between the lens and the observed sample [24]. In the simulations, it is difficult to define the imaging focal point for the microsphere. Therefore, we plan to infer the true imaging focus position of the microspheres through actual experimental phenomena. In order to find the best imaging plane and verify the real magnification mechanism of the microspheres, scanning in the Z-axis direction is first performed. The optical tweezers operate to move the microspheres to the recorded area, and then hold the microspheres firmly on the Blu-ray disc to prevent the imaging shake during the scanning process. At the same time, the PI-stage is slowly raised in step mode, and the cameras work synchronously to acquire the magnified imaging of the microspheres. During this process, the position of the camera and the objective remain motionless, and the magnification of the microsphere imaging is obtained by comparing the periods before and after the data tracks are magnified.

After acquiring a large number of microsphere imaging, the location of the captured microsphere is marked as the region of interest (ROI) and cut to obtain a series of imaging slices. Then utilizing two-dimensional fast Fourier transformation (FFT) to perform a frequency analysis on each slice and obtain the period of the data tracks imaged at different positions. By comparison with the frequency analysis of the original disc image, it can be found that the magnification of Blu-ray disc imaging is increased slightly with the elevation of the sample stage [28]. As shown in Fig. 4(c-e), by using the same measuring scale, the period of the data tracks is continuously magnified as the focal plane moves away from the microsphere. The complete magnification change during the scanning process is shown by the red line in Fig. 4(g). Although the maximum magnification could boost up to 1.8×, the sharpness of the imaging has dropped significantly because the magnified image plane is no longer in the best imaging plane of the microscope module, and the light is unavoidable diverging when it passes through the microspheres.

 

Fig. 4. Magnified imaging of microspheres and its mechanism analysis. (a) demonstrates the schematic diagram of the principle of microsphere imaging. The rays only indicate the separated positions of illuminating and laser focus. The aperture of the objective is actually much larger than microspheres, and the illumination light can cover most of the surface of the microspheres. (b) is the imaging of the Blu-ray disc and a defocused microsphere. (c-e) are the microsphere-assisted imaging of the microsphere in (b) when the sample stage upraises 1, 1.65, and 2.22 µm, respectively. To directly show the magnification effect, these images use the same scale as (b). The lines in (f) are the digit values of the adjacent data points acquired from the imaging shown in (b-e, marked with dashed lines), and the proportions of these waveforms are consistent. The red line in (g) represents the magnification of the Blu-disc imaging by the microspheres in different planes, and the blue and gray lines represent the FFT intensity at 1/315.65 nm^-1 of the magnified imaging and original imaging, respectively. The power spectrum of the magnified imaging is significantly wider and larger than that of the original imaging. This phenomenon indicates that the microspheres can simultaneously improve the DOF and identifiability of the data tracks in imaging.

Download Full Size | PDF

The sharpness of the imaging does not linearly with the position of the sample like the magnification, and the magnified imaging only appears in a specific interval. This interval can be regarded as the DOF of the microsphere lenses, like other optical imaging devices. However, the microspheres have extremely high spherical aberration, which increases the acceptance of the circle of confusion. Coupled with the limitation of the imaging view scope, the DOF of microspheres will be significantly increased. To find the best imaging plane for the long DOF, all the cut-images in the Z-axis scanning process are subjected to FFT analysis again. Different from the magnification measurement, the power spectra are extracted by the FFT intensity at the period of data tracks from all cut-images. As the blue line shows in Fig. 4(g), the amplitude of the curve represents the FFT intensity of the target period at different positions, which could reflect the imaging quality and contrast of the microspheres. Thus, the cut-image corresponding to the maximum power point is regarded as the sharpest magnified image [42], as shown in Fig. 4(d), and its plane is regarded as the best imaging plane. In this plane, the imaging of microspheres will have the highest contrast and sharpest quality, regardless of its horizontal displacement. As the consequence, the clearest magnified plane of the microsphere is located at 1.65 µm below the surface of the Blu-ray disc, and the magnification at this plane can reach 1.62× stably. This is slightly different from the analysis in the discussion, possibly due to a slight change in the refractive index of water due to temperature, but still could confirm that the best imaging focus of the microspheres should be the area of high irradiance, not the paraxial focus or nanojets focus.

According to the calibration before the experiment, the plotting scale of the microscope module without microspheres is 1000 pixels: 29.5 µm, that is, the resolvable size of a single pixel is 29.5 nm. On the best imaging plane, the magnification of microspheres is 1.62× and its pixel resolution can be boosted to 18.21 nm in theory. From this it can be deduced that the period of data tracks is 315 nm, the data points width is 145 nm, and the length of the different data points is 157 nm and 364 nm, respectively. To verify the accuracy of the magnified imaging, the same Blu-ray disc with the data tracks is characterized by the AFM and SEM systems. As the result, the period is 310∼330 nm and the pit width are 100∼150 nm, which is similar to the measurement in Fig. 4. It is noted that since the data recorded on the Blu-ray disc is only a change in the reflection property, no actual morphology change has emerged, and the AFM and SEM system cannot reflect the data points as true as possible (as shown in Supplement 1, Figure S5). By consulting the disc datasheet, the accurate period of the track pitch is 320 nm, the pit width is 130 nm, and the minimum length of a data point is 150 nm. This proves that the microsphere-assisted nanoscope can achieve the minimum characteristic size measurement of ∼150 nm.

4.2 Image stitching scheme

Although the microspheres have an obvious magnification effect for the imaging of Blu-ray discs, the imaging view scope is limited because of its tiny size. However, under the action of the optical tweezers, the 5 µm microspheres can move freely at the drive of the optical trap to achieve XY-scanning. This means that scanning of the Blu-ray discs and stitching of the limited view to achieve large-area imaging is feasible with a well-designed path of the optical trap [20]. In order to reduce the influence of dynamic or static friction on the trajectory, the captured microsphere needs to be further separated from the disc surface. Hence, the captured microspheres are lifted 400 nm away from the disc surface under the action of the optical trap. As the distance between the captured microsphere and the sample surface increases, the imaging plane of the microspheres is further away from the sample surface, so the camera needs to move forward on the linear guide to achieve imaging deblurring and ensure imaging contrast. Despite the magnification will continue boosted when the distance increase within a range, the dissipated irradiance intensity will also be submerged easily in the environment the same as the Z-scanning process shows.

In the optical tweezers system, both PI stage and PM can be utilized to achieve precise scanning. However, the motion of the PI stage generally causes the entire sample chamber to move, the internal solution and microspheres will be perturbed under the effect of inertia, and thus affecting the capture efficiency of the optical trap. In addition, the fluttering microspheres may fall into the optical trap and influence the imaging of the existing microsphere. Therefore, we finally choose to drive the microsphere by rotating the PM, and the PI stage is only used for large-scale movement of the field of view of the imaging system. Figure 5(a) shows a typical XY-scanning process, snake mode, which could avoid the large-distance movement of the optical trap, and maintain the stability of the captured microsphere. Considering that the distortion in the central area (the square acceptable area with a length of 3.5 µm) of the microspheres is acceptable from the previous analysis results, the image acquisition interval is set to 1.75 µm to obtain a sufficient overlapping view for subsequent fusion and stitching. Beyond this, there are a total of 8 lines in the scanning process, and each line contains 9 images with 50 ms of the acquisition interval. As a result, a total of 72 images are acquired and cover more than 16 × 12 µm² within 3 seconds, which can be further improved by replacing the high-speed camera. Since the raw image captured by the camera contains a lot of high-frequency noise signals, which is fatal to the algorithm of following image stitching, all images are Gaussian filtered before the stitching process.

 

Fig. 5. The scanning schematic and the stitching result of the burned Blu-ray disc. (a) represents the stitching process of two adjacent scan pieces in the X-axis direction. The stitching process on the Y-axis is similar. (b) shows the scanning trajectory of the microsphere captured by the optical trap (see Visualization 3). (c) is the original image of the Blu-ray disc. (d) demonstrating the stitched image of the captured microsphere after the scanning process. The image has been adjusted for contrast. Through FFT analysis, the magnification of the stitched image reaches 1.92×. The contrast of (c) and (d) is uniformly adjusted, and the original contrast is shown in Figure S6.

Download Full Size | PDF

Similar to the Z-axis scanning process, the ROI area tracks the approximate grid coordinates of the optical trap, cutting the magnified area into pieces. Because the brightness and contrast of the magnified imaging are not uniform, there are many blemishes existing especially on the edge of the pieces and are hard to eliminate. Besides, the structure of the data points of the Blu-ray disc is highly repeated. All of this cause the artificial intelligence overlap fusion algorithms to fail in finding the correct characteristic to attach them best. Therefore, we improved a general stitching scheme [43], and added the liner feather blending to smooth the edges of image features and maintain the accuracy of the composed graph.

Firstly, we correlate the ROI coordinates with the optical trap positions, establishing a unified reference frame. Then, all image pieces are imported according to approximate grid coordinates and grouped by row and column. In this reference frame, the translational offsets between two adjacent image pieces will be calculated by using Fourier transform-based phase correlation method (PCM). In order to reduce the impact of data point repeatability, we limit the region of PCM to only compute the intersection of the image pieces, as shown in Fig. 5(b-ii), the red rectangles. After that, the grid coordinates are corrected by the translational offsets to initially realize the image stitching [25]. However, this stitched image has an obvious “mosaic” phenomenon. This is not only due to the uneven brightness in the overlapping areas but also because of the distortion of microsphere imaging, introducing additional differences at the edge of image pieces. Since the image acquisition interval is set according to the size of the acceptable distortion area, this distortion can be eliminated by further algorithms. Considering that the distortion rate gradually increases away from the center, the overlap areas can be blended like a feather crossing to correct the imaging, as demonstrated in Fig. 5(a b-iii). The weighting factor for feathering is linear with the distance of the current pixel from its slice boundary. Through this scheme, both uneven brightness and minor deformation in overlapping areas could be well corrected, as Fig. 5(b-iv) shows. Finally, a larger magnification area is formed by arranging and stitching these pieces together. Due to the inconsistent sensitivity of PM in the two axes, the parameters are slightly changed, thus the stitched image is not an ideal square. After the re-calibration through the period of data tracks, the magnification can be obtained up to 1.92×. With the enhancement of large field imaging and fast scanning, this method is a great improvement for microsphere imaging.

4.3 Large field imaging

Since AFM and SEM cannot characterize the data points of Blu-ray discs, the comparison of magnified imaging with the original structure is hard to be achieved. Therefore, a standard through silicon via (TSV, the model MLH230/200/460-15 × 15, made by SHNTI-EU) is used to verify the accuracy of large-field magnified imaging. On the surface of this silicon wafer, holes with a diameter of 230 nm are regularly arranged in regular hexagons with a period of 460 nm. The same microspheres and acquisition interval are used during this scanning process. Therefore, the same approximate grid coordinates can be led in to improve the accuracy and efficiency of image stitching. The result is shown in Fig. 6. Without the fine decoration, the stitched image can directly reflect the original appearance of the sample. Similar to the Blu-ray disc sample, the magnification of the large field imaging reaches 1.92× when the microsphere is raised 400 nm higher than the sample surface by the optical trap. By analyzing the light intensity along the via-hole array on the standard TSV template, it can be seen that a lot of distortion exist in the original image, while the corresponding details are preserved after the microspheres imaging, which is closer to reality as Fig. 6(c-f) shows.

 

Fig. 6. Scanning and stitching results for the standard TSV template. (a) is the original image of the TSV template, which is scanned by the same method as the Blu-ray disc scanning process. (b) demonstrates the stitched image of the TSV array. The boost in brightness and contrast may be due to the smooth surface of the silicon wafer, which is a perfect mirror for reflecting back the illumination light gathered by the microspheres, thereby boosting the contrast of the imaging. (c) and (d) illustrate the details of the TSV structures in (a) and (b), respectively. Gray values have been remapped to emphasize the imaging details. (e) represents the SEM result of the TSV array. (f) is the light intensity comparison along with a line array before and after using microspheres. Although the non-uniform illumination causes the TSV structure to be imaged unevenly at different viewing angles, the template imaging scanned and stitched by microspheres has higher reducibility.

Download Full Size | PDF

These results demonstrate the high efficiency and accuracy of the image stitching method. Additionally, the performance of stitched images can also be further improved by deconvolution processing. It is proved that the proposed microsphere-assisted nanoscope is essential for obtaining clearer and larger magnified imaging of microspheres, and crucial for its promotion and application. Although there are still some problems that have not been solved, such as non-uniform lighting and unpromoted horizontal resolution, this system still serves as a fundamental study deeper reveals the imaging mechanism of microspheres, and thus optimizes imaging performance; the FOV extension scheme also provides a new way of thinking for intuitive optical imaging of nanostructures. In future research, we will focus on how to precisely locate the illumination within the microspheres and eliminate the interference of ambient light, further achieving cleaner imaging of microspheres with higher magnification and resolution. Its feasibility has been verified by the TSV observations.

5. Conclusion

In this paper, we further reveal the imaging mechanism of microspheres and establish a microsphere-assisted nanoscope on an optical tweezer. The simulations and experiments confirm that microspheres do not have a single accurate focus plane, instead the irradiance is a key factor affecting the contrast and magnification of magnified imaging. Besides, the tiny size of the microsphere limits its equivalent aperture and thus increases the DOF of imaging. With the bifocal plane of the modified system, the sharpest imaging plane is found and defined. In order to prevent spherical distortion, we extract the acceptable area from the microsphere imaging during the precise XY-scanning process. Then, the extracted imaging is immediately stitched together to achieve the extension of the imaging view scope of microspheres. This scheme can efficiently achieve a scope of 16 × 12 µm² magnified imaging with ∼2× magnification in 3 seconds, which can be further improved by optimizing the optical trap path or changing high-speed cameras. Unlike AFM and SEM, this system can not only scan physical structures but also observe the visual appearance of samples, such as burned Blu-ray discs. In summary, the combination of optical manipulation and microsphere-assisted nanoscopy provides a new powerful solution for the measurement or observation of the nano-scale morphology and appearance characteristics, such as MEMS industrial inspection or fluorescence detectors.