1. Introduction

Optical microscopy is a powerful tool for observing and analysing the structure and properties of samples at the micro/nano scale. The application of optical microscopy has extended from biology and medicine to materials science, robotics, and other fields [1–2].

With the rapid development of related applications in recent decades, the demand for super-resolution, real-time, non-destructive observation of microscopic sample structures has increased; however, conventional optical microscopes have gradually failed to meet this requirement because of blur imaging as a result of the following factors: 1) Owing to optical diffraction of the source wave, a point on the surface of an object forms an Airy disk on the image plane, rather than a focused point, such that the resolution of optical systems has a theoretical limit of approximately half the wavelength used for illumination [3–5] . 2) Owing to the limit of the depth-of-field (DOF) of a microscope, when the thickness of a sample is larger than the DOF, the parts outside the focal plane are defocused, and these defocused images are superimposed on the image plane, resulting in a blurry superimposed image [6–8].

To reduce the influence of these blur imaging factors and achieve a high-resolution image at the micro/nano scale, the mechanism of light intensity diffusion during microscope imaging must be analysed. In theory, the intensity distribution function generated by a point source is called a point spread function (PSF) or blurring kernel, using which a high-resolution image could be reconstructed through deconvolution with the blurry image. Therefore, it is very important to estimate the blurring kernel parameters as accurate as possible for the purpose of constructing a high-resolution image. Although many researchers have proposed some powerful methods for relieving the effects of out-of-focus light and diffusion light for many microscope systems [9–12], until now, precisely measuring the blurring kernels of a practical high magnification microscopy is still a problem, because of complicated imaging process and various dynamic image features at the micro/nano scale.

Therefore, in this paper, we proposed a method to extract blurring kernels and reconstruct super-resolution images based on StyleGANs. Our approach is novel in several ways. First, the improved StyleGAN model was trained using the optical image set of the Gaussian light source, and an ideal blurry image generation model based on StyleGANs was obtained. Second, a microscopic image feature analysis system, including an ideal blurry image generation module, a blurring kernel extraction module, and an image deconvolution module, was designed. Through the automatic learning of the image features with changed parameters, a series of ideal blurry images that obey the law of blur imaging were generated and the blurring kernel of the optical imaging system was extracted. Finally, a super-resolution image-reconstruction method with respect to the images of the single wavelength Gaussian light source and large-scale visible light images is proposed based on the learnable convolutional half-quadratic splitting and convolutional preconditioned Richardson (LCHQS-CPCR) neural network model. Experiments with dynamic samples were conducted, and the results exhibited that our proposed method obtains the blurring property of an optical system and can reconstruct higher resolution images at the micro/nano scale.

2. Related work

Currently, to describe the blurring features of an optical imaging system, there are three main methods to obtain its PSF: mathematical, analytical, and experimental methods [13–17]. The mathematical method depends on a fixed formulation of the PSF developed using a physical model of the light propagation route [18,19]. However, for advanced microscopic optical systems, the complexity of the system dictates that the aberrations cannot be fully considered in the mathematical method, resulting in a bias in the calculation of PSFs. The analytical method uses a blind deconvolution process to estimate the PSF parameters of the optical system, based on many iterations until some optimisation function is satisfied [20,21]. However, the precision of the estimated PSF is highly related to the form of the optimisation function, and it is difficult to define a general optimisation function for all optical systems. Moreover, the iteration process is time consuming. For experimental methods, PSF is generally measured using embedded microbeads with fluorescent properties in the optical cement at different heights, or fixed fluorescent microbeads on an inclined surface [22,23]. In this case, a single fluorescent microbead can be considered as a single-point light source. Therefore, the measured PSF is obtained by measuring the intensity distribution in the microbead images. This involves a challenge as it is difficult to precisely control the position of a microbead in optical cement, which leads to inaccurate measurement results. However, when the problem of microbeads fixation is well resolved, human intervention in the optical cement can disrupt the local refractive index in the cement, resulting in an inconsistent refractive index in the optical medium, and thus lead to measurement errors.

Further, deep learning is a representation learning method that can work directly and learn autonomously with raw data. It is exceptional at processing large-scale and high-dimensional image data and discovering its hidden data structure, thus making it capable of performing various tasks in the field of image analysis. In the last ten years, a series of networks have been applied to image deblurring, including convolutional neural networks (CNNs) and deep CNN networks (DNCNNs) [24–27]. Generative adversarial networks (GANs) were first proposed to generate more real data by learning [28], followed by a large number of variants of different GANs, such as wasserstein GANs (WGANs) [29], deep convolutional GANs (DCGANs) [30], WGAN gradient penalty (WGAN-GP) [31] and least squares GANs (LSGANs) [32]. Recently, the deblur GANs have been used to achieve blind motion deblurring [33]. Compared to the deblur GANs, the super-resolution GANs (SRGANs), enhanced super-resolution GANs (ESRGANs), StyleGANs, and improved StyleGANs have achieved better results for noise removal and image filtering, because these methods adopted the perceptual loss to optimize their frameworks [34–37]. However, they are difficult to use in the deblurring process of optical microscope imaging, where the blurring property of the images is strongly related to the system parameters, because these deblurring techniques are to generate more real data by learning the main properties of noisy images, rather than extracting of the blurring property in a microscope system.

In a word, measuring the blurring kernel of an optical system is complicated and time consuming with current methods. Furthermore, their measurement precision is easily influenced by various internal and external factors during imaging process. StyleGANs-based methods have potential to achieve high-revolution imaging, however, without extraction of the blurring kernel which describing the blurring property of a microscope system, they are difficult to reconstruct a high-revolution image at the miro/nano scale.

3. Blur imaging and deep learning model

3.1 Basic principles of blur imaging

The blur imaging process of an optical system can be represented by a convolution process denoted as,

In the practical imaging process, the variation of PSF is not limited to the 2D plane perpendicular to the optical axis, but has a three-dimensional spatial variation characteristic containing the direction along the optical axis under the influence of complex factors, such as optical diffraction, defocus and changes in camera parameters. Therefore, the study of PSF is extended to the 3D space, and its spatial convolution process can be written as:

Normally, the distribution of h is considered to have the same scale variation in the x and y directions, then σ_x= σ_y= σ_xy. That means the blurry image of a source point is a round spot, and the relationship between σ_xy and the radius of the spot is given by,

More in general, one can approximate the PSF with other functions, as long as they satisfy the following property:

The properties mentioned above correspond to having a lossless optical system, i.e. an optical system such that all the energy emitted by a source point is transferred to the image plane. According to Eq. (2), reversely, the clear image can be obtained using the method of deconvolution, the process can be described as:

From Eq. (6), we can find that when a blurry image of a sample is known, the deconvolution process depends on the accurate acquisition of the PSF. However, as discussed in Section 2, current mathematical, analytical, and experimental methods all have problems to obtain an accurate and general PSF during imaging of an optical microscope. Therefore, deep learning with independent analysis and learning ability is introduced in this paper.

3.2 StyleGAN and its optimization

StyleGANs are a more effective high-resolution image generation method, as it is born out of GAN, and inherits the idea of generativity and advisability. The main contribution of StyleGANs is to rebuild the traditional generator that directly feeds the effective information hidden code representation to the input layer. The comparison between the design of a traditional generator and the StyleGAN generator is shown in Fig. 1, where AdaIN is the adaptive instance normalization, PixelNorm is the pixel normalization layer, FC is the fully-connected layer, Conv is the convolutional kernel, W denotes an intermediate latent space, w denotes the learned weights, A denotes a learned affine transform from the space W, and B denotes a noise broadcast operation. First, the style based generator nonlinearly maps the hidden code through an 8-layer full connection layer n: Z → W. Then, the AdaIN module performs the function of affine transformation, and the encoded information created in the nonlinear mapping layer is transmitted to the synthesis network. Finally, the image is generated by StyleGAN controlled by w and AdaIN.

 

Fig. 1. Comparison between traditional generator structure (on the left of the dash line) and StyleGAN generator structure (on the right of the dash line)

Download Full Size | PDF

However, most of the images generated by the original StyleGAN model possess characteristic artifacts, which both exist in the final image and the middle features of the generator. To generate a high-resolution image without influence of all types of external noise, it is necessary to tackle this problem. The optimised StyleGAN model changes the part of the generator's architecture where some redundant operations are deleted at the beginning, as shown in Fig. 2, where b denotes the deviation, and Demod is the weight demodulation operation, and the addition of deviation b and weight B of Gaussian noise are moved to the outside of each “style” area, while the standard deviation of each feature map is adjusted to eliminate the operation of calculating the mean value. Therefore, the new architecture enables the weight demodulation operation to replace the original AdaIN operation, and applied this demodulation operation to the weights associated with each convolutional layer.

 

Fig. 2. StyleGAN generator structure (on the left of the dash line) and its optimization (on the right of the dash line)

Download Full Size | PDF

The flow of weight demodulation is as follows:

The main operation of weight demodulation involves moving the scaling parameters into the weights of the convolutional layers, thus allowing better parallelization of the computational paths. Therefore, the optimization method enabled smooth and unhindered transfer, inheritance, and continuous refinement of image features between different resolution layers, while the characteristic artifact problem was solved entirely.

4. Super-resolution image reconstruction based on blurring kernel

The proposed blurring kernel extraction and super-resolution image reconstruction method based on the StyleGAN model is mainly composed of three modules: the ideal image generation module, blurring kernel extraction module, and clear image reconstruction module. A functional block diagram of the proposed method is shown in Fig. 3. Our method consists of three modules including the ideal image generation module, the blurring kernel extraction module and the clear image reconstruction module. Firstly, the image collection of a Gaussian beam is used to train the generator and then the ideal beam images are generated. Secondly, the blurring kernel extraction module processes the ideal beam images to obtain the blurring kernels of the optical system at different depths. Finally, the obtained blurring kernels are used to reconstruct the high-resolution images through deconvolution.

 

Fig. 3. Functional block diagram of our method which is composed of three modules.

Download Full Size | PDF

4.1 Training the image generation model

To obtain the properties of the PSFs of an optical system under different conditions, it is reasonable to capture an image of a point light-source and further analyze its energy distribution property, especially in the central Airy disk. However, owing to various internal and external factors such as, defocus, and diffraction, it was difficult to obtain a stable and noiseless image of a light source with a high magnification microscope. Moreover, the implicit optical features contained in the optical light-source image datasets are complex and varied. This is quite different from the image structure features of image datasets otherwise processed by StyleGANs such as real human faces and animals. Therefore, directly using the existing image data set for migration training of our model will not only affect the training efficiency, but also cause the effect of the generated images to deviate significantly from the physical characteristics. In this study, complete retraining was adopted to obtain a new and more comprehensive training model, which represents the time-varying optical characteristics of the images of a light source.

4.1.1 Pre-processing of data

The optical light source image dataset used in this study was not a public dataset; however the sequence images were collected using a laboratory microscope optical system. Therefore, pre-processing is required to eliminate the unqualified collected images, including (1) Images with lower resolution; (2) Images with too strong or too weak exposure; (3) Image with low contrast and darker brightness; (4) Images with incomplete Airy disk and obvious defects. In addition, the StyleGAN network structure requires that the processed images must be square, so all the image sizes are uniformly cropped to 512×512.

Second, in order to improve the generalisation ability of the model, this study uses the data enhancement of mirror flipping and centre cropping for each image in the dataset, which increases the scale of the dataset, and improves the generalisation ability of the model.

Following the data pre-processing and data enhancement, the dataset must be converted to a format readable by StyleGAN and commands must be input to the Linux server to complete the creation of the dataset.

4.1.2 Model training

Training the StyleGAN model is the adversarial training process of alternating between the generator and discriminator until a Nash equilibrium is reached, at which point the discriminator is unable to discriminate the trues from the fakes to generate the ideal images. The adversarial training process is represented by first, generating a batch of random latent codes to generate a batch of fake images through the generator network G, and later extracting a portion of real images from the training dataset. The true and fake images were provided to the discriminator network D separately to calculate the scores, and the global cross-entropy of these scores was calculated. The global cross-entropy was combined with the regularisation term as the loss function of the discriminator network D. The loss function of the generator network G only considered its own cross-entropy and was further combined with the perceptual path length regularisation term as the loss function of G. The entire optimisation process of the StyleGAN model involves ensuring that the loss functions of D and G obtain the minimum values through gradient descent, simultaneously. The entire training process in this study has been divided into the following steps:

4.2 Blurring kernel extraction module

The main objective of the blurring kernel extraction module involves extraction of the blurring kernel that causes the microscopic image to be blurry. The blurring kernel extraction process is illustrated in Fig. 4. First, the generated image was transformed into greyscale. Next, the midpoint of the Airy disk was taken as the centre to intercept the square image block. To adapt to the CNN network in the image deconvolution module, we called the resize function in the opencv2 library to adjust the size of the intercepted image block to 17×17, and extract it as the blurring kernel.

 

Fig. 4. The blurring kernel extraction process

Download Full Size | PDF

4.3 Image deconvolution module

Unblinded image deblurring is usually expressed as a linear least square problem. Since a blurry image may correspond to many clear images, image blurring is an ill posed problem. In order to solve this problem, the variational method based on the half quadratic splitting (HQS) uses the penalty least squares method to enroll a priori knowledge into the solution set, and solves a least squares problem by iteratively calculating a neighbor operator. In this study, the pre-trained LCHQS-CPCR model was introduced to complete the deconvolution process. It is an end-to-end model, that is, it performs pre-training, extraction of image prior features, image deconvolution and deblurring independently.

During deconvolution, the LCHQS-CPCR model considers the known blurring kernel as the precondition and replaces the traditional linear preconditioner through the CNN network to ensure that the image parameters can be effectively shared. As compared to the classic fast Fourier transform and conjugate gradient descent method, the LCHQS-CPCR method observed a significant improvement in accuracy and speed. The network structure of the deconvolution module is shown in Fig. 5. First, the network deconvolves the blurry input image through the deconvolution module initially, and further convolves the vertical and horizontal gradients to obtain an image with less noise. Subsequently, the deconvolution process was performed on the denoised gradient image to obtain a clear image. The gradients of the clear image was the input of the convolutional layer in the next iteration. After repeating the above steps three times, we finally achieved a deblurring image.

 

Fig. 5. Network structure of the proposed image deconvolution module

Download Full Size | PDF

In this study, we used the LCHQS-CPCR model trained in advance on the BSD500 dataset for migration learning. The training process can be summarised in two parts, which are briefly described here. First, the exact estimated output of each iteration of the supervised LCHQS-CPCR algorithm is shown in Eq. (10–14).

Second, the optimisation was performed using the Adam optimizer with certain predefined parameters, such as the learning rate set, iteration number, and batch size. The network was further trained by supervising the final output of the LCHQS-CPCR on the same training dataset, optimised using the Adam optimizer without per-layer supervision as well.

5. Experiment

The experimental platform is an independently built optical microscope imaging system, as shown in Fig. 6. The microscope was from Navitar, with a maximum magnification of 12X and a working distance of 34 mm. The objective lens was a compound achromatic APO lens. The magnification was 10x, the numerical aperture was 0.28, while the resolution was 1um, and the DOV was 3.5um. The charge coupled device was s Canadian PointGrey 1394B. A three-dimensional nano-positioning piezoelectric ceramic platform NPBIO300 platform from NPOINT was used. The stage was a three-axis XYZ control with a closed-loop travel of 300 µm, an open-loop travel of 360 µm, a positioning accuracy of 2 nm, a frequency of 200 Hz, and a rectification time of 20 ms.

 

Fig. 6. Our self-made microscopy optical imaging system.

Download Full Size | PDF

Because it is difficult to obtain the blurring kernel in optical systems, we used a Gaussian beam as the observed sample and extracted the blurring kernels from the images of the beam. Because the green color was located in the middle of the visible spectrum, a green Gaussian light source with a wavelength of 532 nm was used in this study. The specific environment and configuration version are listed in Table 1.

Table 1. Our environmental parameters

View Table | View all tables in this article

5.1 Ideal image generation

First, 1530 images of the Gaussian beam at different depths were collected using our system, and the depth step size Δd was 2 μm. When Δd = 0 μm, it means that the object distance is equal to the ideal object distance. Some of our captured images are shown in Fig. 7, where it can be observed that the intensity distribution in each image is not stable and there is some random noise which influences the precision of the blurring kernel.

 

Fig. 7. Original Gaussian beam images at different depths.

Download Full Size | PDF

Further, we input the captured images into our image generation model and set the duration of the training kimg to 1000. The parameter kimg is the time taken by the discriminator to recognize one thousand real images. The images generated by our generation model during the training process are shown in Fig. 8. As observed, as the number of iterations increasesd, the generated image got closer to reality and the image quality improved substantially. Figure 8(a) is the initial image generated by the generator; Fig. 8(b) is the image generated when kimg = 80, where only a piece of contour exists; Fig. 8(c) is the image generated when kimg = 160, where the contour of the diffraction ring begins to take shape, and some orange light bands appear at the edges of the images; Fig. 8(d) is the image generated when kimg = 240. At this time, it can be observed that StyleGAN has misunderstood as some cross-shaped streamers appear in the centres of the generated images, and Fig. 8(e) is the image generated when kimg = 500, where StyleGAN successfully got out of the misunderstanding after learning more correct samples. The shape of the diffraction ring can be clearly observed, and the outline of the diffraction ring is more obvious when it is close to the centre; Fig. 8(f) is the image generated when kimg = 1000. At the time, the texture and details of the diffraction ring are very realistic from the perspective of the human eye, and the Airy disk becomes regular and clearly visible.

 

Fig. 8. Generated images during training process of the generator.

Download Full Size | PDF

The loss curve of the generator network and the discriminator network during the training process is shown in Fig. 9. Figure 9(a) shows that the loss of the discriminator gradually stabilized when kimg = 540. In Fig. 9(b), the loss curve of the generator presented a downward trend and the loss gradually stabilized when kimg = 520.

 

Fig. 9. Loss curves of the generator network and discriminator network.

Download Full Size | PDF

To analyse the performance of the trained image generation model, we introduced the Fréchet inception distance (FID), inception score (IS), and kernel inception distance (KID) to evaluate the performance of GANs and their different various variants. The scores are presented in Table 2. According to Table 2, the initial value of FID before training was 523.3526; however, it dropped to 9.0748 following the training. The lower FID score indicates that the distribution of the beam images generated by this model is approaching that of the ideal Gaussian beam images. While the initial value of IS before training was 0.0160, it rose to 1.2982 after training. A larger value of IS indicates better clarity and diversity in the generated results. The value of KID decreased from 0.8377 before training, to 0.0072 after training, thus indicating that the difference between the generated data and real data was reduced. These scores indicate that the implied features of the Gaussian beam images are well learned by the pre-trained image generation model, and the values of the given evaluation metrics verify the image quality and diversity of the model.

Table 2. Model evaluation results

View Table | View all tables in this article

Finally, we obtained a number of generated images of the Gaussian beam at each depth after setting the truncation value which represents the density of the generated images. In this study, we set the truncation value to 0.5, and for each depth, we obtained 50 generated images. We randomly selected one image from the generated image sequence at each depth, and the images are shown in Fig. 10, where it can be observed that the subtle random noise in the original beam images shown in Fig. 7 have disappeared.

 

Fig. 10. The generated Gaussian beam images at different depths.

Download Full Size | PDF

5.2 Extraction of the blurring kernel

In this model, we selected an appropriate image at each depth from the generated images and extracted the blurring kernel of this image that of the system at the position. First, the selected image was greyscale, and then the center of the Airy disk in the image was roughly positioned through gradual interception. Finally, the Airy disk of the energy distribution was extracted as the blurring kernel. The size of the blurring kernel was 17×17 in this study. The greyscale image is shown in Fig. 11(a), and its central part that includes the Airy disk, is shown in Fig. 11(b); the extracted blurring kernel is shown in Fig. 11(c), where it can be observed that the Airy disk is accurately obtained.

 

Fig. 11. Extraction of the blurring kernel from the generated image.

Download Full Size | PDF

To further verify that the blurring kernel extracted from the generated image conforms to the laws of optical wave travelling and distribution during optical imaging, we used the theoretical model described in [26] to calculate the light intensity distribution of the Gaussian beam on the image plane vertical to the optical axis. Its equation can be expressed as Eq. (15). Further, we extracted the blurring kernels from the original image at Δd = 0 μm and its corresponding generated image, and calculated their average intensity curves through the centre of the intensity peak. The results are shown in Fig. 11.

 

Fig. 12. Intensity distribution comparison in the generated image and the original image.

Download Full Size | PDF

As shown in Fig. 12, as compared to the light intensity distribution curve of the original image, the intensity distribution curve of our generated image is closer to the theoretical intensity curve. Additionally, and it is smoother and possess a stronger symmetry of the optical axis. Because most of the energy of the light intensity distribution is concentrated in the Airy disk, the main peak of the distribution curve was chosen for further research, and the root mean square error (RMSE) relative to the theoretical model was calculated. The RMSE between the generated distribution curve and the theoretical curve was 0.071, while the RMSE between the original distribution curve and theoretical curve was 0.133. This proves that the image generated by our module is closer to the theoretical description of the light intensity distribution, and that the generated optical image inherited the entire characteristics of the optical images, and improved the regularity of the light intensity distribution by removing the random noise in the original images.

5.3 Image deblurring with the extracted blurring kernel

5.3.1 Deblurring the Gaussian beam images

After extraction of the blurring kernels at different depths, we used these kernels to reconstruct the clear images of the Gaussian beam at different depths using our clear image reconstruction model. The results and corresponding blurring kernels are shown in Fig. 13. The quality evaluation of the reconstructed Gaussian beam images shown in Fig. 13, the original images shown in Fig. 7 and the generated images shown in Fig. 10 were compared with respect to the image entropy.

 

Fig. 13. Reconstructed images of the Gaussian source light with corresponding kernels at different depths.

Download Full Size | PDF

The results are listed in Table 3, through which it can be observed that the image entropy values of the generated images at different depths were smaller than those of the corresponding original images because some random noises were removed from the original images during the image generation process; While the image entropy values of the reconstructed images were higher than those of the generated images and the original images. This indicates that our deblurring model reconstructing the clear images with the blurring kernel was effective because we reconstructed the images from the respect of blurring theory, rather than filter noises.

Table 3. Model evaluation results

View Table | View all tables in this article

5.3.2 Deblurring the microbead images

After obtaining the blurring kernels of the optical system from the images of a single wavelength Gaussian beam at different depths, we further extended our method to deblur the blurry images of some samples under the illumination of a visual light, because the green color was located in the middle of the visible spectrum. In our experiment, the images of the dynamic microbeads in solution were collected by the system, where the diameter of the microbeads is 10.08 μm, and the images at different depths are as shown in Fig. 14. The result of the method of clear image construction based on the LCHQS-CPCR model is shown in Fig. 15, where it can be observed that the outline of the microbeads has become clearer, and the noise between the microbeads has also been effectively removed.

 

Fig. 14. Original microbead images at different depths.

Download Full Size | PDF

 

Fig. 15. Clear images after reconstruction using our method.

Download Full Size | PDF

To evaluate the degree of clarity of the reconstructed images at different depths, we divided each reconstructed image into four equal regions, and calculated the Laplacian value, the average gradient, and the image entropy of each region, as listed in Table 4. The following conclusions can be drawn from the listed results:

Table 4. Quality evaluations of the reconstructed images with different factors

View Table | View all tables in this article

In other words, the reconstructed images had a better quality at different depths, regardless of the evaluation index used. This implies that the proposed method is effective in improving the resolution of the optical microscopic images.

5.5. Comparison and evaluation

To further compare the reconstruction performance of our method with that of conventional reconstruction methods, in this study, the inverse filtering method, Wiener filtering, and Lucy-Richardson deconvolution algorithm (LR) have been used to reconstruct the same original blurry images. Because the deblurring theory of these conventional methods is to deconvoluted the original image with a PSF, it is reasonable to obtain an appropriate PSF for the same. To estimate the appropriate PSFs for these methods, we randomly selected an image with high resolution from a series of images captured by our microscopic system at this depth position and cropped the Airy disk image from it. Nest, we transferred it to a grey image and calculated its intensity map. Finally, we fit the intensity map with a two-dimensional Gaussian function to obtain the PSF at this position. When Δd = 0 μm, the standard deviation of the fitted Gaussian function was 0.63 and the parameter K of the Wiener filter was 0.05. The reconstruction results obtained from these methods are shown in Fig. 16. Figure 16(a) shows the original image, Fig. 16(b) shows the reconstruction result of the inverse filter, Fig. 16(c) shows the Wiener filter reconstruction, Fig. 16(d) shows the result of the LR deconvolution reconstruction, and Fig. 16(e) shows the result of reconstruction using the proposed method.

 

Fig. 16. Results of different reconstruction methods. (a) Original image. (b) Result of inverse filter. (c) Result of Wiener filter. (d) Result of LR. (e) Result of our method.

Download Full Size | PDF

From Fig. 16, we can obtain the following conclusions:

To compare the reconstructed images quantitatively, we calculated the Laplacian value, average gradient value, and image entropy of each reconstructed image. The calculation results are detailed in Table 5, which shows unusually high Laplacian and average gradient values of the inverse filtering. This is because the inverse filtering is more sensitive to image noise and requires accurate estimation of the noise. This led to more impurities in the result and had a greater impact on the evaluation index. For the proposed method, the Laplacian value increased from 103.63 to 161.47, which proves that our method improves the edge definition to a certain extent, and the average gradient and image entropy of the method proposed in this study are better than those of the Wiener filtering and LR algorithm. Therefore, the experimental results of different factors prove that the proposed method achieves better processing results in the reconstruction of microscopic blurry images than other methods.

Table 5. Quality evaluation with different indexes

View Table | View all tables in this article

To evaluate the image reconstruction at different depths, we calculated the Laplacian values, the average gradient values and the image entropy values of all the reconstructed images, at different depths using the proposed method; the curves are shown in Fig. 17, where, except for the LR algorithm under the Laplacian evaluation metric, the effect of the proposed method is comparatively better than the other methods, in all three indexes. This indicates that for optical microscopic images at different depths, the proposed method is capable of effective deblurring.

 

Fig. 17. Clarity evaluation of the reconstructed image using different indexes. (a) Laplacian. (b) Average gradient. (c) Image entropy.

Download Full Size | PDF

6. Conclusion

In this work, we proposed and developed a method to extract blurring kernels and reconstruct super-resolution images based on StyleGANs by designing a feature analysis system of microscopic images. The first contribution involves training a StyleGAN-based model to automatically generate a series of ideal Gaussian light-source images with a clear Airy disk at different depths. Through these images, the intensity distribution in the Airy disk can be analyzed and the blurring kernel of the given optical system can be extracted. The second contribution constitutes of processing a clear image reconstruction model based on LCHQS-CPCR, where the blurry images of the Gaussian light source with a single wavelength are reconstructed with the extracted blurring kernel. Based on the characteristics of autonomous learning in deep learning, the method can avoid redundant analysis and modelling of influencing factors, such as defocusing, diffraction, or object movement, causing complex disturbances to the imaging results. By learning and simulating the beam images of a light source containing various interference factors, the system generates ideal beam images that suppress random interference, extracts the blurring kernel, and reconstructs the blurry image with super-resolution through deconvolution. We chose the images of suspended microbeads in a solution under visual light illumination to test our method, and we achieved a high-resolution image reconstruction with the blurring kernels from the generated optical images of the Gaussian light source.