Open Access
Add to BibSonomy
Abstract
The Fourier single-pixel imaging technique exhibits great potential for compressive imaging. However, the utilization of low sampling ratio can introduce unwanted ringing artifacts, thereby compromising the fidelity of reconstructed image detail. To address this issue, Vector guided Fourier single-pixel imaging (V-FSI) has been proposed. We analyze the statistical properties in the edge vector field derived from images with low sampling ratio. Based on this information, a tailored sampling map is designed to acquire the significant high-frequency components for image reconstruction. Experimental results demonstrate the remarkable effectiveness of the proposed V-FSI method in enhancing image quality. Notably, V-FSI exhibits exceptional capabilities in perceiving and preserving the details of the objects, particularly for objects characterized by pronounced periodicity and directionality.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
Corrections
10 April 2024: A typographical correction was made to the author affiliations.
1. Introduction
Single-pixel imaging technology has gained popularity in recent years. Entangled photon pairs were first used by Shih et al. to achieve ghost imaging (GI) in 1995 [1]. In 2002, classical light sources were successfully used for GI [2]. In 2008, Shapiro introduced a method called computational ghost imaging (CGI) [3], which eliminated the need for a reference path in conventional correlation imaging systems. This advancement enhanced the flexibility and practicality of the system, and CGI is also referred to as single-pixel imaging (SPI).
In 2015, Zhang et al. proposed the technique of Fourier single-pixel imaging (FSI) [4], which projects the patterns of Fourier basis onto the object, obtains the corresponding Fourier coefficients through N-step phase shifting and reconstructs the image. Compared to traditional SPI based on Hadamard basis, FSI offers higher imaging efficiency [5]. In 2017, Zhang et al. employed the binary pattern to mimic Fourier basis, which significantly enhances the temporal efficiency of FSI [6]. Currently, FSI technology has found extensive applications in various domains, including three-dimensional imaging [7], multi-modal imaging [8], and microscopic imaging [9].
To enhance the imaging efficiency of FSI, Xiao et al. proposed a method that iteratively corrects the Fourier spectrum by sampling a small number of low-frequency coefficients [10]. Meng et al. introduced the sparse Fourier single-pixel imaging (S-FSI) method [11], which employs high-density sampling in the low-frequency region and low-density sampling in the high-frequency region. In 2021, He et al. designed an adaptive sampling method based on radial correlation to improve the performance of FSI. This method achieves results by directly performing inverse Fourier transform for object reconstruction while preserving image details effectively [12]. In 2023, Chen et al. proposed an adaptive sampling FSI method based on discrete coefficients [13]. This method planed circular and rectangular sampling paths in the frequency domain, and sampling is stopped when the discrete coefficient exceeds the threshold, achieving good imaging quality at a lower sampling rate. Thakkar et al. studied how the quality of reconstructed images in FSI is impacted by sampling pathways, SNR ratios, and phase shifts [14]. It provides a feasible solution for real-time imaging applications.
Due to the sparsity of natural image in the Fourier domain, the energy of an image usually concentrated in the low-frequency region. However, if only the low-frequencies are sampled, a ringing effect is induced in the reconstructed image. In 2020, Rizvi et al. reported a method based on deep learning-based deconvolution for mitigating the ringing effect in FSI [15]. Inspired by Elias et al. [16], Zhang et al. proposed a deconvolution algorithm based on subpixel shifting and image fusion in 2022 [17], which achieves ringing-free FSI imaging effectively and rapidly.
However, existing deconvolution algorithms for mitigating the ringing effect do not capture the desired high-frequency information of the physical scene, rather they merely estimate the high-frequency information, instead of measuring them. The development of adaptive sampling techniques in Fourier single pixels is limited by the difficulty in modeling and accurately predicting the significant spectra in the frequency domain.
We propose a novel approach called vector-guided Fourier single-pixel imaging (V-FSI) to address this issue. This method statistically analyzes the edge vector information of the under-sampled FSI images to predict the potential directions where the significant high-frequency components might locate. Subsequent sampling is conducted based on these predictions. Experimental results demonstrate that the approach can accurately predict the distribution of high-frequency components, particularly for the objects with pronounced periodicity and directionality. Even at low sampling ratios (SR), the sampling method captures crucial high-frequency content, resulting in significant optimization of the local image details during reconstruction.
2. Theory
2.1 FSI and ringing oscillations
In Fourier single-pixel imaging (FSI), computer-generated Fourier basis patterns are utilized to project onto the scene. It utilizes a detector with no spatial resolution capability to sequentially measure the weights corresponding to different Fourier basis patterns, which are the coefficients of the Fourier spectra. By employing computational methods, FSI achieves the reconstruction of the object image.
A three-step phase-shifting strategy is typically adopted for fast FSI reconstruction of an $M\times N$ image [17]. The i-th phase-shifting pattern can be represented as follows:
As shown in Fig. 1, when the SR of FSI is 5% ( see Fig. 1(b.1)), the reconstructed result suffers from loss of details and significant ringing oscillations near the edges (see Fig. 1(b.2)). We define the low-frequency sampling map as $S_{L}$ ( see Fig. 1(b.1)). Under the under-sampling conditions, the utilized spectrum is equivalent to applying a two-dimensional ideal low pass filter to the original spectrum. After inverse Fourier transform, the formed result is equivalent to convolving the image with a Sinc function, resulting in pronounced oscillations [18].
2.2 Spectra and orientation of image features
The spatial frequency can be described by the rate of change of the gray level of an image, which reflects how pixel intensities vary in space and serves as an indicator of the degree of grayscale variations in an image. More importantly, spatial frequency is correlated with the orientation of features within an image [19], which is the basis of the proposed V-FSI, and we show the two examples to present this correlation visually, as seen in Fig. 2.
Fig. 2. Spatial gray scale distribution versus high-frequency distribution. (a) spatial gray scale transverse distribution; (b) spatial gray scale oblique distribution.
In Fig. 2(a.1), an image of a set of horizontal lines is presented, and the corresponding Fourier transform is shown in Fig. 2(a.3). Due to the image composed of a series of horizontal stripes, its corresponding significate spectrum distribute along the vertical. We employ a circle mask $S_{L}$ (as shown in Fig. 1(b.1)) to capture low-frequency components at a sampling rate of 5%. Subsequently, we reconstruct the image and extract its edge information, as shown in Fig. 2(a.2) and Fig. 2(a.4) respectively. It is evident that under the under-sampling condition, the image exhibits noticeable oscillations. The oscillatory patterns in the edge image also preserve the directional characteristics of the original image, and the line segments in the edge image (see Fig. 2(a.4)) almost perpendicular to the significant spectra shown in Fig. 2(a.3).
A similar observation in another example is shown in Fig. 2(b.1). Figure 2(b.3) and (b.4) show the spectra and the edge image with the under-sampling condition. It can be noticed that the line segments extracted by the edge image is nearly perpendicular to the prominent frequency components (as indicated by the dashed box in Fig. 2(b.3) and (b.4)).
Inspired by this phenomenon, we propose the technique of the vector-guided Fourier single-pixel imaging (V-FSI). V-FSI utilize the edge information derived from oscillatory images to predict the orientations where the significant spectra might appear. This innovative approach enables precise and efficiency sampling in the mid to high-frequency range.
2.3 Vector-guided Fourier single-pixel imaging (V-FSI)
Based on Section 2.2, the distribution of edge information in the spatial domain is related to the distribution of high-frequency components. Specifically, the direction of the normal vector of the edge image is consistent with the direction of critical high-frequency components.
Building upon this observation, we propose a sampling method called V-FSI. This method predicts the distribution of key high-frequency components based on edge information and performs the samplings on the region of high-frequency for the reconstruction. In this method, the test images used are from the ILSVRC2012 database [20], and each image is uniformly scaled to $256\times 256$.
Algorithm shows in Fig. 3.
- Step1 Capture the low-resolution image with the mask $S_{L}$.
The low-frequency components of the object were sampled by employing $S_{L}$ at a sampling rate of 5% and the image $I_{O}$ is reconstructed, as shown in Fig. 3(a).
- Step2 Perform pre-processing on the $I_{0}$.
- 2.1 Perform edge extraction on $I_{0}$. The extracted edge image $I_{E}$ still has the ringing effects, as shown in Fig. 3(b.1).
- 2.2 Apply the line segment detector (LSD) algorithm to perform line segment detection on $I_{E}$ [21]. The detected line segments, denoted as $L_{j},j=1,2,\ldots,J$. As shown in Fig. 3(b.2), it can be observed that one edge corresponds to the detection of two line segments. This is because the LSD detection is directional, which does not affect the final results in our study.
- Step3 Analyze the statistical distribution of $\left \{ L_{j} \right \}$.
As illustrated in Fig. 3(c.1), an undirected line segment has two normal vectors with opposite directions. Only the angle values within the range of 0 to 180 degrees are considered, because the Fourier spectrum is symmetric about the image center. The angle of the normal vector denoted as $A_{j},j=1,2,\ldots,J$.
In this method, the range of 0 to 180 degrees is divided into 45 intervals of 4 degrees each. The angle distribution histogram of $A_{j}$ within these discrete angle intervals is calculated, marked by $h(\theta )$. This angle histogram is defined as follows:
where $K^{(\theta )}$ represents the count of all line segment normal vectors that fall within the angle interval $\theta$. Figure 3(c.2) and (c.3) shows the spatial vector field and the angle histogram respectively. In Fig. 3(c.2), the black dashed lines represent the line segment $L_{j}$, the arrow angles represent $A_{j}$, and the color of the arrow indicates the interval to which $A_{j}$ belongs, which corresponds to the color of the interval in Fig. 3(c.3). - Step4 Generate sampling map based on $h(\theta )$.
Sampling map is generated based on $h(\theta )$. We take the top 20% of intervals in $h(\theta )$ as the directions where significant spectra are likely to exist and generate the sampling map accordingly. Please find Supplement 1 for the method of generating map. The final generated sampling map $S_{Final}$, along with the sampling spectrum, is shown in Fig. 3(d).
- Step5 Reconstruction and evaluation.
Suppose that the number of sampling points is $N_{Final}$. Then, we sort the moduli of all the Fourier coefficients of the original picture in descending order. After sorting, we record the coordinates of the largest $N_{Final}$ components, and denote the corresponding positions, marked by $S_{ori}$, which represents the positions of the key spectra. We define $S_{cov}$ as the overlap area between $S_{Final}$ and $S_{ori}$, and the number of sampling points in $S_{cov}$ is $N_{cov}$. We define the key spectrum coverage ratio $\eta$ (as shown in Eq. (5)). Clearly, the higher the $\eta$, the more efficient the corresponding sampling method.
By utilizing the CS algorithm (TVAL3) [22], the image is reconstructed. The coverage map of significant spectral components and the reconstruction result is shown in Fig. 3(e). In the coverage map, orange and cyan points indicate those which only appear in $S_{ori}$ and $S_{Final}$ respectively, while red points indicate points which appear in both $S_{ori}$ and $S_{Final}$. Obviously, the number of red points is equal to $N_{cov}$.
The final SR of the image is 8.42%, and the coverage rate $\eta$ reaches 64.51%. It can be observed that the predicted high-frequency distribution based on statistical analysis is quite accurate, allowing for capturing a significant amount of high-frequency information even at low SR. The overall reconstructed result is of good quality with clear edge details (as shown in Fig. 3(e.3)).
3. Simulation and experiment
3.1 Simulation
In this chapter, we simulate and analyze the imaging performance of the sampling methods by comparing the proposed V-FSI with FSI and S-FSI. To ensure significance and fairness, we ensure that the three methods have the same sampling number. In addition, both V-FSI and S-FSI used circular regions with the same diameter (65 pixels) as their initial full-sampling areas. Reconstruction was performed using compressed sensing algorithms, and the best result was selected as the imaging outcome.
We validate the proposed method in 2 different scenes: (a) Grid ($256\times 256$), and (b) Shutter ($256\times 256$) images, and the total SR are 8.54%, and 8.39% respectively. The reconstruction results are shown in Fig. 4(a)(b) respectively.
Fig. 4. Imaging results for 2 different scenarios: (a) Grid ($256\times 256$), (b) Shutter ($256\times 256$). (a.1, b.1) the original images, details, and $I_{E}$ of under-sampled images for each scene. (a.2, b.2) the spatial vector fields. (a.3, b.3) $h(\theta )$. For each scene, (a.4, a.5, a.6), (b.4, b.5, b.6) the significant spectral coverage maps, reconstruction results and magnified details obtained using FSI, S-FSI, and V-FSI respectively, where blue represents FSI, green represents S-FSI, red represents V-FSI.
The left side of Fig. 4 shows the spatial information of the original image, and the right side shows the sampling and reconstruction results by using FSI, S-FSI, and V-FSI respectively. To emphasize the preservation capability of detail, we have zoomed in the local regions.
It can be seen that the image reconstructed by FSI appears smooth but lacks details (as shown in Fig. 4(a.4, b.4)), which due to the absence of high-frequency sampling. The smoothness can be attributed to the utilization of CS algorithms, which suppress ringing effects. Similarly, S-FSI maintains a relatively smooth image quality and exhibits slight superiority in detail preservation compared to FSI (as shown in Fig. 4(a.5, b.5)). Notably, V-FSI demonstrates a significant advantage over FSI and S-FSI in terms of detail preservation (as shown in Fig. 4(a.6, b.6)).
In order to evaluate the efficiency of the different sampling strategies, the key spectrum coverage ratio $\eta$, and the structural similarity index measure (SSIM) [23,24] are used, with the evaluation results are shown in Table 1. The $\eta$ of V-FSI is approximately 30.3%, and 15.7% higher than FSI, and approximately 46.2%, and 29.9% higher than S-FSI. It is evident that V-FSI offers higher sampling efficiency. Additionally, our method also achieves enhancements in terms of both $\eta$ and SSIM (Bold red text indicates optimal values).
Table 1. The evaluation parameters of different sampling methods in Simulation.
3.2 Experiment
The experimental setup is illustrated in Fig. 5. A series of Fourier basis patterns are generated by the computer and loaded into the DMD (Vialux, V4395). The scene is illuminated by a light source (Thorlabs, LIUCWHA, 250mW), and is imaged on the DMD. A lens ($f=30mm$) is used to collect the reflected light intensity onto a single-pixel detector (Thorlabs, DET100A2). Subsequently, the signal is transmitted to the computer via a digital acquisition card (NI, 9222C). This system forms a $128\times 128$ image using a three-step phase-shifting strategy. The reference image is obtained through FSI with a 100% sampling rate. In this experiment, the modulation frequency of DMD is 1KHz. The project is run on a computer with an Intel Xeon CPU W-2235, 16GB RAM, and an NVIDIA Quadro P2200 GPU.
We further conduct experimental comparisons of different sampling methods in two different test scenes. The test objects, (a) Box and (b) Bookshelf. All parameters are consistent with the settings in the simulation. The diameters of the full-sampling area is changed to 25 pixels, resulting in overall SR of 8.75% and 8.49% respectively. The results are shown in Fig. 6 and the resolution is in the size of $128\times 128$.
Fig. 6. Comparison of experimental results for different scenarios: (a) Box, (b) Bookshelf. (a.1, b.1) the original images, details, and $I_{E}$ of under-sampled images for each scene. (a.2, b.2) the spatial vector fields. (a.3, b.3) $h(\theta )$. For each scene, (a.4, a.5, a.6), (b.4, b.5, b.6) the significant spectral coverage maps, reconstruction results and magnified details obtained using FSI, S-FSI, and V-FSI respectively, where blue represents FSI, green represents S-FSI, red represents V-FSI.
Similar to the simulation, FSI and S-FSI partly suppresses certain ringing effect, however, they lack in preserving the details within the reconstructed images. On the other hand, the proposed V-FSI technique offers superior imaging outcomes. We still use $\eta$ and SSIM as quality evaluation functions, with the evaluation results are shown in Table 2.
Table 2. The evaluation parameters of different sampling methods in experiment.
As shown in Table 2, it can be seen that these evaluations further confirm the proposed V-FSI achieves the superior image quality. Compared to FSI and S-FSI, V-FSI achieves improved performance in terms of $\eta$ and SSIM metrics (bold red text indicates optimal values).
Note that $\eta$ and SSIM are global image quality assessment methods, but our method primarily focuses on recovering local image details. Therefore, the slight improvements in these assessment methods may not fully demonstrate the superiority of our proposed method in terms of imaging performance.
In terms of detail preservation, a comparison between Fig. 6(a.4, a.5, a.6), (b.4, b.5, b.6) reveals that both FSI and S-FSI fail to reconstruct detail information effectively. This limitation primarily arises from the absence of critical frequency spectra in the high-frequency region. However, the proposed V-FSI method in this study successfully captures crucial information within the high-frequency range, hence the proposed V-FSI reconstruct the local details of the image, including the fidelity of the reconstructed edges and fine structures.
Clearly, the performance of our proposed method is controlled by the region of the initial full-sampling. The area of this region can be roughly determined based on the following criteria. When applying a region of initial full-sampling, it is crucial to identify the essential details of the object in the corresponding formed image. In this case, the propose method can still analyze the edge and the corresponding vector information, despite the suboptimal image quality (as discussed in the article, the formed image has a ringing effect). However, when the total sampling region is sufficiently small, the necessary details become unrecognizable, and subsequent algorithms inapplicable.
However, it should be pointed out that we have demonstrated the effectiveness and superiority of this proposed method, but there are still some issues to be optimized. Taking Fig. 6(a) as an example, the time spent on DMD modulation is the shortest one in all steps in the whole process, and this time can be further shortened by increasing the speed of DMD modulation. On the other hand, it takes 3.48 seconds to analyze the vector information and make the sampling map of the high frequency region, which influences the efficiency of this technology. In future work, we will look for ways to quickly compute these two types of information (vector information, high-frequency sampling map).
4. Summary
This paper introduces a novel approach called vector-guided Fourier single-pixel imaging (V-FSI). This technique reconstructs the images with low sampling ratios and calculates the corresponding vector field of the edge image. Due to the significant correlation between this vector field and the distribution of key frequency components, we have proposed a frequency-domain sampling method based on this vector field. Experimental results demonstrate that V-FSI can effectively preserve the fine details of the image even at lower sampling ratios, thus driving advancements in the field of single-pixel imaging. Moreover, the concept of utilizing the vector field to perceive crucial high-frequency components in V-FSI can also serve as a means to explore interdependencies among different dimensions of an image, offering valuable insights for further research in related areas.
Funding
National Natural Science Foundation of China (61905108, 62375129).
Disclosures
The authors declare no conflicts of interest.
Data Availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
Supplemental document
See Supplement 1 for supporting content.
References
1. T. B. Pittman, Y. Shih, D. Strekalov, et al., “Optical imaging by means of two-photon quantum entanglement,” Phys. Rev. A 52(5), R3429–R3432 (1995). [CrossRef]
2. R. S. Bennink, S. J. Bentley, and R. W. Boyd, ““two-photon” coincidence imaging with a classical source,” Phys. Rev. Lett. 89(11), 113601 (2002). [CrossRef]
3. J. H. Shapiro, “Computational ghost imaging,” Phys. Rev. A 78(6), 061802 (2008). [CrossRef]
4. Z. Zhang, X. Ma, and J. Zhong, “Single-pixel imaging by means of fourier spectrum acquisition,” Nat. Commun. 6(1), 6225 (2015). [CrossRef]
5. Z. Zhang, X. Wang, G. Zheng, et al., “Hadamard single-pixel imaging versus fourier single-pixel imaging,” Opt. Express 25(16), 19619–19639 (2017). [CrossRef]
6. Z. Zhang, X. Wang, G. Zheng, et al., “Fast fourier single-pixel imaging via binary illumination,” Sci. Rep. 7(1), 12029 (2017). [CrossRef]
7. Z. Zhang and J. Zhong, “Three-dimensional single-pixel imaging with far fewer measurements than effective image pixels,” Opt. Lett. 41(11), 2497–2500 (2016). [CrossRef]
8. Z. Zhang, S. Liu, and J. Peng, “Simultaneous spatial, spectral, and 3d compressive imaging via efficient fourier single-pixel measurements,” Optica 5(3), 315–319 (2018). [CrossRef]
9. J. Peng, M. Yao, and J. Cheng, “Micro-tomography via single-pixel imaging,” Opt. Express 26(24), 31094–31105 (2018). [CrossRef]
10. Y. Xiao, L. Zhou, and W. Chen, “Fourier spectrum retrieval in single-pixel imaging,” IEEE Photonics J. 11(2), 1–11 (2019). [CrossRef]
11. M. Wenwen, S. Dongfeng, and H. Jian, “Sparse fourier single-pixel imaging,” Opt. Express 27(22), 31490–31503 (2019). [CrossRef]
12. R. He, Z. Weng, and Y. Zhang, “Adaptive fourier single pixel imaging based on the radial correlation in the fourier domain,” Opt. Express 29(22), 36021–36037 (2021). [CrossRef]
13. G. Chen, Y. Xu, J. Zhou, et al., “An adaptive sampling fourier single-pixel imaging method based on discrete coefficients,” in 2023 28th International Conference on Automation and Computing (ICAC), (IEEE, 2023), pp. 1–6.
14. T. Thakkar, P. K. Sokhi, A. Kadu, et al., “Single pixel measurements through fourier phase shifting in imaging system,” in 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), (IEEE, 2023), pp. 1–6.
15. S. Rizvi, J. Cao, K. Zhang, et al., “Deringing and denoising in extremely under-sampled fourier single pixel imaging,” Opt. Express 28(5), 7360–7374 (2020). [CrossRef]
16. E. Kellner, B. Dhital, V. G. Kiselev, et al., “Gibbs-ringing artifact removal based on local subvoxel-shifts,” Magn. Reson. Med. 76(5), 1574–1581 (2016). [CrossRef]
17. H. Peng, S. Qi, and P. Qi, “Ringing-free fast fourier single-pixel imaging,” Opt. Lett. 47(5), 1017–1020 (2022). [CrossRef]
18. R. C. Gonzalez, Digital image processing (Pearson education india, 2009).
19. M. Nixon and A. Aguado, Feature extraction and image processing for computer vision (Academic press, 2019).
20. O. Russakovsky, J. Deng, and H. Su, “Imagenet large scale visual recognition challenge,” Int. J. Comput. Vis. 115(3), 211–252 (2015). [CrossRef]
21. R. G. Von Gioi, J. Jakubowicz, J.-M. Morel, et al., “Lsd: A line segment detector,” Image Process. On Line 2, 35–55 (2012). [CrossRef]
22. C. Li, “An efficient algorithm for total variation regularization with applications to the single pixel camera and compressive sensing, Ph.D. thesis, Rice University.; (2009).,” Phys. Rev. A 51(5), R3429–R3432 (1995). [CrossRef]
23. Z. Wang and A. C. Bovik, “A universal image quality index,” IEEE Signal Process. Lett. 9(3), 81–84 (2002). [CrossRef]
24. Z. Wang, A. C. Bovik, H. R. Sheikh, et al., “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. on Image Process. 13(4), 600–612 (2004). [CrossRef]
