Open Access
Add to BibSonomy
Abstract
We propose a pre-processing method of source image to reduce blur of aerial image that is formed with aerial imaging by retro-reflection (AIRR). The image blur in the AIRR optical system depends on the optical elements and the incident angle. In this paper, the point spread function (PSF) was measured by changing the optical arrangements and the retro-reflectors. A deconvolution processing based on the obtained PSF was applied on the aerial image. The accuracy of the deconvolution processing was quantitatively evaluated by use of three types of image quality indices. These numerical experimental results show the possibility of removing the blur of the aerial image by applying the deconvolution process based on the PSF of AIRR optical system.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Aerial video display technology [1–3] is expected as a new era display device. Since this technology has the feature of forming a real image in space, it can be displayed in front of the eyes without physical contact, and highly visible signage can be realized. For example, in the case of signage display such as a map, the aerial image can be expected to have an intuitive visual effect for the user by displaying the direction information to the destination in front of the user [4]. In addition, by combining with sensing information in the real world such as a camera, this technology can be applied to an augmented reality system in which an aerial image and a real object interact [5–9].
The problem of this technique is that the display image quality is deteriorated by the aerial imaging optical system. When the diffused light from the light source is re-converged in space, all the light cannot be completely converged and spreads. Then, the aerial image is blurred, which means that sharpness of the aerial image is reduced. Our previous work shows that as the size of the displayed image increases, the imaging optical path from the light source to the aerial image increases, and the blurring increases [10].
A blur correction method has proposed by generating an inverse filter based on a filter matrix showing the characteristics of blur and deconvolving it into an image light source [11]. However, the blurring of aerial images due to the imaging optical system depends on the performance of the optical components and their arrangement. Due to the characteristics of this technology, which forms an image in the air through a complicated optical path due to multiple reflections and transmissions, the imaging accuracy of the aerial image changes depending on the wavelength of the light source and the optical path that changes according to the viewpoint position. Therefore, when applying this technology to an aerial image display device that shows contents of various colors to multiple users such as signage, the single filter in the existing research cannot realize the optimum blur correction. Thus, it is necessary to correct the aerial images in response to changes in the structure. We have studied an image processing method that can correct blurring even in aerial display using a full-color light source, especially assuming the application of aerial image display technology to signage display devices.
The purpose of this paper is to propose a deconvolution processing method based on the PSF that is obtained from the formed aerial image to reduce the blur. In this paper, we conducted experiments to measure point spread function (PSF) in several types of optical arrangements by changing retro-reflectors. Furthermore, the effectiveness of the deblur processing was verified by generating a corrected image from the obtained blurring characteristics and displaying it on an aerial image. The quality of the deblurred image is evaluated by image quality indices.
2. Principle
2.1 Aerial imaging
There are several ways to form aerial images. We have proposed aerial imaging by retro-reflection (AIRR) [12–14]. Other aerial imaging technologies include optical plate that apply optical members that combine a plurality of micromirror array [15,16].
Figure 1 shows the principle of AIRR. Light emitted from a light-source display is reflected once by a beam splitter. The reflected light returns to the original direction by a retro-reflector. A portion of the retro-reflected light transmits through the beam splitter. The transmitted light converges to the plane-symmetrical position of the light-source display regarding the beam splitter.
The cause of blurring of aerial images is because the retro-reflected light spreads due to the diffraction caused by the aperture of the corner-cube element in the retro-reflector. As shown in Fig. 2, the aperture width of the retro-reflective element changes in accordance with the incident angle of light. Since the diffraction pattern depends on the aperture width, the magnitude of the effect of blurring changes depending on the angle at which the light is incident on the retro-reflective material and the distance until the retro-reflected light reaches the viewpoint [17].
Fig. 2. Schematic diagram showing change of effective aperture size according to the incident angle to a corner cube. (a) A corner-cube in a retro-reflector. (b) When the incident angle increases, the effective aperture size reduces.
In addition, there are several types of corner cube element structures that realize retro-reflector. For example, to improve the retroreflection accuracy, the element structure is designed so that the element region having low retro-reflection accuracy is cut out and the elements are arranged closer to each other. Since the diffracted aperture region changes when the element structure is different, it is considered necessary to measure the correction parameters according to the element specifications.
2.2 Deconvolution processing
This section describes the parameters that indicate how the aerial image is blurred and the image processing that realizes image quality correction by applying them. The blurring of aerial images can be expressed by the PSF, which shows how the two-dimensional light spreads when a light source is placed at the center point and projected. The magnitude and direction of blurring of the aerial image is shown by the luminance distribution in the two-dimensional grayscale image.
For the calculation of PSF, the data obtained by capturing the aerial image in which the bright spots are displayed is acquired. First, we shoot a position-aligned light source and aerial video with RAW data, and then reverse-correct the gamma to the data so that the pixel value is linear with the luminance value. After that, we calibrate the pixel pitch and the actual distance. The polar coordinate system PSF is calculated as follows:
In Eq. (1), each parameter has $S({\theta ,R + r} )\; $ as the light source image with the radius R of the bright spot as the direction $\theta $. and the distance r from the center of the bright spot, and $A({\theta ,R + r} )$ as the aerial image. The position of R is the edge portion of the boundary between high luminance and low luminance in the image $S({\theta ,R + r} )$, and its inclination is steep. On the other hand, the image $A({\theta ,R + r} )$. has a gentle inclination toward the outside from the edge portion R due to blurring during aerial imaging. Therefore, as shown in the Fig. 3, the PSF can be calculated from the captured image by mapping the difference with the direction outside the R position and the distance as parameters to thedeal gradation value.
The obtained PSF can be regarded as a function that can reproduce the blurring of aerial images in digital space. In other words, forming an image on an aerial image is the same as displaying an output image that is a convolution of the input image and the PSF. The blur correction image g(x, y) can be realized by convolving the inverse function m(x, y) from the PSF into the input image f(x, y) in advance, as follows:
Then, the PSF is convoluted in the aerial imaging, and by canceling the convolution with the reverse filter, the aerial image display with the blur corrected is finally displayed. To calculate the function m(x, y) in Eq. (2), the Wiener filter is firstly calculated from the Fourier-transformed PSF result. The calculation method of the Wiener filter is shown in the Eq. (3).
where M is a Fourier transform function of the correction function m (x, y). H is a filter parameter indicating blurring, and can be obtained by Fourier transforming the result of PSF. N is a constant representing noise, and the larger the value, the smoother the corrected image. When N is extremely small, a fine noise pattern appears. Therefore, in this experiment, a correction function is generated by qualitatively selecting a highly visible value in the range of 0.1 to 0.5 for a filter size of 150 pixels.3. Experiment
3.1 Experimental setup
In this experiment, PSF analysis is performed using two parameters, the placement angle of the retro-reflector and the element structure of the retro-reflector. Figure 4 shows a PSF measurement environment. In this measurement, the analysis is performed from an image captured by projecting a test image on aerial image. For the light source display, we used Mitsubishi Electric's product: AA106TA01, which is an IPS high-brightness LCD with a peak brightness of 960 cd/m2. We used 3M DBEF-Qv2, which is a reflective polarizing plate, as a beam splitter, and add a λ/4 retardation film on the retro-reflector. Thereby, the linearly polarized light from the LCD converges to the aerial image position with a high efficiency [18]. The camera for measurement used was Canon EOS 6D Mark II. We set the camera parameters manually (F20, 1/10sec, ISO800). The distance among the camera, the aerial image, and retro-reflector were fixed.
3.2 Measurement of changes in diffraction pattern
In order to show the change in the influence of diffraction in the retro-reflective element in this experiment, the arrangement angle of the retroreflective material is used as a parameter. The retro-reflector is rotated about one axis orthogonal to the camera direction. The rotation settings are 0 degrees, ± 15 degrees, ± 30 degrees, and ± 45 degrees, assuming that the orthogonal relationship between the camera's optical axis and the retro-reflector is 0 degrees. The retro-reflector used in the experiments is composed of corner-cube elements. The specifications of the retroreflective sheet used one type: Nikkalite RF-Ax. As an image used for PSF measurement, a white circle with a diameter of 30 mm is displayed in the center of the aerial image.
3.3 Measurement of changes in the element structure of retro-reflector
In this experiment, in order to confirm the difference in element structure of retroreflection accuracy, the following three types of retro-reflector were subjected to the experiment: Nikkalite RF-Ay, Nikkalite RF-Ax, and Nikkalite RF-AC.
The position of the retro-reflector set to the 0 degree angle position. Also, considering the comparison of PSF results with white light, a green (530 nm) circle, which is a light source with a single light frequency, is used as the input light source. Figure 5 shows photographs of the retro-reflective element illuminated by the front illumination. Since the viewfinder of the camera can be seen in the retro-reflective area, the black part is the retro-reflective area. RF-Ay and RF-Ax have a large region size, whereas RF-AC has a small region.
3.4 Evaluation of deconvolution results
This section describes a method for evaluating the effect of blur correction. Correcting an aerial image is the same as restoring an image changed by aerial imaging to an input image. In this paper, we propose three types of image quality evaluation methods that quantify the restoration rate to the original image [19,20].
The first method is root means squared error (RMSE), which quantitatively evaluates the difference in brightness from the original image. The equation for RMSE can be expressed as:
The second is the peak signal to noise ratio (PSNR), which considers blurring as noise and indicates the noise ratio. PSNR is expressed as the following equation using the result of Eq. (4):
The third is structural similarity (SSIM). SSIM is an evaluation method that shows the difference in brightness, contrast, and structure between two images. SSIM is a value calculated for each small window in the image and is expressed as:
Regarding the obtained evaluation values, the higher the similarity, the lower the RMSE value and the higher the PSNR and SSIM values.
In addition, in order to compare these evaluation results with the qualitative correction effect, deconvolution processing is performed on the image obtained by capturing the aerial image, and the blur correction effect is visually confirmed. We display the character image shown in Fig. 6 in the same environment as the PSF experiment, and use the image captured by the same camera as the input image. As a result, the size of the correction filter is matched with the pixel pitch of the input image, and the correction process is applied to the input image.
Fig. 6. Light source image for visual evaluation of correction processing.
(a) Input image for experiment of changes in diffraction pattern.
(b) Input image for experiment of changes in the element structure of retro-reflector.
4. Results
4.1 Results of changes in diffraction pattern
Figure 7 shows aerial images by rotating the retro-reflector. The 0-degrees angle means the angle orthogonal to the camera direction. The positive and the negative angles are the upward and the downward rotations, respectively. Figure 8 shows the result of calculating PSF from the image in Fig. 7. Shape of PSF depends on the angle. As the angle increases, the aerial image is blurred and dark, and the PSF spreads concentrically.
Figure 9 shows the result of the normalized PSF value for the vertical direction in Fig. 8. Figure 10 shows the result of the normalized PSF value for the 45-degrees downward direction in Fig. 8. In Fig. 9, the PSF spreads regardless of whether the angle is positive or negative, while there is a difference in how the PSF spreads in the positive and negative directions in Fig. 10. The larger the angle in the negative direction, the greater the blur in the 45-degree direction.
Figure 11 shows the results of the first experiment that applying the deconvolution processing to the captured aerial images. The input is an image and the aerial image is blurred in outline according to the angle of the retro-reflector. The deconvolution processing uses the PSF shown in Fig. 8. Thus, different kernel was used depending on the angle. As shown in Fig. 11, image sharpness is improved, although the peak luminance is low depending on the angle of retro-reflector.
Table 1 shows the evaluation results of RMSE, PSNR, and SSIM for Fig. 11. For all three results, the higher the angle, the lower the similarity, which is consistent with the qualitative feature of blurring. Next, as a result of performing the correction processing, the values of RMSE and PSNR decreased and the SSIM increased compared to the original aerial image. This indicates that the correction process increases the similarity with the original image and reduces blurring.
Table 1. RMSE, PSNR and SSIM results for the correction image shown in Fig. 11.
4.2 Results of changes in the element structure of retro-reflector
Figure 12 shows an aerial image set captured for each type of retro-reflector. Figure 13 shows the results of calculating PSF from the image set of Fig. 12. From the results of this experiment, it was confirmed that the PSF spreads differently depending on the element structure of retro-reflector.
Figure 14 shows the result of applying the deconvolution process to the captured aerial image. The input image displays the characters “AIRR”. In this process, the PSF shown in Fig. 13 is used as a filter function, an inverse function is calculated, and a deconvolution process is applied to the input image to generate a corrected image. In the results of all specifications, the edges of the characters are emphasized after the deconvolution compared to before the deconvolution, and the readability is improved.
Table 2 shows the evaluation results of RMSE, PSNR, and SSIM for Fig. 14. The quantitative results of RMSE and PSNR are consistent with the qualitative features of blur. Then, within the same specifications, as a result of performing the correction processing, the values of RMSE and PSNR decreased and the SSIM increased compared to the original aerial image. This indicates that the correction process increases the similarity with the original image and reduces blurring. On the other hand, when comparing those with different specifications, the amount of change in SSIM value due to deconvolution is significantly different.
Table 2. RMSE, PSNR and SSIM results for the correction image shown in Fig. 14.
5. Discussion
In this experiment, it was confirmed that the magnitude of blurring qualitatively changes according to the color wavelength of the light source and the shape of the retro-reflective element, and that the feature is quantified by the difference in how the PSF spreads. Furthermore, it was shown that the deconvolution process created for each of these PSFs appropriately corrects the characteristics of different special blurring. From the results of this experiment, it can be seen that in order to correct the image, it is necessary to set it according to the installation position of the retro-reflector and the specifications of retroreflection elements. In other words, to improve the image quality of the aerial display device, it is necessary to perform processing according to the structure of the optical member and the viewpoint position of the user.
For example, as shown in Fig. 15, consider the correction setting of the display device structure that projects from the ceiling with an image size of 500 mm. When the size of both the beam splitter and the retroreflective material is 1500 mm and the floating distance is 1000 mm, the viewpoint position where the user can see the entire image is 873 mm away from the aerial image. Assuming that the angle of the retro-reflector in the center pixel of the aerial image is orthogonal, the angle at the upper end position of the aerial image is 75.6 degrees and the lower end position is 71.9 degrees. In order to perform the optimum blur correction processing, it is necessary to apply the PSF result according to the change in the angle to each pixel position. Furthermore, since this angle changes depending on the position of the user who views the aerial image, in the case of signage in which a wider viewing area is set, correction processing corresponding to the detection of the user position is required.
Assuming the above device configuration, our future research subject is to examine a method for introducing the proposed technology as a display system for aerial displays. Since this experiment was a principal verification for the captured aerial image, it is necessary to verify the effect of the correction in the pre-process in the future. There are two types of methods for implementing correction to the image source before aerial imaging: an optical filter and a digital filter for image processing. In the case of an optical filter, it is necessary to design it according to the color frequency and the position of each light source. Therefore, when applying an optical filter, it is limited to an environment in which the displayed content and the user's observation position are fixed. In the case of a digital filter, deconvolution processing is performed on the input display signal as image processing software. It is possible to design a dynamic filter according to the content and the position of the user, but processing capability that enables real-time convolution processing for each pixel is required. From the results of this experiment, it is necessary to optimize the filter according to the optical component structure, so we will construct a system that introduces digital filter processing and consider the evaluation of the correction effect in the future.
Next, we consider the blurring characteristics of aerial images depending on the retro-reflector. From the results in Section 4.1, the PSF shows anisotropic blurring characteristics as the placement angle increases. It is considered that this is because the retro-reflector has a prismatic shape, so that the aperture width of the two-dimensional surface spreads unevenly according to the change of the rotation, and anisotropic diffraction appears in the PSF. Since the effect of blurring of the anisotropic PSF changes depending on the shape of the light source, it is necessary to design the arrangement according to the azimuth angle of the rotation axis on the plane perpendicular to the line of sight. From the results in Section 4.2, it was confirmed that the PSF characteristics also changed depending on the element shape, and the anisotropic shape and size of the blur changed. Therefore, it is necessary to design the device structure by measuring the PSF according to the specifications and arrangement of the retro-reflective material to be used.
Finally, we consider a method for quantitatively evaluating the correction effect of deconvolution. For RMSE and PSNR, the results are better after correction than before correction for almost all parameters. However, for the image results with a large arrangement angle in Table 1 and RF-Ac in Table 2, the difference between the numerical values before and after the correction is small. This is because when the influence of diffraction by the aperture is large, the brightness is much lower than the resolution. Therefore, the degree of similarity with the input image deteriorates due to the decrease in brightness, and the correction effect is unlikely to appear. In addition, since RMSE and PSNR evaluate the similarity of the entire image, the numerical values change depending on the brightness shape of the original image. In particular, the deconvolution process has a large effect of correction on the edge portion of the character, so the evaluation value tends to depend on the input image.
Then, SSIM has a large difference in values between corrected and uncorrected images with large blur compared to RMSE and PSNR. On the other hand, the difference between the values is less likely to appear in an image with small blur. This feature is because SSIM calculates the similarity of local regions and considers the shape of the image such as edges. In an image with a large amount of blur, the shape is largely corrected, so that the evaluation value is likely to be improved. However, in an image with small blur, not only the effect of improving the edge of the original image is small, but also the gradation around the edge is increased by deconvolution, and as a result, the similarity of shapes is further deteriorated.
Regarding the accuracy of the image quality, it is necessary to consider an evaluation method that excludes the brightness profile while considering the shape similarity like SSIM. For example, it seems that the similarity of shapes can be accurately evaluated by obtaining the SSIM after normalizing the gradation value with respect to the imaging result of the aerial image. In addition, in this paper, the character image that qualitatively confirms the visibility is also used for the evaluation image of SSIM. However, since the evaluation value of SSIM depends on the shape of the characters, it is necessary to study a test image that can evaluate the similarity of edge shapes in all directions as a future task. On the other hand, when evaluating the user's visibility by correction, it is necessary to evaluate the brightness and the resolution according to the visual characteristics. For example, from the aspect of image resolution characteristics, a quantitative evaluation value called MTF can be mentioned. As an existing study, there is a measurement by MTF as an evaluation of the resolution characteristics of aerial images [21]. However, in the case of an aerial image to which deconvolution processing is applied as in this paper, not only the visibility is improved, but also unnecessary artifacts are generated around the edge region. Since it adversely affects specific frequency regions in MTF results, it is necessary to consider a method for appropriately correcting and evaluating and analyzing those noises.
6. Conclusion
We have analyzed how the light spreads according to the incident angle to the retro-reflector and the structure of retroreflection elements, and confirmed the deblurring correction effect by the convolution processing by the inverse filter. Then, as a study of a method for quantitatively evaluating the blur correction effect, we evaluated the change between the original image and the aerial image using RMSE, PSNR, and SSIM, which show image similarity. In the PSF measurement experiment, it was confirmed that the anisotropic blur increases as the placement angle of the retro-reflector increases. This indicates that the apparent aperture width of the retroreflective element causes anisotropic blurring.
The deconvolution processing can be expected as an improvement approach of reducing the blur of AIRR by image processing without changing optical hardware. However, depending on the specifications of the retro-reflector, it may not be possible to correct because the amount of blurring is too large. When applying this correction technique to a display device, the optical design is required to have a degree of blurring that can be corrected. We will continue our research with the expectation that this technology will be useful for the construction of higher-definition aerial display systems.
Disclosures
The authors declare no conflicts of interest. This work is original and has not been published elsewhere.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. J. Hong, Y. Kim, H.-J. Choi, J. Hahn, J.-H. Park, H. Kim, S.-W. Min, N. Chen, and B. Lee, “Three-dimensional display technologies of recent interest: principles, status, and issues,” Appl. Opt. 50(34), H87–H115 (2011). [CrossRef]
2. J. Geng, “Three-dimensional display technologies,” Adv. Opt. Photonics 5(4), 456–535 (2013). [CrossRef]
3. Yole Developpement, Report, “Next Generation 3D Display.” http://www.yole.fr/iso_upload/News/2019/PR_NEXT_GENERATION_3D_DISPLAY_OverviewYOLE_October2019.pdf. Accessed 2 September 2021.
4. Mitsubishi Electric Corp, News Release, http://www.mitsubishielectric.com/news/2016/0217-e.html. Accessed 2 September 2021.
5. Y. Tokuda, A. Hiyama, M. Hirose, and H. Yamamoto, “R2D2 w/AIRR: Real time & Real space Double-Layered Display with Aerial Imaging by Retro-Reflection,” Proc. SIGGRAPH Asia’15, Emerging Technologies 20, 1–20 (2015). [CrossRef]
6. Y. Matsuura and N. Koizumi, “A Method of Scooping Mid-Air Images on Water Surface,” In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces (2018).
7. D. Miyazaki, Y. Maeda, and S. Maekawa, “Floating three-dimensional image display using micro-mirror array imaging element,” Proc SPIE 9495, 949508 (2018). [CrossRef]
8. S. Maekawa and S. Markon, “Airflow interaction with Floating images,” SIGGRAPH ASIA 2009 Art Gallery & Emerging Technologies. SIGGRAPH ASIA’09 Art Gallery & Emerging Technologies, p. 61 (2009).
9. Y. Ueda, K. Iwazaki, M. Shibasaki, Y. Mizushina, M. Furukawa, H. Nii, K. Minamizawa, and S. Tachi, “Mid-air autostereoscopic display for seamless interaction with mixed reality environments,” Proc. SIGGRAPH’14, Emerging Technologies 10, 1–10 (2014). [CrossRef]
10. H. Suginohara, H. Kikuta, Y. Nakamura, K. Minami, and H. Yamamoto, “An Aerial Display: Passing through a Floating Image Formed by Retro-Reflective Reimaging,” SID Symposium Digest of Technical Papers 48(1), 406–409 (2017). [CrossRef]
11. D. Miyazaki, S. Onoda, Y. Maeda, and T. Mukai, “Blurring Correction for Aerial Image Formed by Dihedral Corner Reflector Array,” CLEO Pacific Rim, paper 25B1_3(2015).
12. C.B. Burckhardt, R. J. Collier, and E. T. Doherty, “Formation and Inversion of Pseudoscopic Images,” Appl. Opt. 7(4), 627–631 (1968). [CrossRef]
13. H. Yamamoto and S. Suyama, “Aerial imaging by Retro-Reflection (AIRR),” SID’13 DIGEST, 895 (2013).
14. H. Yamamoto, Y. Tomiyama, and S. Suyama, “Floating aerial LED signage based on aerial imaging by retro-reflection (AIRR),” Opt. Express 22(22), 26919–26924 (2014). [CrossRef]
15. S. Maekawa, K. Nitta, and O. Matoba, “Advances in Passive Imaging Elements with Micromirror Array,” Proc. SPIE 6392, 63920E (2006). [CrossRef]
16. ASKA 3D. https://aska3d.com/en/technology.html Accessed 2 September 2021.
17. R. Zhang and H. Hua, “Imaging quality of a retroreflective screen in Head-Mounted Projection Display,” J. Opt. Soc. Am. A 26(5), 1240–1249 (2009). [CrossRef]
18. M. Nakajima, K. Onuki, I. Amoimori, and H. Yamamoto, “Polarization State Analysis for Polarized Aerial Imaging by Retro-Reflection (pAIRR),” Proc. IDW 22, 429–432 (2015).
19. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13(4), 600–612 (2004). [CrossRef]
20. N. Mukojima, M. Yasugi, Y. Mizutani, T. Yasui, and H. Yamamoto, “Deep-Learning-Assisted Single-Pixel Imaging for Gesture Recognition in Consideration of Privacy,” IEICE Transactions on Electronics E105.C(2), 79–85 (2022). [CrossRef]
21. N. Kawagishi, R. Kakinuma, and H. Yamamoto, “Aerial image resolution measurement based on the slanted knife edge method,” Opt. Express 28(24), 35518–35527 (2020). [CrossRef]
