Open Access
Add to BibSonomy
Abstract
Augmented reality (AR) three-dimensional (3D) display is the hardware entrance of metaverse and attracts great interest. The fusion of physical world with 3D virtual images is non-trivial. In this paper, we proposed an AR 3D display based on a pixelated volume holographic optical element (P-VHOE). The see-through combiner is prepared by spatial multiplexing. A prototype of AR 3D display with high diffraction efficiency (78.59%), high transmission (>80%) and non-repeating views is realized. Virtual 3D objects with high fidelity in depth is reconstructed by P-VHOE, with a complex wavelet structural similarity (CW-SSIM) value of 0.9882. The proposed prototype provides an efficient solution for a compact glasses-free AR 3D display. Potential applications include window display, exhibition, education, teleconference.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
As the metaverse hardware entrance, augmented reality (AR) display has received extensive attention. There is an increasing demand in applications of education, medical, military and online office [1,2]. By combining virtual images with physical objects, AR display can greatly enhance human perception of information. Glasses-free AR display can be divided into reflection-type AR display and optical see-through AR display. Compared with reflection-type AR display, optical see-through AR display allows the direct perception of physical world through a transparent combiner. Elegant setup and natural fusion of virtuality with reality makes see-through AR display a favorable solution for metaverse [3–7].
Extensive studies have been presented in the field of glasses-free 3D display, such as multiview 3D displays [8–15], holographic displays [16–23], and volumetric displays [24–26]. Geometric optics based view modulator has been extensively studied to create multiple views. Most recently, pixelated nanostructure complex has been proposed to reconstruct the lightfield with the features of light weight, thin form factor, and large motion parallax [27–32]. However, neither geometric optics based nor planar optics based view modulator is designed for see-through purpose. Indeed, in the field of AR 3D display, the design of combiners is challenging because it requires high transparency in addition to the function of 3D lightfield reconstruction. An optical see-through AR display system with dual virtual image planes was proposed by time-multiplexing. Reflective polarization-dependent lenses successively project left-handed and right-handed circularly polarized images for multiple depth cue [33]. Furthermore, continuous depth cue can be generated by a reflective volume holography based integral imaging. The hologram records the optical function of a lenticular lens array or a microlens array while keeps the features of high transparency [34–37]. However, AR 3D display based on integrated imaging inherit self-repeating views. Incorrect motion parallax results in false depth cue, which is a critical issue in AR display. Most recently, a 32-inch glasses-free AR 3D display was proposed by combining a laser projector with a holographic see-through combiner. Pixelated metagratings generated 16 views to form natural motion parallax over a viewing angle of 47°, while a transmission larger than 75% is maintained over the entire visible spectrum. In another study, multilevel blazed gratings are adopted to increase the light efficiency. The sparsely arranged nanostructures provide a promising solution to create a large viewing angle with correct motion parallax for AR application. Yet the spatial resolution was scarified in the system to increase the transmittance [30,37].
In this paper, we propose an optical combiner based on pixelated volume holographic optical element (P-VHOE) for AR 3D display. P-VHOE reconstructs multiple views without blocking ambient light. By integrating the P-VHOE with an off-the-shelf purchased projector, an AR 3D display with high diffraction efficiency, high transmittance and natural motion parallax is achieved.
2. System design
Figure 1 illustrates the schematic diagram of the proposed display system consisting of a P-VHOE and a projector. The projector provides multi-view images on the P-VHOE. Each pixel in the P-VHOE is well aligned with the corresponding pixel from the projector. Through the P-VHOE, the emitted light is modulated to multiple views. The P-VHOE is prepared by reflective volume holography. With high transmittance and superior light modulating capability, P-VHOE serves as a see-through combiner to reconstruct 3D virtual image and to fuse reality with virtuality.
One should note the correct motion parallax is of great importance for AR 3D display. Self-repeating views produced by a microlens array or cylindrical lens array provide limited motion parallax or even incorrect motion parallax. Hence, we explore view modulator that provides correct motion parallax for AR application. In prior studies, we have proposed view modulators covered with sparsely covered blazed gratings for light manipulation and 3D lightfield reconstruction [28,30]. Simply, we design surface-relief structures to modulate the emitted beam from each pixel. From the perspective of each view, the pixelated structures focus light beams to the view and work as a focusing lens (Fig. 2(a)). So four grayscale diffractive lenses (GDL) are intertwined to form a view modulator [32]. In a 4-view 3D display, a voxel is comprised of four pixels from 4 GDL. When illuminated by a collimated backlight, the intertwined GDL based view modulator forms 4 views for natural motion parallax, as shown in Fig. 2(b).
Fig. 2. (a) 4 GDL with horizontally shifted optical axis. (b) Four views are generated by a view modulated covered with intertwined GDLs
The diffraction efficiency of GDL directly affects the light efficiency of holographically recorded P-VHOE. The focusing efficiency of GDL can be expressed as
Illuminated by a collimated planar backlight, the intertwined GDL based view modulator is incompatible with AR display. Therefore, a P-VHOE was adopted as an optical combiner by recording the wavefront of a intertwined GDL based view modulator. According to the Kogelnik coupled wave theory [38], the diffraction efficiency of reflective volume holography can be described as
Since the diffraction efficiency is a vital parameter that greatly affects the display performance, we further theoretically analyzed the properties in Fig. 3. Figure 3(a) shows that the diffraction efficiency climbs up to 100% when the incident angle is larger than 60°. So we choose an incident angle of 69.86° in the experiment. Figure 3(b) illustrates the dependence of diffraction efficiency on absorption ecoefficiency and medium thickness. While a photopolymer with a thickness of 5 µm cannot provide a maximized diffraction efficiency, the diffraction efficiency drops rapidly as the increment of the absorption coefficient in a thicker recording medium. Therefore, the thickness of the photopolymer is set to 15 µm in the experiment. We further analyzed the angular and spectral sensitivity of reflective volume holography. From Fig. 3(c), the diffraction efficiency maintains high when the deviation of incident angle is within ±5°. From the system level, adequate tolerance of incident angle is favorable to provide a robust and user-friendly AR display. Moreover, the diffraction efficiency is very sensitive to the irradiance wavelength with a full width at half maximum of 3.02 nm (Fig. 3(d)). Benefit from the sensitive spectral response, no chromatic aberration will be observed even when a white light source is adopted to reconstruct the phase information from the photopolymer.
Fig. 3. Theoretical analysis of diffraction efficiency. (a) Diffraction efficiency dependence on incident angle. (b) Diffraction efficiency dependence on absorption coefficient and material thicknesses.(c) Sensitivity of diffraction efficiency on incident angle. (d) Sensitivity of diffraction efficiency over irradiance wavelength
The recording and reconstructing principle of the P-VHOE is introduced in Fig. 4. In the recording process, a plane wave signal beam passes through the view modulator and interferes with a spherical wave reference beam. The optical properties of the view modulator are then recorded in the photopolymer. The wavefront information of the view modulator is then reconstructed by a displaying beam as the reference beam in the recording scheme. As the P-VHOE modulates only Bragg-matched lights, Bragg mismatched lights from the physical world directly pass through the optical combiner without deflection. P-VHOE recorded by a view modulator covered with GDL eliminated self-repeating views produced by a lens array.
3. Experiments and results
3.1 Fabrication of the P-VHOE
We first fabricated an intertwined GDL based view modulator by a self-developed grayscale lithography system (MICROLAB, SVG Optronics). A detailed description of the fabrication process can be found in the prior studies. The resolution of view modulator is 200×200 pixels, and the size of each pixel is 209.088×209.088 µm. The light efficiency of the GDL is 82% in experiment.
The experimental setup for fabricating P-VHOE is shown in Fig. 5. The central wavelength of the laser (IK4151R-C, KIMMON) is 442nm, in consistent with the designed wavelength of the GDL based view modulator. A beam splitter divides the laser into two beams: the signal beam and the reference beam. The signal beam is expanded and normally incident on the view modulator. The reference beam illuminates the view modulator at an angle of 69.86° in the form of a spherical wave. The photopolymer (PP-P-G) is provided by Beijing Spectrum Treasure Printing Technology with a thickness of 15 µm. In the recording process, the photopolymer is exposed to a light intensity of 30mJ/cm2. The device is then UV-cured by a UV lamp at a power of 1kW for 3 mins. Finally, the P-VHOE is baked in the oven at 100°C for 15 mins.
Fig. 5. (a) Schematic diagram of the experimental setup for recording the P-VHOE. ES, electronic shutter; BS, beam splitter; M, mirror; SF, spatial filter; CL, collimating lens; PP, photopolymer plate.
3.2 Characteristics of the P-VHOE
The P-VHOE diffracts the incident reference beam into views with large diffraction angle. As a result, it is difficult to directly measure its diffraction spectrum. Alternatively, we calculate the diffraction efficiency by measuring its transmission spectrum and reflection spectrum. The transmittance and reflectance over the visible spectrum were measured by a spectrometer (SPECORD 210 PLUS, Analytikjena). As shown in Fig. 6, the transmittance of Bragg-mismatched light is higher than 80% for the light wavelength larger than 500nm. The minimum transmittance is 13.08% at the wavelength of 431.5nm, suggesting a slightly shift of the Bragg condition. The reflectance of P-VHOE is 8.33%. If we ignore scattering and absorption of the photopolymer, the diffraction efficiency is calculated as 78.59%.
Since the P-VHOE is working as a see-through combiner, the transparency is a critical parameter. We evaluated the transparency of P-VHOE and GDL, respectively, by placing a physical object behind them. The images are captured by a CCD camera (D810, Nikon). Here we introduce the complex wavelet structural similarity (CW-SSIM) index evaluation function to analyze image similarity between the original image and the images pass through a combiner, namely the intertwined GDL or the P-VHOE [39]. The CW-SSIM is defined as
3.3 Characteristics of the AR 3D display
Finally, we built an AR 3D display prototype by integrating a laser projector (ZH33, Optoma) with P-VHOE, as shown in Fig. 8(a). The distance between the projector and P-VHOE is 570 mm, and the incident angle is 69.86°, corresponding to the Bragg condition. In the proof-of-concept experiment, 200×200 pixels are projected on and aligned with the pixels of P-VHOE.
Fig. 8. (a) Experimental setup of the proposed AR 3D display system. (b) 4 separated convergent viewpoints at a distance of 500 mm of intertwined GDL based view modulator. (c) The intensity distribution of the convergent viewpoints of intertwined GDL and P-VHOE.
The view distribution is captured by a CCD camera (D810, Nikon) placed at the focal plane (Fig. 8(b)). The angular resolution is 3°. With an angular divergence of 0.85°, the views are well separated. The viewing angle of natural motion parallax is 9.85°. From the view distribution, the wavefront information of intertwined GDL based view modulator has been recorded in P-VHOE with high fidelity, as shown in Fig. 8(c). Compared with intertwined GDL, the view intensity of P-VHOE decreases, which verified a diffraction efficiency of less than 100%. Material absorption and insufficient refractive index modulation attribute to the experimental deviation.
We further test the crosstalk between views by four letters “S”, “U”, “D”, and “A”. Four letters are nicely separated and projected to four views by the P-VHOE combiner (Fig. 9(a)). Then we built a 3D model in the 3D computer graphic software 3DS MAX to test the display performance. From Fig. 9 (b), the virtually projected label “36km/h” has a good fusion with a toy car, placed 100 mm behind the P-VHOE. The 3D arrow fuses well with the physical road, providing a natural motion parallax with correct depth cue. In addition, video-rate 3D display is achieved by simply projecting video clip with multiview information on the P-VHOE (Visualization 1).
Fig. 9. (a) Experimental results of viewpoints separation. (b) Experimental results of the AR 3D display (Visualization 2).
4. Discussion and conclusions
In this paper, we proposed an AR 3D display system based on a P-VHOE and a projector. The P-VHOE is fabricated by recording the wavefront information of intertwined GDLs with reflective volume holography. The viewing angle of the P-VHOE is 9.85° with an angular separation of 3°. P-VHOE possesses a transmittance higher than 80% for the wavelength of 500 nm and above. The diffraction efficiency is as high as 78.59%. The CW-SSIM value of images captured through a P-VHOE is 0.9882, similar to the value of images captured through a blank glass (0.9974). In an AR 3D display prototype, we demonstrate a natural fusion between virtual 3D images and physical objects.
Although monochromatic AR 3D display is demonstrated, full-color AR 3D display can be realized by spatial multiplexing. Moreover, we demonstrated a 4-view 3D display with horizontally parallax. Full motion parallax with tailorable view distribution can be easily achieved by re-organizing the GDLs in the view modulator. The number of views and the viewing angle can be further increased by optimizing the GDLs or adopting time division multiplexing.
In summary, with the feature of high transmittance, high light efficiency, high transparency, and non-repeating views, P-VHOE serves as an excellent candidate for optical combiner. The AR 3D display based on P-VHOE provides natural motion parallax. Consisting of only two components, the AR 3D display setup is simple, robust, and efficient. The proposed AR 3D display system provides a feasible solution for various commercial applications, such as head-up displays, head-mounted displays, window display, exhibition and advertisement.
Funding
National Key Research and Development Program of China (2021YFB3600500); National Natural Science Foundation of China (61975140, 62075145); Jiangsu Provincial Key Research and Development Program (BE2021010); Leading Technology of Jiangsu Basic Research Plan (BK20192003); Priority Academic Program Development of Jiangsu Higher Education Institutions.
Disclosures
The authors declare no conflicts of interest.
Data availability
The paper and Supplementary Information provide the data that support the results of our study. Additional supporting data generated during the study can be obtained from the corresponding authors.
References
1. X. Li, W. Yi, H. L. Chi, X. Wang, and A. P. Chan, “A critical review of virtual and augmented reality (VR/AR) applications in construction safety,” Automation in Construction 86, 150–162 (2018). [CrossRef]
2. J. Xiong, E. L. Hsiang, Z. Q. He, T. Zhan, and S. T. Wu, “Augmented reality and virtual reality displays: emerging technologies and future perspectives,” Light: Sci. Appl. 10(1), 1–30 (2021). [CrossRef]
3. K. Wakunami, P.-Y. Hsieh, R. Oi, T. Senoh, H. Sasaki, Y. Ichihashi, M. Okui, Y.-P. Huang, and K. Yamamoto, “Projection-type see-through holographic three-dimensional display,” Nat. Commun. 7(1), 12954 (2016). [CrossRef]
4. S. Liu, Y. Li, P. Zhou, Q. Chen, and Y. Su, “Reverse-mode PSLC multi-plane optical see-through display for AR applications,” Opt. Express 26(3), 3394–3403 (2018). [CrossRef]
5. K. Rathinavel, G. Wetzstein, and H. Fuchs, “Varifocal occlusion-capable optical see-through augmented reality display based on focus-tunable optics,” IEEE Trans. Visual. Comput. Graphics 25(11), 3125–3134 (2019). [CrossRef]
6. X. Shen and B. Javidi, “Large depth of focus dynamic micro integral imaging for optical see-through augmented reality display using a focus-tunable lens,” Appl. Opt. 57(7), B184–B189 (2018). [CrossRef]
7. C. Yao, D. Cheng, T. Yang, and Y. Wang, “Design of an optical see-through light-field near-eye display using a discrete lenslet array,” Opt. Express 26(14), 18292–18301 (2018). [CrossRef]
8. D. Teng, Y. Xiong, L. Liu, and B. Wang, “Multiview three-dimensional display with continuous motion parallax through planar aligned OLED microdisplays,” Opt. Express 23(5), 6007–6019 (2015). [CrossRef]
9. L. Liu, Z. Pang, and D. Teng, “Super multi-view three-dimensional display technique for portable devices,” Opt. Express 24(5), 4421–4430 (2016). [CrossRef]
10. H. Liao, “Super long viewing distance light homogeneous emitting three-dimensional display,” Sci. Rep. 5(1), 1–5 (2015). [CrossRef]
11. X. Xia, X. Zhang, L. Zhang, P. Surman, and Y. Zheng, “Time-multiplexed multi-view three-dimensional display with projector array and steering screen,” Opt. Express26(12), 15528–15538 (2018). [CrossRef]
12. L. Yang, X. Sang, X. Yu, B. Liu, B. Yan, K. Wang, and C. Yu, “A crosstalk-suppressed dense multi-view light-field display based on real-time light-field pickup and reconstruction,” Opt. Express 26(26), 34412–34427 (2018). [CrossRef]
13. Y. Meng, Z. Yu, C. Zhang, Y. Wang, Y. Liu, H. Ye, and L. L. Chen, “Numerical simulation and experimental verification of a dense multi-view full-resolution autostereoscopic 3D-display-based dynamic shutter parallax barrier,” Appl. Opt. 58(5), A228–A233 (2019). [CrossRef]
14. W. Wang, G. Chen, Y. Weng, X. Weng, X. Zhou, C. Wu, T. Guo, Q. Yan, Z. Lin, and Y. Zhang, “Large-scale microlens arrays on flexible substrate with improved numerical aperture for curved integral imaging 3D display,” Sci. Rep. 10(1), 1–9 (2020). [CrossRef]
15. Y. Meng, Y. Lyu, L. L. Chen, Z. Yu, and H. Liao, “Motion parallax and lossless resolution autostereoscopic 3D display based on a binocular viewpoint tracking liquid crystal dynamic grating adaptive screen,” Opt. Express 29(22), 35456–35473 (2021). [CrossRef]
16. G. Li, D. Lee, Y. Jeong, J. Cho, and B. Lee, “Holographic display for see-through augmented reality using mirror-lens holographic optical element,” Opt. Lett. 41(11), 2486–2489 (2016). [CrossRef]
17. P. Zhou, Y. Li, S. Liu, and Y. Su, “Compact design for optical-see-through holographic displays employing holographic optical elements,” Opt. Express 26(18), 22866–22876 (2018). [CrossRef]
18. S.-B. Kim and J.-H. Park, “Optical see-through Maxwellian near-to-eye display with an enlarged eyebox,” Opt. Lett. 43(4), 767–770 (2018). [CrossRef]
19. H. Amano, Y. Ichihashi, T. Kakue, K. Wakunami, H. Hashimoto, R. Miura, T. Shimobaba, and T. Ito, “Reconstruction of a three-dimensional color-video of a point-cloud object using the projection-type holographic display with a holographic optical element,” Opt. Express 28(4), 5692–5705 (2020). [CrossRef]
20. Y. Li, Q. Yang, J. Xiong, K. Yin, and S.-T. Wu, “3D displays in augmented and virtual realities with holographic optical elements,” Opt. Express 29(26), 42696–42712 (2021). [CrossRef]
21. T. Wu, J. Ma, C. Wang, H. Wang, and P. Su, “Full-Color See-Through Three-Dimensional Display Method Based on Volume Holography,” Sensors 21(8), 2698 (2021). [CrossRef]
22. S. Lee, C. Jang, J. Cho, J. Yeom, J. Jeong, and B. Lee, “Viewing angle enhancement of an integral imaging display using Bragg mismatched reconstruction of holographic optical elements,” Appl. Opt. 55(3), A95–A103 (2016). [CrossRef]
23. Z. He, X. Sui, G. Jin, and L. Cao, “Progress in virtual reality and augmented reality based on holographic display,” Appl. Opt. 58(5), A74–A81 (2019). [CrossRef]
24. D. Smalley, E. Nygaard, K. Squire, J. Van Wagoner, J. Rasmussen, S. Gneiting, K. Qaderi, J. Goodsell, W. Rogers, and M. Lindsey, “A photophoretic-trap volumetric display,” Nature 553(7689), 486–490 (2018). [CrossRef]
25. R. Hirayama, D. M. Plasencia, N. Masuda, and S. Subramanian, “A volumetric display for visual, tactile and audio presentation using acoustic trapping,” Nature 575(7782), 320–323 (2019). [CrossRef]
26. K. Suzuki, Y. Fukano, and H. Oku, “1000-volume/s high-speed volumetric display for high-speed HMD,” Opt. Express 28(20), 29455–29468 (2020). [CrossRef]
27. W. Wan, W. Qiao, D. Pu, R. Li, C. Wang, Y. Hu, H. Duan, L. J. Guo, and L. Chen, “Holographic sampling display based on metagratings,” iScience 23(1), 100773 (2020). [CrossRef]
28. F. Zhou, J. Hua, J. Shi, W. Qiao, and L. Chen, “Pixelated blazed gratings for high brightness multiview holographic 3D display,” IEEE Photonics Technol. Lett. 32(5), 283–286 (2020). [CrossRef]
29. J. Hua, D. Yi, W. Qiao, and L. Chen, “Multiview holographic 3D display based on blazed Fresnel DOE,” Opt. Commun. 472, 125829 (2020). [CrossRef]
30. J. Shi, J. Hua, F. Zhou, M. Yang, and W. Qiao, “Augmented Reality Vector Light Field Display with Large Viewing Distance Based on Pixelated Multilevel Blazed Gratings,” in Photonics, (Multidisciplinary Digital Publishing Institute, 2021), p. 337.
31. J. Hua, E. Hua, F. Zhou, J. Shi, C. Wang, H. Duan, Y. Hu, W. Qiao, and L. Chen, “Foveated glasses-free 3D display with ultrawide field of view via a large-scale 2D-metagrating complex,” Light: Sci. Appl. 10(1), 1–9 (2021). [CrossRef]
32. F. Zhou, F. Zhou, Y. Chen, J. Hua, W. Qiao, and L. Chen, “Vector light field display based on an intertwined flat lens with large depth of focus,” Optica 9(3), 288–294 (2022). [CrossRef]
33. Y. Li, Q. Yang, J. Xiong, K. Li, and S.-T. Wu, “Dual-depth augmented reality display with reflective polarization-dependent lenses,” Opt. Express 29(20), 31478–31487 (2021). [CrossRef]
34. K. Hong, J. Yeom, C. Jang, J. Hong, and B. Lee, “Full-color lens-array holographic optical element for three-dimensional optical see-through augmented reality,” Opt. Lett. 39(1), 127–130 (2014). [CrossRef]
35. H. Zhang, H. Deng, J. Li, M. He, D. Li, and Q. Wang, “Integral imaging-based 2D/3D convertible display system by using holographic optical element and polymer dispersed liquid crystal,” Opt. Lett. 44(2), 387–390 (2019). [CrossRef]
36. H. Deng, C. Chen, M. He, J. Li, H. Zhang, and Q. Wang, “High-resolution augmented reality 3D display with use of a lenticular lens array holographic optical element,” J. Opt. Soc. Am. A 36(4), 588–593 (2019). [CrossRef]
37. J. Shi, W. Qiao, J. Hua, R. Li, and L. Chen, “Spatial multiplexing holographic combiner for glasses-free augmented reality,” Nanophotonics 9(9), 3003–3010 (2020). [CrossRef]
38. H. Kogelnik, “Coupled wave theory for thick hologram gratings,” in Landmark Papers On Photorefractive Nonlinear Optics (World Scientific, 1995), pp. 133–171.
39. M. P. Sampat, Z. Wang, S. Gupta, A. C. Bovik, and M. K. Markey, “Complex wavelet structural similarity: A new image similarity index,” IEEE Trans. on Image Process. 18(11), 2385–2401 (2009). [CrossRef]
