Open Access
Add to BibSonomy
Abstract
In this paper, a locally controllable two-dimensional (2D)/ three-dimensional (3D) mixed display system and corresponding image generation method are proposed. The proposed system is mainly composed of a collimating backlight module (CBM) and a light control module (LCM). The CBM provides collimating polarized light. The LCM modulates a part of the collimating polarized light to form point light sources for 3D display and the other part to form scattered light sources for 2D display. The 2D and 3D display states can be locally controlled by using a pixelated mask loaded on a polarization switching layer. In addition, a corresponding image generation method is proposed. According to observer’s demand, the parallax image is divided into target image area and residual image area by using deep learning matting algorithm, and a 2D/3D mixed light field image with full parallax 3D target image and high-resolution 2D residual image is generated. We developed a prototype based on the proposed locally controllable 2D/3D mixed display structure and generated two sets of 2D/3D mixed light field image with different target objects and residual objects from the same parallax images. The experimental results demonstrated the effectiveness of our proposed system and the corresponding image generation method. High-resolution 2D image and full parallax 3D image were displayed and locally switched in the experimental system.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
As an indispensable part of the modern world, display technologies have been developing rapidly. A high-definition two-dimensional (2D) display screen can present the details of the object as small as 33 µm [1]. Light field display technologies such as integral imaging [2,3], holographic display [4,5], and volumetric display [6,7] can reconstruct vivid three-dimensional (3D) images with various physiological depth cues similar to a real 3D object. Both 2D and 3D display technologies have been extensively applied in industrial design, education, advertisement, especially in the medical domain. Take surgical operation for example, an ideal display device should have the following three characteristics. Firstly, it should be able to present high-definition texture images with the details of human tissues clearly. Secondly, it should be able to display 3D depth information with various natural physiological depth cues, in order to help surgeons intuitively see the spatial hierarchy of internal tissues. Finally and most importantly, the high-definition texture image and 3D depth information of each tissue should be able to independently controlled and arbitrarily switched to assist the surgical operation. However, to the best of our knowledge, there are barely approaches that meets all the requirements in one single display.
Integral imaging based 2D/3D convertible or mixed display has been gradually studied, because of its relatively compact form factor and fairly low cost, compared with other light field display technologies. In the earliest proposal of Prof. Lee, a polymer-dispersed liquid crystal combined with a lens array modulated the collimated light to generate point light source array or scattering backlight for 2D/3D convertible display [8,9]. But the driving voltage of the polymer-dispersed liquid crystal should be no less than 60V. A projection-type 2D/3D convertible display system was proposed, in which two projectors were needed to project 2D image and 3D image onto a concave half mirror array, respectively [10]. To reduce the system volume, a projector combined with a holographic optical element was proposed [11]. Prof. Lee and Prof. Wang used light-emitting diode arrays combined with a diffuser [12] or a light guide plate [13] to realize 2D/3D convertible display. In our former research, we used a quarter-wave retarding film with pinholes combined with a polarization-dependent liquid crystal micro-lens array (LCMLA) to realize 2D/3D mixed display [14]. But the image quality was restricted by the density of the pinholes. In addition, pinhole array on a polarizer [15,16] and liquid crystal active lens [17] were also good candidates to modulate the perpendicular polarized light for 2D/3D convertible display. However, the problems of low light efficiency and high refresh rate are ineluctable. Recently, the geometric phase lens array was used to replace the conventional lens array [18]. The chromatic aberration problem of geometric phase lens should be considered. Besides, there would be crosstalk at the boundary of 2D image and 3D image. Although these approaches could realize 2D/3D convertible or mixed display, it was still quite challenging to meet all the requirements listed above and avoid the problems, such as fast refresh rate, high driving voltage, large system volume, and low optical efficiency.
To meet the above requirements, we proposed a locally controllable 2D/3D mixed display system and corresponding image generation method, in which the display screen can be locally controlled to present high-definition 2D texture image or natural 3D image according to observer’s demand. Besides, our proposed system has no need of fast refresh rate or high driving voltage, and has relatively compact system volume and high optical efficiency.
2. Principle of the proposed locally controllable 2D/3D mixed display
The schematic diagram of the proposed locally controllable 2D/3D mixed display is shown in Fig. 1. It mainly consists of a collimating backlight module (CBM), a light control module (LCM), and a transmission-type display panel.
The CBM contains a point light source, an absorbing polarizer, and a Fresnel lens. The point light source is placed at the focal point of the Fresnel lens. The absorbing polarizer whose transmission axis is paralleled with the y-axis is attached onto the Fresnel lens. The light emitted from the point light source is transformed into collimated y-polarized light after going through the CBM. The LCM is composed of a polarization switching layer, a polarization-dependent LCMLA, and a scattering polarizer. The polarization switching layer is electrically controlled by loading a pixelated mask, so as to locally switch the polarization direction of the incident light into x-polarized light or y-polarized light. A twisted nematic liquid crystal panel is a good candidate to realize locally switch of the polarization direction. The polarization-dependent LCMLA has focusing effect to the x-polarized light and transmission effect to the y-polarized light [14], hence, the collimated x-polarized light forms a point light source array at the focal plane of the polarization-dependent LCMLA, and y-polarized light is still collimated. Figure 2 shows the optical properties of the scattering polarizer. On one hand, the scattering polarizer has scattering effect when the polarization direction of incident light is parallel to its scattering axis. On the other hand, it has transmission effect when the polarization direction of incident light is perpendicular to its scattering axis. Behind the polarization-dependent LCMLA, the scattering polarizer whose scattering axis is paralleled to the y-axis scatters y-polarized light and allows the x-polarized light to pass directly without any loss. To effectively utilize the display panel as well as avoid the overlap of adjacent point light sources, the transmission-type display panel locates in front of the polarization-dependent LCMLA with twice focal length distance. The point light source array formed by x-polarized light directionally illuminates onto the transmission-type display panel to reconstruct 3D images. While the scattered y-polarized light uniformly illuminates the transmission-type display panel for 2D display. The pixelated mask can be arbitrarily generated and locally controlled, so as to change the 3D display areas and 2D display areas according to observer’s demand. Hence locally controllable 2D/3D mixed display is realized.
3. Generation of locally controllable 2D/3D mixed light field image
A direct way to generate 2D/3D mixed light field image is shooting the scene twice, one is 2D capture using a normal camera and the other is 3D capture using a light field camera, and then merging them according to the occlusion relationship [19]. But the acquisition process is complicated, and it is difficult to capture dynamic video due to the alternate use of the normal camera and the light field camera. To solve the problem, we propose a method to generate locally controllable 2D/3D mixed light field image in which only one single shooting is needed.
Figure 3 shows the flow diagram of our proposed locally controllable 2D/3D mixed light field image generation method. A set of parallax images are captured by a light field camera or a camera array. According to the observer’s demand, objects in the parallax images are roughly divided into target objects for 3D display and residual objects for 2D display. Then, a trimap is generated through the dilation and erosion based morphological processing operations. The parallax images combined with the trimap generate the alpha matte of the target objects by using a deep learning matting (DLM) algorithm [20]. Thus, each parallax image is accurately segmented into the target image area and residual image area. The target image areas are used to synthesize a full parallax 3D target image by using our previously proposed elemental image array generation (EIAG) algorithm [21]. A fast super-resolution convolutional neural network (FSRCNN) algorithm is used to generate a high-resolution 2D residual image [22]. As a result, the high-resolution 2D residual image and the full parallax 3D target image are fused into a 2D/3D mixed light field image. A pixelated mask is generated simultaneously. When the pixelated mask is loaded onto the polarization switching layer, the white region changes the collimated y-polarized light into x-polarized light for 3D display, while the black region maintains the collimated y-polarized light for 2D display. In the generation process of 2D/3D mixed light field image, target objects and residual objects can be arbitrarily selected according to observer’s demand. Therefore, we can generate locally controllable 2D/3D mixed light field images. The proposed method simplifies the acquisition process, realizes local controlling of 2D and 3D image area, and provides the possibility of real-time 2D/3D mixed light field image generation.
4. Experiments and results
In the experiment, we built up a proof-of-concept prototype to evaluate the feasibility of the proposed locally controllable 2D/3D mixed display structure and corresponding image generation method. The experimental prototype is shown in Fig. 4. The transmission-type display panel is a twisted nematic liquid crystal panel with a resolution of 3840 × 2160 pixels and a pixel pitch of 0.09 mm. Another liquid crystal panel with the same resolution and pixel pitch as the transmission-type display panel was acted as a polarization switching layer. The driving voltage of the polarization switching layer is 5 V. The polarization-dependent LCMLA was fabricated and the maximum birefringence $\varDelta n$ of liquid crystal is about 0.23 at 589 nm and 20°C. The polarization-dependent LCMLA has a lens pitch of 1 mm, a focal length of 4.2 mm, and its total size is 100 mm×100 mm, as shown in Fig. 5(a). To evaluate the optical performance of the fabricated polarization-dependent LCMLA, we built up a testing system composed of a light source, a green filter, a collimator, an absorbing polarizer, the polarization-dependent LCMLA, and a complementary metal oxide semiconductor (CMOS) camera. The light from the incandescent lamp passed through the green filter and the collimator to form a beam of collimated light. The direction of the absorbing polarizer was adjusted to make the polarization-dependent LCMLA in focusing effect. A set of spots were observed on the CMOS camera. By moving the camera position and observing the change of these spots, the distance from the brightest and thinnest spots to the polarization-dependent LCMLA was the focal length of the polarization-dependent LCMLA. Similarly, the transmission effect of the polarization-dependent LCMLA can also be captured on the CMOS camera by adjusting the orientation of the absorbing polarizer. Figures 5(b) and 5(c) show the good focusing effect of the polarization-dependent LCMLA to the x-polarized light and transmission effect to the y-polarized light, respectively. Because the transmittances of the polarization-dependent LCMLA and the scattering polarizer are 86% and 94%, respectively, the prototype can maintain high optical efficiency. The reconstructed 2D/3D mixed light field images were captured by a camera (D810, Nikon). The detailed specifications of the prototype are listed in Table 1.
Fig. 5. (a) Fabricated polarization-dependent LCMLA, (b) focusing effect and (c) transmission effect of the polarization-dependent LCMLA.
Table 1. Specifications of the developed 2D/3D mixed display prototype.
Figure 6(a) shows the spatial distribution of objects in the 3D scene, which includes a “green virus”, a “blue virus”, a “human phantom” and a dark blue background. These objects are 60 mm, 50 mm, and 10 mm away from the background, respectively. 11× 11 parallax images are captured and each of them has 275 × 275 pixels. Figure 6(b) shows two samples of the (6,1)-th and (6,11)-th parallax images, respectively. In order to verify the feasibility of the locally controllable 2D/3D mixed display, we carried out two experiments.
In the first experiment, the “green virus” and the “blue virus” were selected as target objects for 3D display, and the “human phantom” and the dark blue background were residual objects for 2D display. Figure 7(a) shows the alpha mattes of the target objects after segment. Figure 7(b) shows two examples of target image areas extracted from the (6,1)-th and (6,11)-th parallax images, respectively. We can see that the target image areas and residual image areas are segmented accurately. Even though the “blue virus” in the target image has similar color with the dark blue background in the residual image, the DLM algorithm can still obtain satisfying segmentation results. Figure 7(d) shows the full parallax 3D target image synthesized from target image areas, and its resolution is 1100 × 1100 pixels. To match the resolutions of the 2D residual image and the full parallax 3D target image, the resolution of the 2D residual image was enhanced by four times by using the FSRCNN algorithm. Figure 7(c) shows the resolution enhanced 2D residual image extracted from the (6, 6)-th parallax image, and its resolution is also 1100 × 1100 pixels.
Fig. 7. (a) Samples of the alpha mattes of the target objects, (b) samples of the target image areas, (c) high-resolution 2D residual image, and (d) full parallax 3D target image.
Figures 8(a) and 8(b) show the pixelated mask and the generated 2D/3D mixed light field image, which were loaded onto the polarization switching layer and the transmission-type display panel, respectively. Figures 8(c) and 8(d) show the reconstructed results of the 2D/3D mixed image at different viewpoints. When the viewpoint moves from left to right, we could clearly see that the “green virus” and “blue virus” had relative movement, and the horizontal parallax was obvious, as shown in the red dotted box. Additionally, the “green virus” had larger parallax than “blue virus”, which coincided with their 3D depths. The display result proved that the 3D target images could be successfully reconstructed in 3D state. At the same time, the 2D residual images, including the “human phantom” and the dark blue background, were also clearly presented in 2D state without any distortion.
Fig. 8. (a) Pixelated mask and (b) 2D/3D mixed light field image in which “green virus” and “blue virus” are selected as target objects, (c) left and (d) right views of the displayed 2D/3D mixed image.
In the second experiment, only “green virus” was selected as a target object, and the others were residual objects. Similar to the first experiment, a new pixelated mask and 2D/3D mixed light field image were generated, as shown in Figs. 9(a) and 9(b). Figures 9(c) and 9(d) show the reconstructed results of the new 2D/3D mixed image at different viewpoints. Different from the first experiment, only “green virus” showed the horizontal parallax as the viewpoint changes, which demonstrated that the new 3D target image was successfully reconstructed in 3D state. Meanwhile, the new 2D residual objects were clearly displayed in 2D state. Therefore, the proposed locally controllable 2D/3D mixed display and its corresponding image generation method were proved to be effective and feasible.
Fig. 9. (a) Pixelated mask and (b) 2D/3D mixed light field image in which only “green virus” was selected as target objects, (c) left and (d) right views of the displayed 2D/3D mixed image.
Our proposed locally controllable 2D/3D mixed display and corresponding image generation method can be further improved. Firstly, a Fresnel lens array with small focal length and a LED array can be used to make the system more compact and increase the brightness of the system. Secondly, the information in the parallax image is underutilized in the generation of high-resolution 2D residual image. Therefore, a multi-frame image super-resolution reconstruction algorithm would be a good way to generate 2D residual image with more details and higher resolution.
In addition, the polarization-dependent LCMLA quality is very important for high-quality 3D display. First of all, in the experiment, the rubbing method is used to align the liquid crystals, which is the common method at present. Although the rubbing method is easy to realize, a large number of dust particles are generated in the rubbing process, which will damage the alignment film and decrease the uniformity of rubbing. In recent years, non-rubbing methods, such as photoalignment technology [23] and oblique evaporation technology [24], have made great progress in eliminating pollution caused by static electricity and dust. Secondly, a spherical lens is an imperfect imaging element with aberration problems. In order to solve the aberration problem, the aspheric polarization-dependent LCMLA is a good candidate [25].
5. Conclusions
We have proposed a locally controllable 2D/3D mixed display and corresponding image generation method. The incident light is controlled by the CBM and the LCM to form a point light source array for 3D display or scattered light source for 2D display in arbitrary regions. Moreover, a corresponding image generation method is proposed to locally switch the 2D and 3D contents according to observer’s demand. In the experiment, a prototype of the proposed locally controllable 2D/3D mixed display was built up, which has no need of fast refresh rate nor high driving voltage, and has relatively compact system volume and high optical efficiency. And two 2D/3D mixed light field images with different target objects and residual objects from the same parallax images were generated. The experimental results showed that the locally controllable 2D/3D mixed image can be reconstructed with high-definition 2D texture image and natural 3D image in the proposed system, which brings a good visual experience to the viewer. In the future, we believe that the proposed locally controllable 2D/3D mixed display and corresponding image generation method will contribute to the commercialization of 2D/3D mixed display, particularly in the field of medical imaging.
Funding
National Natural Science Foundation of China (U21B2034); Sichuan Science and Technology Program (2022YFG0326).
Acknowledgments
We would like to thank Dr. Bo-Ya Jin for his generous help in the fabrication of the polarization-dependent liquid crystal micro-lens array.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. https://boe.com/Enterprise/TVDisplay.
2. G. Lippmann, “Epreuves reversibles donnant la sensation du relief,” J. Phys. Theor. Appl. 7(1), 821–825 (1908). [CrossRef]
3. M. Martínez-Corral and B. Javidi, “Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems,” Adv. Opt. Photonics 10(3), 512–566 (2018). [CrossRef]
4. L. C. Cao, Z. Wang, H. Zhang, G. F. Jin, and C. Gu, “Volume holographic printing using unconventional angular multiplexing for three-dimensional display,” Appl. Opt. 55(22), 6046–6051 (2016). [CrossRef]
5. Z. H. He, X. M. Sui, G. F. Jin, D. P. Chu, and L. C. Cao, “Optimal quantization for amplitude and phase in computer-generated holography,” Opt. Express 29(1), 119–133 (2021). [CrossRef]
6. K. Rathinavel, H. P. Wang, A. Blate, and H. Fuchs, “An extended depth-at-field volumetric near-eye augmented reality display,” IEEE Trans. Visual. Comput. Graphics 24(11), 2857–2866 (2018). [CrossRef]
7. K. Suzuki, Y. Fukano, and H. Oku, “1000-volume/s high-speed volumetric display for high-speed HMD,” Opt. Express 28(20), 29455–29468 (2020). [CrossRef]
8. J. H. Park, H. R. Kim, Y. Kim, J. Kim, J. Hong, S. D. Lee, and B. Lee, “Depth-enhanced three-dimensional–two-dimensional convertible display based on modified integral imaging,” Opt. Lett. 29(23), 2734–2736 (2004). [CrossRef]
9. J. H. Park, J. Kim, Y. Kim, and B. Lee, “Resolution-enhanced three-dimension / two-dimension convertible display based on integral imaging,” Opt. Express 13(6), 1875–1884 (2005). [CrossRef]
10. J. Hong, Y. Kim, S. G. Park, J. H. Hong, S. W. Min, S. D. Lee, and B. Lee, “3D/2D convertible projection-type integral imaging using concave half mirror array,” Opt. Express 18(20), 20628–20637 (2010). [CrossRef]
11. K. Hong, J. Yeom, C. Jang, G. Li, J. Hong, and B. Lee, “Two-dimensional and three-dimensional transparent screens based on lens-array holographic optical elements,” Opt. Express 22(12), 14363–14374 (2014). [CrossRef]
12. S. W. Cho, J. H. Park, Y. Kim, H. Choi, J. Kim, and B. Lee, “Convertible two-dimensional-three-dimensional display using an LED array based on modified integral imaging,” Opt. Lett. 31(19), 2852–2854 (2006). [CrossRef]
13. Z. Wang, A. T. Wang, S. L. Wang, Y. M. Xing, Z. B. Deng, X. H. Ma, and H. Ming, “High optical efficiency lensless 2D–3D convertible integral imaging display using an edge-lit light guide plate,” J. Disp. Technol. 12(12), 1565–1569 (2016). [CrossRef]
14. H. Deng, Q. Li, W. He, X. W. Li, H. Ren, and C. Chen, “2D/3D mixed frontal projection system based on integral imaging,” Opt. Express 28(18), 26385–26394 (2020). [CrossRef]
15. H. Choi, S. W. Cho, J. Kim, and B. Lee, “A thin 3D-2D convertible integral imaging system using a pinhole array on a polarizer,” Opt. Express 14(12), 5183–5190 (2006). [CrossRef]
16. H. Deng, Z. L. Xiong, Y. Xing, H. L. Zhang, and Q. H. Wang, “A high optical efficiency 3D/2D convertible integral imaging display,” J. Soc. Inf. Disp. 24(2), 85–89 (2016). [CrossRef]
17. P. Y. Chou, J. Y. Wu, S. H. Huang, C. P. Wang, Z. Qin, C. T. Huang, P. Y. Hsieh, H. H. Lee, T. H. Lin, and Y. P. Huang, “Hybrid light field head-mounted display using time-multiplexed liquid crystal lens array for resolution enhancement,” Opt. Express 27(2), 1164–1177 (2019). [CrossRef]
18. H. Watanabe, T. Omura, N. Okaichi, H. Sasaki, J. Arai, M. Kawakita, and B. Javidi, “Integral 3D/2D partially convertible display using geometric phase lens array,” Results in Optics 3, 100061 (2021). [CrossRef]
19. H. H. Lee, P. J. Huang, J. Y. Wu, P. Y. Hsieh, and Y. P. Huang, “A 2D/3D hybrid integral imaging display by using fast switchable hexagonal liquid crystal lens array,” Proc. SPIE 10219, 1021910 (2017). [CrossRef]
20. N. Xu, B. Price, S. Cohen, and T. Huang, “Deep Image Matting,” The IEEE Conference on Computer Vision and Pattern Recognition311–320 (2017).
21. Z. L. Xiong, Y. Xing, H. Deng, and Q. H. Wang, “Planar parallax based camera array calibration method for integral imaging three-dimensional information acquirement,” SID Symposium Digest of Technical Papers47(1), 219–222 (2016).
22. C. Dong, C. C. Loy, and X. O. Tang, “Accelerating the super-resolution convolutional neural network,” in European Conference on Computer Vision391–407 (2016).
23. V. Chigrinov, A. Kudreyko, and Q. Guo, “Patterned potoalignment in thin films: physics and applications,” Crystals 11(2), 84 (2021). [CrossRef]
24. W. R. Liou, C. Y. Chen, J. J. Ho, C. K. Hsu, C. C. Chang, R. Y. Hsiao, and S. H. Chang, “An improved alignment layer grown by oblique evaporation for liquid crystal devices,” Displays 27(2), 69–72 (2006). [CrossRef]
25. T. H. Lee, K. I. Joo, and H. R. Kim, “Switchable lens design for multi-view 2D/3D switching display with wide-viewing window,” Crystals 10(5), 418 (2020). [CrossRef]
