Open Access
Add to BibSonomy
Abstract
Among various specifications of near eye display (NED) devices, a compact formfactor is essential for comfortable user experience but also the hardest one to accomplish due to the slowest progresses. A pinhole/pinlight array based light-field (LF) technique is considered as one of the candidates to achieve that goal without thicker and heavier refractive optics. Despite those promising advantages, however, there are critical issues, such as dark spots and contrast distortion, which degrade the image quality because of the vulnerability of the LF retinal image when the observer’s eye pupil size changes. Regardless of previous attempts to overcome those artifacts, it was impossible to resolve both issues due to their trade-off relation. In this paper, in order to resolve them simultaneously, we propose a concept of multiplexed retinal projections to integrate the LF retinal image through rotating transitions of refined and modulated elemental images for robust compensation of eye pupil variance with improved conservation of contrast distribution. Experimental demonstrations and quantitative analysis are also provided to verify the principle.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
In recent years, as the interest and demands for a near eye display (NED) has been increasing, there were significant improvements in various limitations of early NEDs, such as expanding the eye-box [1,2], providing multi-focal planes [3–6], and inducing natural three-dimensional presence based on the light-field (LF) and the super multi-view principles [7–9]. However, regardless of the significant technological improvements above, the NED devices should be thinner and lighter to be practical further and attract more users. Recently, the pinhole/pinlight array based LF NED scheme has been proposed to achieve that goal without using thick and heavy refractive optical components [10–14].
Figure 1 shows the basic principle of the conventional LF NED using a pinhole array. Each single elemental image on a display panel is directly projected onto the observer’s retina through the corresponding pinhole and the eye pupil. It is necessary that Nth single elemental image should be projected to the retina only through the corresponding Nth pinhole and the eye pupil with a maximum size as shown in Fig. 1. This condition of proper projection is essential to prevent a pseudo projection that a neighboring (e.g., (N-1)th or (N + 1)th) elemental image is projected through the Nth pinhole to cause a duplication of retinal image at improper position. If the condition of proper projection is satisfied, an integrated retinal image can be formed as a combination of multiple retinal projections with partial overlapping areas between them [11].
Although above scheme can acquire a slim LF NED by removing the refractive optical components from its structure, still there are several issues. The most significant one is low luminance due to the low optical efficiency of the pinhole array although this issue can be compensated by adopting a brighter display device. In addition, there is a tradeoff between the formfactor of it and the resolution of the retinal image. From the basic principles of LF NED shown in Fig. 1, as the display panel locates closer to the pinhole array to reduce the thickness, the size of partial overlapping is increased. In other words, as the LF NED becomes thinner, the resolution of the integrated retinal image is degraded because more pixels are used to project duplicated images to the retinal surface. Since this issue is based on the principles to realize the LF NED, the only solution for it is improving the resolution of the display panel. Regarding that approach, recent developments of display devices with finer pixel pitch are promising progresses for LF NED to resolve this issue. Besides, since the field of view (FoV) of the LF NED can be enhanced if the display panel can be closer to the eye, the thinner formfactor using a higher resolution display panel with same size can also improve the FoV.
However, even though we can improve the picture quality by adopting a high-resolution panel and finding an optimal design considering that tradeoff, there remain more issues related to the picture quality of the LF NED. First, the brightness of the projected image is inconsistent due to the different number of overlapping, which results in the distorted contrast distribution across the retinal image as shown in Fig. 1. To resolve this issue, two different approaches have been proposed. One of them is to adjust the size of each elemental image so that they can be projected as like a tiled arrangement without the overlapping areas among them [12]. However, this solution only works properly when the eye pupil diameter of the observer is fixed to a pre-defined size to satisfy the non-overlapping condition. Thus, it makes the tiling method be extremely vulnerable to the change of eye pupil size of observer, which is impractical. The other one is to compensate the luminance profile of the partial overlapping area to pursuit a nonlinear curve by a square of cosine function [13]. Using this scheme, it is possible to prevent the overlapping area from being brighter than the non-overlapping region when only two retinal projections overlap to each other along a horizontal or vertical axis. Nevertheless, this method also has a restriction to resolve the issue of contrast distortion because more than three retinal projections along two-dimensional (e.g., diagonal) axes can also overlap in real NEDs.
Second, the conventional LF NED generates the dark spot [14]. As the diameter of real eye pupil varies with the luminance of displayed images on the NED, that response affects the shape and presence of partial overlapping areas by changing the portion of elemental images projected. If the real eye pupil shrinks, only part of the designated Nth elemental image can be projected to the retina and the partial overlapping area will be diminished as a result. Thus, if the diameter of real eye pupil is below a critical size to maintain the partial overlapping area between the retinal projection of a single elemental images as shown in Fig. 2, there will be a dark spot to cause disintegrated and defective retinal images since no elemental image is projected to there [14]. Above mentioned two approaches could reduce the contrast distortion issue but cannot resolve the dark spot issue. Therefore, the degradation of retinal image quality in the current LF NED is inevitable.
Fig. 2. Dark spot issue of conventional LF NED method due to diminished retinal projection through the critically small pupil size.
In this paper, we propose a novel method that resolve both the contrast distortion and the dark spot issues in the LF NED. It is based on a multiplexed retinal projection for robust compensation of eye pupil variance with improved conservation of contrast distribution. Comparisons of experimental demonstrations with various conditions (i.e., different pupil sizes) are also provided to verify the proposed principle.
2. Principles
Evaluating the above issues of degrading the quality of retinal image, we conclude the issue of dark spots is more critical than the other. Even after compensating the contrast distortion issue shown in Fig. 1 with the methods suggested by Maimone et al. [12] and Yao et al. [13], any assessment on the quality of the disintegrated retinal image would be meaningless if there are losses of original image information due to the areas with no retinal projection (dark spots) shown in Fig. 2. Thus, it is important to develop a technique to prevent the appearance of dark spots prior to the contrast distortion. After eliminating the dark spot issue, we will also apply new adaptive compensation for any contrast distortion caused by the dark spots eliminated images. By combining those two resolutions, the goal of realizing an NED based on multiplexed retinal projections to enhance the quality of retinal image with robust compensation of eye pupil variance and improved contrast conservation can be achieved.
The principle of the first solution to eliminate the dark spots is described in Fig. 3. The basic principle is to project additional images onto the locations of dark spots using a time-multiplexing scheme to fill out the missing information. From the basic principle of pinhole-based LF NED described in Fig. 1, each Nth elemental images should be projected onto the retina only through the corresponding Nth pinhole. In other words, each elemental image has a pairing pinhole of its own. Thus, those Nth elemental image and Nth pinhole can be considered as a proper Nth elemental image-pinhole pair (EIPP) to make an appropriate retinal projection. From that principle, it can be understood that what we need is new additional proper EIPP to project the additional elemental images to the locations of dark spot through the additional pairing pinhole. However, according to the principle described in Fig. 1 and 2, the conventional elemental images already occupy the expected location of additional elemental image to cover various cases of eye pupil diameters. For example, though only part of the conventional image is projected to the retina in case of a shrunken eye pupil shown in Fig. 2, the size of the elemental image should not be diminished to prepare a case with a maximum eye pupil too as shown in Fig. 1. In other words, we are not able to fill the dark spot just by adding new EIPP to the same frame timing to cover both cases of maximum and shrunken eye pupils shown in Fig. 1 and 2, respectively. As a new approach, we propose a multiplexed retinal projection adding a new 2nd sub-frame with additional EIPP and switch it with conventional EIPP in different 1st sub-frame with a temporal multiplexing scheme as shown in Fig. 3(a). Then, by inducing an afterimage effect to visually combine those two sub-frames as shown in Fig. 3(b), it is possible to eliminate the dark spot depicted in Fig. 2.
Fig. 3. Principle of the proposed multiplexed retinal projection: (a) consolidating of multiple sub-frames by temporal multiplexing, and (b) expected retinal image on the retina combined by an afterimage effect.
Besides, since the basic principle in Fig. 3 is how to resolve the dark spots at diagonal direction only, it is also necessary to consider resolving the dark spots in horizontal and vertical distributions to achieve a full compensation. For that purpose, we adopt a scheme of two-dimensional multiplexed retinal projection composed of four sub-frames rotating transition composed of the horizontal and the vertical round-trips as shown in Fig. 4. By repeating the procedures above, we can implement an LF NED based on a multiplexed retinal projection providing robust compensation of eye pupil variance to resolve the issue of retinal image degradation caused by dark spots.
As a second step, we deal with the issue of contrast distortion due to the brighter partial overlapping areas after the elimination of the dark spots (brightness inconsistency in Fig. 4). In the prior approach of Maimone et al., they prevented the partial overlapping areas from appearing since they are causing the contrast distortion due to their brighter luminance [12]. They generated the elemental images on the display panel so that the retinal projections of them have a tiled structure without any overlapping area. Thus, with no overlapping area, no brighter area exists ideally. However, the lack of overlapping area makes the tiled retinal image extremely be vulnerable to the variance of eye pupil and the appearance of dark spot easily occur. The other approach of Yao et al. is adopting a gradual luminance profile fitted to a square of cosine function [13]. However, this one also has a restriction that there should be only one-dimensional overlapping along horizontal/vertical axes between neighboring retinal projections. For that purpose, it is necessary refining the elemental images to have a rhombus shape. Nevertheless, those modulation increases a chance of dark spot appearance in diagonal direction because of the removal of overlapping area.
From the review of previous schemes above, it is obvious that both of them are under an unavoidable paradox – whether removing or preserving the overlapping area, dark spots or brighter overlapping areas will appear. Therefore, it is necessary to break the paradox to resolve both issues at once. For that purpose, we attempt to achieve that goal by adopting the scheme of multiplexed retinal projection through multiple sub-frames and revise it to deal the contrast compensation also.
The 1st step is to generate rhombus-shaped elemental image array to present no overlapping area on retina within a single sub-frame and to modulate the luminance of them with square of cosine function as shown in Fig. 5(a). Then, it is expected that the brightness of one-dimensional overlapping area along horizontal or vertical axis on the retina can be uniform. However, the above step only is still under a restriction that a dark spot would appear in the retina without the overlapping area. Thus, the next step is to prevent the dark spots by adopting the multiplexed retinal projections as shown in Fig. 5(b). With the rotating transitions of rhombus-shaped elemental images by the proposed scheme, there only exist one-dimensional overlapping areas in the integrated retinal image as shown in Fig. 5(c). Thus, both issues of retinal image degradation can be resolved with robust compensation of eye pupil variance with improved conservation of contrast distribution.
Fig. 5. Proposed scheme to improve the conservation of contrast distribution. (a) 1st step to refine and modulate the elemental images, (b) 2nd step: the revised multiplexed retinal projection with only one-dimensional overlapping areas for preventing the dark spots, and (c) resolving the contrast distortion of one-dimensional partial overlapping area.
3. Experimental setup and results
The experimental setup for verifying the proposed method is shown in Fig. 6. It consists of a 5.5-in. (1 in. = 2.54 cm) liquid crystal display (LCD) panel with a pixel size of 31.5 µm as a display device and another LCD panel with a pixel pitch of 250 µm as a pinhole panel. Since the pixels on the LCD panel can control the passage of light, it is possible to implement a movable pinhole array on the LCD panel for the proposed multiplexed retinal projection. The distance between the display device and the pinhole panel was set to 45 mm, and the gap between the pinholes was 4 mm as depicted in Fig. 6. Besides, the eye relief of our demonstration is set to be 45 mm due to the thickness of the commercial camera lens used for experimental setup. In a real implementation with an eye of an observer, we expect that it is enough to have an eye relief around 20 mm. In addition, from the specifications of the experimental setup above, the maximum size of the eyebox is expected to be 3.82 mm. Since the size of the eyebox is proportional to the eye pupil diameter, it will be narrower than the maximum case in most situations. The experimental reference images of a mandrill, astronaut, and USAF resolution chart and examples of generated elemental image arrays for each sub-frame from an astronaut image among them are shown in Fig. 7. The expected maximum eye pupil size in the LF NED design is set to be 8 mm, which is expected to be a realistic condition for most of observers under scotopic condition [15]. The other reason to set the maximum eye pupil size to have an over-sized condition is that it is necessary to prevent the flipped view issue that a part of elemental image is projected to the retinal surface through improper pinhole [14]. In other words, we focused to improve the picture quality by resolving other issues such as flipped view as well as compensating the contrast distortion and the dark spots.
Fig. 7. An example of experimental reference image and generated elemental image arrays for each sub-frame.
In the experiment, the effect of varying eye pupil is captured by controlling the lens aperture of digital camera. We have used three different aperture diameters of 7.9 mm, 5.5 mm, and 3.8 mm to represent realistic pupil size changes in scotopic, mesopic, and photopic conditions, respectively [15]. As shown in Fig. 8, the experimental results of conventional method with demonstrates the basic principles and issues with observer’s pupil size changes in conventional methods [12,13]. At first, the condition of 7.9 mm aperture (upper row in Fig. 8) is free from dark spot but have the most severe contrast distortion due to wide partial overlapping areas. In contrast, the retinal image with the 3.8 mm aperture (lower row in Fig. 8) has the most significant appearance of dark spots although the contrast distortion due to partial overlapping areas is not observed. As a most common case, the condition of 5.5 mm aperture (middle row in Fig. 8) presents both issues of dark spots and contrast distortion. Therefore, it is obvious that the conventional method has no optimum point of resolving the dark spot and contrast distortion simultaneously.
To quantitatively illustrate the image difference between the reference and the perceived retinal images, we also performed normalized cross-correlation (NCC) evaluation between images, which represents similarity of patterns between two images [16,17]. Again, overall very low NCC and variation with different pupil size represent the inferior image quality issue in the conventional LF NED.
Figure 9 shows the experimental results with the proposed method. In comparison of the results with the conventional method (Fig. 8), the proposed method (Fig. 9) demonstrated superior and more uniform image quality without dark spot and severe contrast distortion regardless of changes of aperture diameters (i.e., pupil changes). The NCC values in the proposed method are all higher than the conventional methods in all aperture conditions and images, and the proposed method provides robust image qualities regardless of the aperture size changes. Therefore, the proposed method can provide better retinal image quality with robust compensation against the eye pupil variance than the conventional LF NED.
In addition, experimental luminance profiles with 3.8 mm aperture along the diagonal axis (red line) from a flat grey reference image are shown in Fig. 10. In the comparison, the luminance profile with a unit of captured greyscale of the conventional NED has periodical drops fallen to near black due to the dark spots. In contrast, the proposed method with same unit is providing significantly stable profile with less fluctuations. The proposed method provides much better luminance uniformity across the diagonal FoV (12.89 of standard deviations (σ) in the conventional and 3.33 in the proposed methods).
Therefore, throughout the experimental results and comparisons of them in Figs. 8,9, and 10, it can be confirmed that the proposed scheme with multiplexed retinal projections can realize a robust compensation of eye pupil variance with improved conservation of contrast conservation.
4. Discussion
Both dark spot and contrast distortion issues directly affect the luminance uniformity of LF NED system. This is even more important when we consider the pupil size changed by the luminance of the LF NED system. Although we demonstrated different pupil size independently from the display luminance, note that the pupil size is always dependent on the luminance of the display system itself [15]. It is further emphasized in such multiple aperture optical systems such as LF NED due to the pupil sharing effect and overlapping and/or dark spots [18]. Since brighter display induces smaller pupil size, in conventional LF NED, the dark spot issue (3.8 mm case in Fig. 8) may be more noticeable than the contrast distortions (7.9 mm case in Fig. 8). The significantly lower NCC verified much degraded image qualities in the perception of conventional method. However, in the proposed method, regardless of the brightness of the display (3.8 mm case in Fig. 9 for the bright display and 7.9 mm case in Fig. 9 for the dim display), both dark spot and contrast distortions are not noticeable. The benefits of the proposed method will be more appreciated when we consider the human visual perception of such images, which is highly rely on the contrast not the brightness [19]. It is the reason why we used NCC (comparison based on pattern, feature, and saliency) instead of PSNR (localized difference) in the qualitative comparison [20]. Significantly improved NCC across images and pupil sizes verifies the improvement and robustness of the proposed method compared with the conventional methods. Therefore, regardless of the luminance of the LF NED system and thus the pupil size, the perceived image through the proposed method will be qualitatively (i.e., luminance uniformity) and quantitatively (i.e., contrast consistency) better than any conventional methods.
The LF NED has been developed to aim for the stereoscopic headsets [10–14]. However, in the stereoscopic LF NED systems, the conventional methods may result in highly different patterns of the dark spots and contrast distortions between images on both eyes. Since the binocular visual perception is highly driven by the contrast difference between retinal images on both eyes, the conventional methods may result in the binocular rivalry while the proposed method could be perceived even better under the binocular summation process [21]. Therefore, the proposed method could be more beneficial in more practical stereoscopic LF NED systems.
In addition, though the issues of contrast distortion and dark spot can be resolved by the proposed scheme, the change of eye location in real usage can cause another issue such as loss of retinal projections through some pinholes. Therefore, adopting an active compensation scheme using an eye tracking device to modify the rendered elemental images can be a good candidate to secure more robustness of eye pupil variance and movements.
5. Conclusion
The formfactor of commercial NED is one of the most decisive specifications and also the hardest one to improve. Though the conventional LF NED technique effectively realizes an integrated retinal images through a pinhole array without the refractive optical components, its vulnerability to the variance of eye pupil of observer causes critical issues such as dark spots and contrast distortion. Herein, we propose a multiplexed retinal projection for robust compensation of eye pupil variance with improved conservation of contrast distribution. Regard to the principles and experimental demonstrations with quantitative analysis, it is expected that the proposed scheme can achieve a challenging goal of enhancing the quality of retinal image with an improved formfactor without refractive optics.
Funding
Institute of Information & Communications Technology Planning & Evaluation (2020-0-00924); National Research Foundation of Korea (2021R1A2C1011803).
Acknowledgments
This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-00924, Technology development of authoring tool for 3D holographic printing contents, 50%), and National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2021R1A2C1011803, Research on gaze-contingent hybrid volumetric display, 50%).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. C. Jang, K. Bang, G. Li, et al., “Holographic near-eye display with expanded eye-box,” ACM Trans. Graph. 37(6), 1–14 (2018). [CrossRef]
2. J. Jeong, J. Lee, C. Yoo, et al., “Holographically customized optical combiner for eye-box extended near-eye display,” Opt. Express 27(26), 38006–38018 (2019). [CrossRef]
3. S. Lee, Y. Jo, D. Yoo, et al., “Tomographic near-eye displays,” Nat. Commun. 10(1), 2497 (2019). [CrossRef]
4. J.-H. R. Chang, B. V. K. V. Kumar, and A. C. Sankaranarayanan, “Towards multifocal displays with dense focal stacks,” ACM Trans. Graph. 37(6), 1–13 (2018). [CrossRef]
5. T. Zhan, Y.-H. Lee, and S.-T. Wu, “High-resolution additive light field near-eye display by switchable Pancharatnam–Berry phase lenses,” Opt. Express 26(4), 4863–4872 (2018). [CrossRef]
6. X. Wang and H. Hua, “Design of a digitally switchable multifocal microlens array for integral imaging systems,” Opt. Express 29(21), 33771–33784 (2021). [CrossRef]
7. C. Gao, Y. Peng, R. Wang, et al., “Foveated light-field display and real-time rendering for virtual reality,” Appl. Opt. 60(28), 8634–8643 (2021). [CrossRef]
8. W. Han, J. Han, Y.-G. Ju, et al., “Super multi-view near-eye display with a lightguide combiner,” Opt. Express 30(26), 46383–46403 (2022). [CrossRef]
9. D. Teng, C. Lai, Q. Song, et al., “Super multi-view near-eye virtual reality with directional backlights from wave-guides,” Opt. Express 31(2), 1721–1736 (2023). [CrossRef]
10. W. Song, Q. Cheng, P. Surman, et al., “Design of a light-field near-eye display using random pinholes,” Opt. Express 27(17), 23763–23774 (2019). [CrossRef]
11. K. Akşit, J. Kautz, and D. Luebke, “Slim near-eye display using pinhole aperture arrays,” Appl. Opt. 54(11), 3422–3427 (2015). [CrossRef]
12. A. Maimone, D. Lanman, K. Rathinavel, et al., “Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources,” ACM Trans. Graph. 33(4), 1–11 (2014). [CrossRef]
13. C. Yao, D. Cheng, and Y. Wang, “Uniform luminance light field near eye display using pinhole arrays and gradual virtual aperture,” in International Conference on Virtual Reality and Visualization (ICVRV) (2016), pp. 401–406.
14. H. Lee, U. Yang, and H.-J. Choi, “Analysis of the design parameters for a lightfield near-eye display based on a pinhole array,” Curr. Opt. Photonics 4(2), 121–126 (2020). [CrossRef]
15. A. B. Watson and J. I. Yellott, “A unified formula for light-adapted pupil size,” J. Vis. 12(10), 12 (2012). [CrossRef]
16. J. P. Lewis, “Fast Normalized Cross-Correlation,” in Vision Interface (VI) (1995), pp. 120–123.
17. J.-H. Jung, K. Hong, G. Park, et al., “Reconstruction of three-dimensional occluded object using optical flow and triangular mesh reconstruction in integral imaging,” Opt. Express 18(25), 26373–26387 (2010). [CrossRef]
18. J.-H. Jung, N. M. Kurukuti, and E. Peli, “Photographic Depiction of the Field of View with Spectacles-mounted Low Vision Aids,” Optom. Vis. Sci. 98(10), 1210–1226 (2021). [CrossRef]
19. E. Peli, J. Yang, R. Goldstein, et al., “Effect of luminance on suprathreshold contrast perception,” J. Opt. Soc. Am. A 8(8), 1352–1359 (1991). [CrossRef]
20. T. F. Karim, M. S. H. Lipu, M. L. Rahman, et al., “Face recognition using PCA-based method,” in IEEE International Conference on Advanced Management Science (ICAMS) (2010), pp. 158–162.
21. L. Liu, C. W. Tyler, and C. M. Schor, “Failure of rivalry at low contrast: Evidence of a suprathreshold binocular summation process,” Vision Res. 32(8), 1471–1479 (1992). [CrossRef]
