Open Access
Add to BibSonomy
Abstract
This paper shows how a phase only spatial light modulator (SLM) can be used to implement compact, wide field of view (FOV) projection systems, using a foveal imaging approach. The SLM is tested in two different configurations. First, as a wavefront corrector to locally compensate aberrations in wide angle projection, when illuminated by a white (RGB) light source. Second, as re-configurable diffractive optical element (DOE) to display phase computer generated holograms (CGHs) of wide angle images, that include aberration corrections for peripheral regions of interest (ROI) of the output image, when illuminated by a coherent source. Finally, the foveal projection system is coupled with a commercial combiner to project 11° ×4° virtual images at 2~m and assess the foveal correction method on different ROIs in the FOV.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Combining wide field of view (FOV), high resolution and compactness has always been challenging. Such wide angle systems require strong deviations of light rays to form a wide angle image, these deviations induce wavefront aberrations [1]. Moreover, the systems tend to become bulky as the FOV increases. This is particularly true with automotive Head Up Displays (HUDs) [2]. Car manufacturers are taking a growing interest to increasing the HUD FOV for a better immersive AR driving experience. But this FOV increase entails a strong growth of the HUD volume, particularly in the critical space underneath the dashboard. In 2010, the typical FOV for an automotive HUD was 3$^{\circ }\times$1.5$^{\circ }$ for a given eye-box at a short projection distance (< 2m) for a total volume of around 2L. In 2020, a typical automobile HUD exhibits a virtual image size of 7$^{\circ }\times$3$^{\circ }$ at a projection distance between 2 and 4 m [3], and an overall volume of 7L. However, the trend towards Augmented Reality (AR) is becoming more and more pressing with a requirement for multiple projection distances (e.g. 2 planes system with a near field at 2m and an AR far field > 10m), with large virtual images for the AR channel (> 10$^{\circ }\times$5$^{\circ }$) and a large eye-box for driving comfort. These characteristics imply a massive increase in volume, significantly exceeding 18L [4], which is extremely difficult to address, as automotive HUD systems have to share space with other critical car elements (e.g. instrumentation cluster, steering wheel column, crossbeam, ventilation ducts etc.). The most HUD constrained volume is the imaging system: picture generated unit (PGU) and freeform mirror size and distance between them. A HUD system is typically designed for resolutions higher than 60 ppd (pixel per degree), 85 ppd is a very typical resolution target for HUD designs and most thin film transistor (TFT) displays have 85${\mathrm {\mu }}$m pixel size. TFT size is given by : $T=FOV\times Reso \times PS$, where $T$ is the TFT size, $FOV$ is the field of view of the HUD, $Reso$ is the virtual image resolution in pixels per degree, and $PS$ is the imager pixel size (TFT for most HUDs). $PS$ is relative to the TFT which does not exceed 85 ${\mathrm {\mu }}$m. As $FOV$ and $Reso$ are set by the HUD specifications, and the TFT pixel size is around 85 ${\mathrm {\mu }}$m, the required size of the TFT depends only on the FOV. The greater the FOV, the greater size the TFT. Typically, for a 7$\times$3$^{\circ }$ virtual image at $2~m$, the TFT has to be 50$\times$21mm$^{2}$. and for a 11$\times$4$^{\circ }$ HUD at $2~m$, to be 79$\times$28mm. Besides, for increasing FOVs, it becomes more and more difficult to create the HUD image with only one freeform mirror. Therefore, multiple freeform mirrors are needed which explodes the volume of the HUD and complicate tolerancing.
There is thus a strong need to compact the system to integrate it in the vehicle, while maintaining a given size of aperture (eye-box in HUD) and increasing FOV and projection distance (i.e. higher focal length for the optical system). An obvious solution is to increase the magnification of the optical path. The drawback is an increase in aberrations [1] [5]. In this paper we propose a foveated projection approach capable of concentrating high resolution in delimited zones (or ROI, Regions of Interest), particularly suited to wide FOV automotive demands, where only specific driving symbols covering a maximum surface of 30% of the windshield can be displayed at any time because of the need to maintain road visibility. It is important to notice that it removes the need for a driver eye-tracking system to implement our foveal approach as symbols are only ever displayed over small areas of the FOV at any one time and foveal correction can thus be confined to these zones. In addition, the projection system can be coupled with only one freeform mirror to generate a wide FOV image in which aberrations are corrected in a foveal manner.
Foveated systems act like human eyes, providing high resolution in the gaze direction while leaving the rest of the image field blurred. The concept first appeared as foveated image capture systems in [6], where the phase compensator was simulated as a transmissive phase SLM with a 60$^{\circ }\times$60$^{\circ }$ FOV, which was impossible to reproduce in practice because of the unavailability of the transmissive phase SLMs with a sufficient resolution. Later Curatu et al. [7] [8] proposed a 60$^{\circ }\times$60$^{\circ }$ foveated capture system with a reflective phase SLM, placed at the pupil stop to dynamically correct the aberrations. Our idea is to reverse the setup from an image capture to an image projection system, locally correcting wide FOV images thanks to a phase SLM, placed at the pupil stop. A few similar systems have been developed for near to eye displays (NED) such as Head Mounted Displays (HMDs) [9] [10], but to our knowledge never used in the very demanding case of automotive HUDs. We assess here the principle of such foveated HUD systems for different light sources. In the first, the SLM is used as a real-time aberration wavefront corrector in a conventional HUD setup based on the projection of color images displayed on a DMD. In the second approach, the SLM operates as a holographic display, in a wide FOV holographic display system as described in [11] where an aberration and distortion free image with a 90$^{\circ }\times$45$^{\circ }$ FOV was realized and in local aberration correction demonstrated. A low cost/volume wide angle lens is deliberately used in each case to generate wavefront aberrations and correct them in a foveal manner. In [11] both Zemax computed and adaptive-optical correction parameters were tested. The adaptive-optical method tended to perform better in the outer edges. In this paper we design and study systems both for the incoherent and coherent approaches in an automotive application. In a second part, we give a preliminary demonstration of how our foveal projection systems can be incorporated in a HUD configuration by using an existing automotive HUD combiner to create virtual images. Finally, we discuss the trade-off between volume compactness and optical aperture opening which is specific to such devices.
2. Foveated projection set-up using a wavefront corrector
Martinez et al. [6] proposed a foveal optical imaging system with a wide FOV that corrects wavefront aberrations of images using a pure phase SLM. This system is compact compared to generally large adaptive optical systems. The SLM simulated in the literature was a low-cost transmissive device providing wavefront correction up to kHz speed. Numerous studies were subsequently carried out [7][12] to improve the system and to take into account phase SLMs which are reflective and then adapt it to different applications such as HMD [13] or foveal cameras [14] [15]. An example of a foveal detector system that was simulated and implemented experimentally is given in Fig. 1. The FOV of the system is 60$^{\circ }$. We reverse this set up and use it as a projector of an incoherent light source image (TFT or digital light processing (DLP)). The idea is to display a wide angle image using a pair of plano-convex and plano-concave lenses, this combination offers a FOV with low wavefront deformation. The SLM is placed at the pupil stop and modifies the wavefront to locally compensate the aberrations in a given ROI. The process of the correction is to first display the wide angle image without correction : switching off the SLM lets its surface act like a mirror. Once the uncorrected image is displayed, the local phase in the ROI which is to be considered spatially invariant is measured and displayed by the SLM. The ROI is then corrected leaving the rest of the image deteriorated. This is the operating principle of human eye: which has an approximately 180$^{\circ }$ horizontal FOV but only the gazing region (foveal region 3$^{\circ }$-5$^{\circ }$) is well resolved. This approach is advantageous in the case of a compact wide angle display for automotive applications, for instance, the HUD should insure a wide FOV, high resolution and low packaging volume. For an automotive application, there is limited number of images to display, and they only need to be displayed on a small area of the windshield at a given time, but each image with a certain size needs to be well resolved and geometrically corrected. Foveal projection is interesting in this case because each image displayed on the windshield can be considered as a ROI whose wavefront can be corrected by the SLM. It is known from the literature that the size of the correctable ROI, is linked to its location in the the FOV. The further the ROI from the optical axis, the narrower it should be to perform an optimal correction. We present here a compact 30$^{\circ }$ FOV foveal display system with a packaging volume of 1 liter. We then show how this foveal projection system can be incorporated into the final automotive HUD system.
Fig. 1. Foveated imaging set up using a reflective phase SLM as wavefront corrector simulated in [6]
2.1 Simulation analysis of the optical foveated projection system
The set up presented in Fig. 1 was simulated in the case of a foveal projector system in Fig. 2. In this set up, two plano convex and concave lenses were used to create a 30$^{\circ }$ FOV image over a distance of 2.4 meters. The aim was to display wide angle images using a compact optical system that locally corrects the wavefront leaving the rest of the image deteriorated. Two types of aberrations can be treated separately : wavefront and geometrical aberrations. The wavefront aberrations are typically tilt x and y, astigmatism 0 and 45$^{\circ }$, defocus, spherical, coma and trefoil aberrations. These aberrations can be quantified by a set of zernike coefficients. Without correction, the output image is well resolved in the central region, and increasingly aberrated with off axis angle. The image is also geometrically distorted due to the fact that the image is formed on a 3D sphere and projected on a 2D plane, therefore, the more the FOV increases, the more blurred, and the more distorted the whole image becomes. The blur at extreme fields is locally corrected by displaying a phase correction map on the SLM. The system was first simulated without correction using Zemax, where the SLM acts as a flat mirror. The output zernike data of the desired aberrated ROI were calculated and encoded in the SLM which was, in a second step, modeled as a standard phase sag in Zemax. When the SLM displays the phase map to correct the ROI located at the lower left corner, the output image is aberrated everywhere except at the ROI. Figure 2 shows the process of local correction and pre-compensation of a simulated grid. As the geometrical aberrations can be treated separately from the wavefront aberrations, a pre-compensation is calculated from the simulated non-corrected output image. This calculation is done in Python using OpenCV camera calibration functions designed for camera distortion correction [16]. The program uses a test pattern image displayed by the simulated system on Zemax without correction. The corners are detected and their coordinates are extracted and compared to the original image. The displacements of the corners are then calculated, and the radial and tangential distortion coefficients obtained. These coefficients are inversed and applied to the original target pattern image. The inversely distorted image is now displayed onto the imager and the new output simulated image is now distortion free and straight grid pattern is obtained. This routine can be applied either on the basis of simulation or experimental images that are processed to detect the corners. Now that the distortion is canceled, other wavefront aberrations can be locally corrected.
Fig. 2. Wavefront correction procedure for incoherent source. a) Display of a grid of lines on the DLP, an aberrated and distorted image is obtained at the output plane, b) pre-distorted image displayed on the DLP, correction phase CGH displayed on SLM, local aberrations at the left lower corner are corrected and global distortion is compensated
The correction is done by uploading the local zernike coefficients of the output image onto the SLM which is now modeled as a zernike standard phase surface in Zemax. The phase introduced by the SLM added to the phase responsible of the aberrations in the ROI cancel each other and the ROI is now corrected. The straight grid target and predistorted grid target images are given in 3, and final simulated output images are given in Fig. 2 where the distortion has been pre-compensated and the aberrations inside the ROI reduced.
2.2 Experiments
This section presents the results of the system simulations and experiments. The system FOV is 30$^{\circ }$ and the image is projected 2.4 meters in front of the observer. The display used in this experiment is a DLP technology pico projector Polaroid VPP800 with an output angle of 47$^{\circ }$ which projects images at distances from $7~cm$ to $3~m$. This projector was chosen for its high brightness display whose output can be relayed onto the SLM and magnified for a HUD application. The SLM used is a PLUTO-VIS056 Holoeye [17] device with a surface of $15.36\times 8.64~mm ^{2}$, a resolution of $1920\times 1080$ pixels, a 8 µm pixel pitch and a maximum phase delay of $2\pi$
2.2.1 Incoherent light source foveal projector optical bench
Figure 4 shows the optical bench built around the DLP based pico-projector whose projection lens was removed. The vacated position is considered as the entrance pupil of our projection system. This plane is imaged by a first lens $L_1$ onto the SLM, which is placed at the system pupil stop. The association of the first $L_1$ and second lens $L_2$ magnifies the image generated by the DLP pico-projector. The pre-compensated images calculated in the simulation were displayed by the pico-projector and the experimental distortion correction was observed. In order to verify the local corrections across different ROIs, the zernike coefficients calculated in the simulations were transformed into a grayscale phase map using software written in Python, and then displayed on the SLM to correct the ouput image in the chosen ROI. The yellow circle in Fig. 5 represents the ROI that was corrected for a wavelength of $533~nm$.
2.2.2 Results
We show in this section the local corrections performed experimentally. When a region is corrected, the rest of the image is slightly deteriorated and blurred. This is due to the fact that when the phase introduced by the SLM is applied to the whole image, it minimizes the aberrations in the ROI but amplifies them in the other regions. First experimental corrections were performed on a grid to better assess the improvement of the sharpness and contrast inside the ROI. The ROI is shown with a yellow circle and the difference between uncorrected and corrected ROI is shown in Fig. 5 where distortions are pre-compensated and other wavefront aberrations are corrected inside the ROI. For the HUD applications, it was important to observe the impact of the wavefront correction on specific driving symbols. We chose three symbols located at: -15$^{\circ }\times$-7$^{\circ }$, 0$^{\circ }\times$-7$^{\circ }$ and 0$^{\circ }\times$0$^{\circ }$. The weather data, the fuel gauge and a gps arrow. Wavefront correction is calculated for the displayed wavelength. Here we displayed in a green color so we calculated corrections at $\lambda =533~nm$. The correction in Fig. 6 shows that when the weather sign at the extreme field -15$^{\circ }$ is corrected, then the gps arrow at the center is deteriorated as well as the fuel gauge located at the lower center of the image.
Fig. 4. Experimental optical bench made up of a DLP pico projector, a first lens $L_1$, a beam splitter (BS), the SLM, and a second lens $L_2$
Fig. 5. Experimental results: (a) Projected image without wavefront correction, (b) With only distortion correction, and (c) with distortion and aberration correction in the ROI delimited with yellow circle
Fig. 6. a) Target image for a HUD application b) Experimental correction of the weather image located at the extreme angle (15$^{\circ }\times$7$^{\circ }$) b) while the GPS arrow and fuel gauge symbols located at the center are deteriorated
2.2.3 Chromatic dependence
Chromatic dispersion may appear depending on the width of the spectrum of the displayed image: If we display a white image for a correction calculated for a single wavelength (center of the spectrum $\lambda =533~nm$) we observe chromatic dispersion at the largest angles. Whereas if the displayed image is yellow (a combination of red and green), no observable chromatic dispersion occurs for limited FOV. In a first step, the ROI corrections were calculated for a wavelength of $533~nm$ (green). The same correction was then applied to the same predistorted grid for red, green and yellow colors. In Fig. 7 we observe the visual impact of the chromatism. The analysis of the output images was exclusively subjective because HUD images are to be observed visually so visual acceptability is the final criteria. A grid of lines was initially projected in order to evaluate the correction performance in terms of resolution and contrast, before testing it on symbols (GPS arrow, warning and signaling symbols $\cdots$) For the automotive HUD application, half of the spectrum from red to green is used (blue is rarely used in practice). Therefore for the HUD symbols we chose to display symbols in yellow. The images in Fig. 8 are a combination of two primary colors red and green. Chromatic dispersion is not noticeable even in the large FOVs. This indicates that all SLM correction phase maps can be calculated for an average wavelength and a multitude of colors can be displayed without increasing the number of phase maps required. This is important in the case of a HUD where a red-green (RG) display can be used over the whole FOV (30$^{\circ }$) not only in the narrow FOVs which is the case of the current HUDs. The system behaves as a quasi achromatic display for 30$^{\circ }$ FOV.
Fig. 7. Zoom on the ROI of experimental output images to show the impact of a correction calculated for a single wavelength $533~nm$ on the three colors images a) red b) green and c) yellow: a mix of green and red color. Left: uncorrected. Right: corrected
Fig. 8. Experimental correction for yellow color of the weather image located at the extreme angle (15$^{\circ }\times$7$^{\circ }$) while the GPS arrow and fuel gauge symbols located at the center are deteriorated
3. Holographic approach
The same SLM was also used as a holographic projector to generate an aberration reduced image. In this approach the design in 9 of a compact (1 liter), wide angle (30$\times$15$^{\circ }$) holographic display was achieved for the HUD application.
3.1 Optical set up
Holographic projection requires a coherent light source, in our case a $650~nm$ diode laser with a $20~mW$ optical output power. The SLM is the same as in section 2, its maximum phase delay is $2\pi$ and the $8~$µ$m$ pixel pitch permits diffraction angles up to 4.5$^{\circ }$. This diffraction angle is not sufficient for the desired FOV, therefore it needs to be optically magnified. However, this magnification leads to image distortion and aberration at high angles. We again use a foveal approach to achieve a 30$^{\circ }$ FOV and correct both aberrations and distortion. The set up is made up of a polarized laser module including a focusing lens $L_1$ which produces a Fourier transform of the CGH displayed on the SLM, and a second lens $L_2$ that magnifies the diffraction angles to 30$^{\circ }$ at a distance of $2.4~m$. We now present the system design choices followed by the experimental results
3.2 Calculations and experimental results
In this section, calculations and phase CGH generation algorithms are presented for the 30$^{\circ }$ holographic projector. To achieve this angle over $2.4~m$, the output image should be $1\times 0.5~m^{2}$. The SLM maximum diffraction angle of $4.5~^{\circ }$, must be magnified by $6.66$. For the design of our holographic projector, only the upper half of the image is displayed, to spatially filter the zeroth order and higher orders of diffraction. $L_2$ has a focal length of $25~mm$ and is placed $130~mm$ behind the SLM. For this configuration, distortion was first calculated and used to pre-compensate the input target images supplied to the Iterative Fourier Transform Algorithm (IFTA) [18] [19] used to calculate the CGHs displayed on the SLM. The wavefront aberrations were then simulated and their compensation added to the CGH as a corrective phase map before displaying it on the SLM. The simulation of the geometrical and wavefront aberrations can be done either by simulating a simple lens imaging set up in Zemax and calculating the pre-distortion and zernike coefficients for each ROI, or by an adaptive optical system (AOS) [11] that experimentally converges to the optimal coefficients based on the intensity distribution captured by a camera. In [11] both of these methods were tested and compared, the coefficients found by the experimental AOS gave the best rendering.
3.2.1 Distortion compensation
The distortion coefficients were calculated in the same way as for the foveal approach. The algorithm that calculates the phase function to be sent to the SLM follows the three steps:
STEP 1: The ROI is inserted with X and Y offset in an empty image and pre-distortion is applied to the image by the Python camera calibration algorithm.
STEP 2: This predistorted image is used as the IFTA target image in the CGH optimization. Noise is distributed across the whole output image outside the ROI area.
STEP 3: Spatially duplicate the phase CGH [20] [18] to sample the output image in order to produce a low speckle output image made up of separate dots [21], these dots become blurred by aberrations in the wide FOVs. The corrected phase CGH includes a correction based on the intensity distribution of the dots. The experimental distortion correction results are shown in Fig. 10 where a +-15$^{\circ }$ FOV holographic image is projected onto a screen at a distance of $2.4~m$ with and without geometrical aberration correction.
Fig. 10. Top: projection of a square in 4 different ROIs in increasing off axis position from 3.7$^{\circ }$ to 15$^{\circ }$ with an increment of 3.7$^{\circ }$, bottom: and the distortion correction results.
3.2.2 Foveal aberration correction
If the phase across the ROI is invariant, then one correction is sufficient to obtain a visually acceptable output image. In Fig. 11 symbols are inside the ROI, therefore they are sufficiently corrected. When the phase varies across the ROI as in Fig. 12, we need to divide the ROI into a number of separate sub zones depending on the ROI size. The closer the ROI is located towards the edge of the FOV the smaller it should be. For images larger than correctable ROIs, a temporal multiplexing of the corrections can be applied so that the eye sees the whole image corrected.
Fig. 12. Aberration correction of a distortion corrected warning sign located at +15$^{\circ }$. Left: With correction across two different ROIs 1 and 2: When a single zone is corrected the rest of the image is deteriorated
4. Foveated systems performance
We now assess here some key parameters related to imaging quality in the two foveal approaches. It is clear that both approaches (coherent source holographic projection and incoherent source imaging), behave differently with respect to a number of these parameters. The goal was to test the SLM’s ability to implement both approaches as well as possible and thus to compare its appropriateness to each approach. Even though the use of the SLM is different in each approach, we tried to make the comparison fair, by using the same optical elements and parts (e.g. FOV optics) as much as possible to highlight the advantages and limitations of both approaches. In particular we use the same phase only SLM in each set up, a key element in implementing the ROI corrections. However, we did not consider an important point concerning the time multiplexing of the corrections. The comparison was performed for fixed corrections, because constraints on the refresh rate differ for the two approaches. Furthermore, these constraints impact the SLM choice significantly as discussed below. Thus, our comparison only addresses features such as image quality, FOV, chromatic dependence, lighting efficiency and programmability.
4.1 Field of view and compactness
Both optical configurations were designed to fit in the same volume and the same FOV (30$^{\circ }$). However, in the incoherent case, the FOV, system volume and wavefront correction were impacted by the DLP imager’s divergence angle (here 47$^{\circ }$), and SLM active area ($15\times 7.68~mm$) which are technically imposed. The pupil stop size is a major constraint in the system design. An SLM with a larger pupil size will relax this constraint. This is not the case, in the coherent system where the beam width can be easily adjusted to the SLM dimensions and diffraction angles < 5$^{\circ }$. Of course, the overall compactness also depends on the source size.
4.2 Chromatic dependence
Here there is clearly an advantage for the incoherent case. We have seen that for the considered FOV, the chromatic dispersion can be largely neglected and that the optical system can be used from the red to green, maintaining a good imaging quality. However, for larger FOV values, this point should be reconsidered. In contrast, the coherent case is intrinsically chromatic. However, poly-chromatic images could be obtained by increasing the number of laser sources, with the counterpart of an increase in cost and possibly in volume.
4.3 Image quality
In both cases the image quality for a large FOV is a result of the wavefront correction applied by the SLM. We tried to keep almost the same optical set-up in order to assess the impact of the corrective optics. However, in the coherent case, the speckle effect must also be considered and could become predominant as the FOV increases. As we have seen, speckle reduction can be included in the optimization algorithm itself with the counterpart of a reduction of the CGH encoding capacity.
4.4 Brightness and radiometric efficiency
A major difference between the approaches is related to the light source. In the incoherent case the image source is a DLP, its brightness is approximately the same for each activated pixel. Depending on the ROI activated, only a part of the full image (around 10 %) is switched on, while the rest is switched off. Therefore, its brightness represents (in average) 10 % of the full DLP energy ($58 mW/cm^{2}$). Consequently, the ROI area on the DLP is a sub-image with a brightness of 6 mW . The main losses are due in this case to aperturing effects of the SLM pupil, with respect to the DLP divergence and the absorption in polarizers and the beam-splitter (roughly 70 to 75 %). The situation is not the same, of course, in the coherent case, where nearly all the laser energy is directed into the holographic image. In this case, light is lost in the undesired diffraction orders (spatially filtered here) and in the zeroth order due to the laser beam profile, wavelength mismatch with respect to the recommended operating wavelength, fill factor, spatial phase non-uniformity of the SLM and hologram computation noise. For instance, we measured here almost 30 % lost in the $0^{th}$ order. This value can be optimized by improved SLM calibration to reach only a few percent. In addition, the brightness of the holographic image, in contrast to the incoherent case, is not spatially constant, it depends on the ROI position and size. Finally, losses due to the beam-splitter are added as in the incoherent case. The estimated comparative radiometric efficiencies based on power measurements are given in Table 1.
$P_{in}$ is the measured input power of the system, $P_{out}$ the output of both measured and estimated power of the system. The radiometric ratio value is the output measured and estimated power divided by the input measured power:
Table 1. Radiometric efficiency of the coherent and incoherent approaches: $P_{in}$ data are the measured power at the entrance of the system, $P_{out}$ data are both measured and estimated and the radiometric ratio is the ratio of $P_{out}$ and $P_{in}$ both measured and estimated
4.5 Flexibility and programmability
In the incoherent case, the phase maps are calculated and optimized once, but for real time projection, images must be displayed and synchronized with the corresponding phase maps displayed on the phase SLM. In contrast, in the coherent case, the holograms are calculated with the corresponding wavefront corrections and so no synchronization is necessary. However, in this case a real time display is more complicated to achieve because of the CGH computation time. Finally as mentioned above, the time multiplexing capability remains an important aspect directly impacting the SLM choice. Time multiplexing is required in both cases, if multiple ROIs must be displayed at the same time with respect to the retinal persistence ($> 50 Hz$). This generally results in a trade-off between refresh rate and spatial resolution.
5. Foveal HUD set up
The aim of this section is to illustrate the performance of foveal corrections in a virtual image projection configuration representative of an automotive HUD. Due to the prohibitive cost of a freeform combiner designed specifically for our system, we chose to demonstrate the foveal method using the same incoherent foveal projector in section 2 coupled with an existing commercial 200$\times$100 mm$^{2}$ combiner as a deflecting and magnification freeform mirror. This off-the-shelf combiner is designed and optimized to form a 7$^{\circ }\times$3$^{\circ }$ virtual image at 2 m in front of the user’s eyes for an eye-box of 120$\times$60 mm$^{2}$. An anti reflective coated poly-carbonate sheet replaced the windshield and a driving simulation video was projected on a white screen at $2~m$ from the driver’s eyes. The commercial combiner used is designed to magnify a 44$\times$20 mm$^{2}$ TFT to a 250$\times$160 mm$^{2}$ virtual image at 2 m, so not optimized for our system. We adapted our set up to conjugate the SLM plane with the eye-box, and the DLP plane with the virtual image and tried to increase the FOV up to 11$^{\circ }\times$4$^{\circ }$ using the same combiner. To do so, we decreased the eye-box-combiner distance and decreased the eye-box size to the pupil diameter. Therefore, only one position of the eye in the eye-box is shown. The scheme of the set up in 13 is a demonstration of the foveal method to correct by sectors the FOV. In the foveal projector in Fig. 13 we use a $25~mm$ focal length lens $L_1$ to image the DLP output (system entrance pupil) onto the phase SLM and a $50~mm$ focal length lens $L_2$ combined with $L_1$ to conjugate the DLP image with the virtual image and the SLM with the eye-box. We first displayed a grid of lines on the DLP as in Fig. 14 to validate the system FOV, focus adjustments, and observe the aberrations that occur at the wide FOV angles.
Fig. 14. 11$^{\circ }\times$4$^{\circ }$ Virtual image through the combiner of a grid of lines without wavefront correction
To illustrate the aberration correction on the virtual image we placed a camera in a fixed position in the eyebox, to observe the aberrations across the FOV and select an ROI on which to perform aberration correction. Once the wavefront correction phase map of the selected ROI was displayed on the SLM for a given ROI, we were able to use the same correction successfully for a variety of symbols such as a warning triangle symbol as an example in Fig. 15.
6. Conclusion
In this paper, we have investigated how a re-configurable phase only SLM can be used to design a compact foveated automotive HUD considering the most volume constrained part is the imaging or projection system. Hence, first we designed and tested the projection system, before embedding it in an automotive HUD configuration. For the projection system, we tested two different light sources which enabled us to assess the specific advantages of two SLM configurations: holographic display and wavefront corrector, to implement optical foveated imaging systems under strong constraints of volume and compactness, which are intrinsic in automotive HUDs. We have emphasized how the foveated approach can be beneficial in this particularly demanding case. We tried, as much as possible to maximize the number of common elements in both systems. This enabled us to show the versatility of the SLM and how it can be used efficiently in both configurations. In the holographic case we demonstrated how the inclusion of aberration correction in the CGH design can significantly improve the quality of the displayed HUD symbols. More generally, this paper demonstrates how a foveal based approach can be used to optimize the local image quality which is particularly relevant in the framework of augmented reality automotive HUD systems where localized information (sub-images or symbols) must be superimposed in specific zones across a wide FOV. We demonstrate in both the incoherent and the coherent cases how foveal correction can be beneficial in automotive HUDs. The wavefront correction was tested in a HUD set up using a commercial HUD combiner (which were designed for 7$^{\circ }\times$3$^{\circ }$ virtual image at $2~m$) coupled with the foveal incoherent projection system to display and locally correct a wider 11$^{\circ }\times$4$^{\circ }$ FOV virtual image. The necessary use of an existing commercial combiner in the final HUD operation demonstration, rather than one specifically optimized for our system, does not allow us to compare the total HUD system volume (including magnification optics, freeform combiner) to that of existing commercialized HUDs. However, the feasibility and advantages of our foveal approach in improving perceived image quality over a larger FOV and in reducing the system volume in the critically important below dashboard space have clearly been demonstrated. Although, our system is not optimized (using an unmatched combiner, aberrated lenses and a SLM with a maximum phase delay of 2$\pi$ and pixel pitch of 8 µm), if we used a curved mirror, achromatic lenses, and an SLM with higher phase delay, pixel pitch and resolution (such as phase SLMs proposed by Meadowlark Optics) we would have performed better resolution in the virtual projection plane. A 11$^{\circ }\times$4$^{\circ }$ at 2m freeform based HUD will be constrained to use at least 2 freeforms to address the wavefront and a TFT with at least 79$\times$28 mm$^{2}$ size. With our system we can couple the foveal PGU that controls the wavefront and whose volume is less than 2L, with only one freeform mirror to achieve a FOV of 11$^{\circ }\times$4$^{\circ }$. Future work now concentrates on leveraging the time multiplexing resource to enable the display of numerous sub-images in different ROIs at the same time and hence permit a better image quality trade off across the whole FOV.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. J. C. Wyant, “Basic wavefront aberration theory for optical metrology,” APPLIED OPTICS AND OPTICAL ENGINEERING XI, 28–37 (2015).
2. E. Marín, P. Gonzalez Prieto, M. Maroto Gómez, and D. Villegas, “Head-up displays in driving,” researchgate (2016).
3. A. Corning, “Ensuring the quality of next-generation automotive hud,” (August 17, 2020).
4. “New volkswagen id.3 and id.4 augmented head-up display,” (21 December 2020).
5. TI, “Dlp technology: Solving design challenges in next generation of automotive head-up display systems,” (December 2014-Revised November 2017).
6. T. Martinez, D. V. Wick, and S. R. Restaino, “Foveated, wide field-of-view imaging system using a liquid crystal spatial light modulator,” Opt. Express 8(10), 555–560 (2001). [CrossRef]
7. T. M. D. v. Wick and S. R. Restaino, , and B. R. Stone, “Foveated imaging demonstration,” Opt. Express 10, 60–65 (2002). [CrossRef]
8. https://docs.opencv.org/master/dc/dbb/tutorial_py_calibration.html.
9. C. Chang, K. Bang, G. Wetzstein, B. Lee, and L. Gao, “Toward the next-generation vr/ar optics: a review of holographic near-eye displays from a human-centric perspective,” Optica 7(11), 1563–1578 (2020). [CrossRef]
10. M. Weier, T. Roth, E. Kruijff, A. Hinkenjann, A. Pérard-Gayot, P. Slusallek, and Y. Li, “Foveated real-time ray tracing for head-mounted displays,” Comput. Graph. Forum 35(7), 289–298 (2016). [CrossRef]
11. A. Kaczorowski, G. S. D. Gordon, and T. D. Wilkinson, “Adaptive, spatially-varying aberration correction for real-time holographic projectors,” Opt. Express 24(14), 15742–15756 (2016). [CrossRef]
12. G. Curatu, D. V. Wick, D. M. Payne, T. Martinez, J. Harriman, and J. E. Harvey, “Wide field-of-view imaging system using a liquid crystal spatial light modulator,” Proc SPIE 5874, 80–86 (2005). [CrossRef]
13. G. Curatu and J. E. Harvey, “Lens design and system optimization for foveated imaging,” Proc SPIE 7060, 170–178 (2008). [CrossRef]
14. L. H. Sverdrup and M. Belenkii, “Dynamic foveal vision display,” (2010). Patent US9529191B2.
15. A. D. O’Neill, J. C. Davidson, T. J. Kane, R. M. Narayanan, and N. K. Dhar, “Considerations in the development of a foveated imaging system for unmanned aerial vehicles (uavs),” Proc SPIE 10656, 146–155 (2018). [CrossRef]
16. O. Ripoll, V. Kettunen, and H. P. Herzig, “Review of iterative fourier-transform algorithms for beam shaping applications,” Opt. Eng. 43(11), 2549 (2004). [CrossRef]
17. https://holoeye.com/slm-pluto-phase-only/.
18. F. Wyrowski, R. Hauck, and O. Bryngdahl, “Computer-generated holography: hologram repetition and phase manipulations,” J. Opt. Soc. Am. A 4(4), 694–698 (1987). [CrossRef]
19. A. Kaczorowski, G. S. Gordon, A. Palani, S. Czerniawski, and T. D. Wilkinson, “Optimization-based adaptive optical correction for holographic projectors,” J. Display Technol. 11(7), 596–603 (2015). [CrossRef]
20. F. Wyrowski and O. Bryngdahl, “Speckle free reconstruction in digital holography,” J. Opt. Soc. Am. A 6(8), 1171–1174 (1989). [CrossRef]
21. P. M. Bernard C. Kress, Applied Digital Optics: From Micro-Optics to Nanophotonics (John Wiley and Sons, Ltd, 2009).
