This paper presents the first proof-of-concept implementation and the principle that realizes a projection display whose contrast does not decrease even with existing inter-reflection of projection light or environmental light. We propose the use of photochromic compounds (PhC) to control reflectance of a projection surface. PhC changes color chemically when exposed to UV light. A PhC is applied to a surface to control its reflectance by radiating UV light from a UV-LED array. An image is projected from a visible projector onto the surface to boost the contrast. The proof-of-concept experiment shows that the prototype system achieves approximately three times higher contrast than a projection-only system under natural light.
© 2014 Optical Society of America
Following the pioneering work by Seetzen et al. , a significant amount of research and development of high contrast or high dynamic range (HDR) display technologies has been conducted. The displayable contrast of recent off-the-shelf LCD TVs equipped with an LED array backlight has increased by spatially varying backlight modulation. In computer graphics and information display fields, researchers are currently focusing on increasing the contrast of not only flat panel displays but also of projectors [2, 3]. Projectors are widely applied to immersive virtual reality (VR) and augmented reality (AR) applications to display images on nonplanar (e.g., cylindrical and hemispherical) immersive screens, as an alternative to head-mounted displays. Such technology merges physical and virtual worlds by mapping graphical textures on complex real-world surfaces [4, 5]. Because higher contrasts of projectors could realize more immersive experiences, it is regarded as an important technical issue in VR and AR research fields to increase contrasts of projectors.
However, even if contrasts of projectors are increased, contrasts of displayed images still suffer from other global lighting effects. For example, the inter-reflection of projected light within immersive concave screens increases the black offset of a projected image in VR applications. Normal AR applications are designed to be used under environmental light, which also increases the black offset. Because contrast is defined as the ratio of the brightest intensity to the darkest intensity, the contrast of projected image becomes lower with higher black offset.
In this paper, we propose a high contrast display technique by decreasing the black offset while maintaining the brightest intensity. To achieve this, we modulate the reflectance of a projection screen spatially by applying photochromic compounds (PhCs). In general, a PhC chemically changes its color when it is exposed to ultraviolet (UV) light. The basic idea of the proposed method is to paint a PhC onto the surface of a screen and spatially modulate its reflectance by radiating UV light from an array of UV LEDs. Then, an image or spatially modulated light, is projected from an ordinary projector onto the surface to boost the contrast (Fig. 1).
We propose a technique that controls PhCs in order to display desired reflectance patterns. Specifically, our technique deals with the spread of UV light from UV LEDs, by which a single point on a screen is illuminated by multiple LEDs. Further, we describe how to split the original high contrast image into reflectance distribution and projected illumination. We have constructed the first proof-of-concept system and conducted a simple evaluation to determine if the proposed principle increases the contrast compared to a normal projection system under the condition where environmental light is illuminated on the projection screen.
2. Related works
In general, HDR or high contrast display is achieved by double modulation of light. For example, Seetzen et al. applied an LED array backlight as the first modulator and an LCD panel as the second modulator . Suppose the contrasts of the first and second modulators are L1 : 1 and L2 : 1, respectively, the final contrast of the display is L1L2 : 1.
High contrast projectors are also realized by double modulation of light. Hoskinson et al. applied an analog micromirror array (AMA) as the first modulator and a digital mirror device (DMD) as the second modulator . They achieved approximately 2.5 times higher contrast than normal projection. Kusakabe et al. applied liquid crystal on silicon (LCOS) modules as the first and second modulators to realize a high contrast projector . Despite the fact that these projectors can display images with high contrast, global lighting effects inevitably degrade the contrast of projected images in immersive VR applications where nonplanar projection surfaces cause inter-reflection of projected images and in AR applications that are generally used under environmental light. Although several techniques have been proposed to compensate the inter-reflection by precorrecting projection images [6, 7], these techniques cannot compensate for environmental light.
Bimber and Iwai took an approach called superimposing dynamic range (SDR) to spatially modulate the reflectance of a planar projection screen so that light is modulated in a projector and then on a printed hardcopy before being perceived by the observer . More specifically, bright pixels are projected onto the white pigment of a hardcopy, and dark pixels are projected onto the black pigment. This approach potentially solves the contrast degradation problem caused by both inter-reflection and environmental light. Shimazu et al. extended this concept to solid surfaces to realize 3D high contrast representations . They employed a mul-tiprojection system to superimpose images onto a textured solid hard copy output by a color 3D printer. However, these techniques only allow static representations because the texture on the hardcopy is comprised of printed pigments. We propose to use PhCs to control the reflectance of the projection surface dynamically to realize dynamic high contrast display based on the SDR principle.
3. Dynamic modulation of screen reflectance using PhC
In the proposed technique, a PhC is painted on the surface of a projection target. Then, the reflectance of the surface is dynamically controlled by radiating an optimal amount of spatially varying UV light from an array of UV LEDs. In general, the color of PhC is white originally, which then becomes darker when exposed to UV light and reverts to its original state after UV radiation stops. In other words, the reflectance of the surface where the PhC is painted is originally high, decreases when exposed to UV light and returns to its high state after UV radiation ends.
This section explains how our proposed technique determines appropriate input values for each UV LED to reproduce a target reflectance distribution on the projection surface. We consider a system in which a 2D array of UV LEDs spatially projects modulated UV light to a PhC painted surface and an ordinary projector projects a visible image onto the same surface, as shown in Fig. 2.
3.1. Reflectance control
We propose a reflectance control technique that can reproduce a desired reflectance distribution by modulating each UV LED in the array appropriately. The reflectance of the PhC monotonically decreases with increase in the illuminance of the UV light. The relationship between the reflectance of the PhC at a point x on the surface and the incident illuminance of UV light at that point can be expressed as
Because UV lights are spread from each LED, each point on the screen is radiated from multiple LEDs simultaneously. A normalized input value sent to the k-th LED is denoted as ik (0.0 ≤ ik ≤ 1.0) (Fig. 3). The illuminance of UV light radiated from the k-th LED with the input value of 1.0 (i.e., ik = 1.0) at point x is denoted as wk(x). Assuming that the relationship between an input value sent to an LED and the resulting UV output is linear, the illuminance at x from the k-th LED with ik can be computed as ikwk(x). Consequently, the UV illuminance at x radiated from n LEDs can be computed as the sum of the illuminance from the LEDs as shown below
Suppose a target reflectance at x is represented as r̂(x), the optimum input value ik for each LED in the array must be determined by minimizing the sum of squared errors of the generated reflectance from the target reflectance
3.2. Reformulation for calibration
It is difficult to control the UV illuminance e(x) on the surface directly. Therefore, we introduce a controllable parameter e′(x) as follows. When an equal input value iall is sent to all LEDs of the array simultaneously, then, according to Eq. (2), the UV illuminance e(x) at each point x isEq. (5), the input value iall is proportional to the UV illuminance e(x). Therefore, we regard iall as the mediating parameter e′(x), and define the new monotonically decreasing function f′x, which describes the relationship between e′(x) and reflectance, as follows Eq. (3), to find the optimum input value ik as follows
The function f′x and the distribution of each LED w′k(x) is calibrated to solve Eq. (7). The calibration is required once for each experimental setup. We assume that the environmental light condition is stable.
3.3.1. Function f′x
The function f′x represents the relationship between surface reflectance and the mediating parameter e′(x). The surface reflectance is measured under different UV illuminance by using a calibrated camera, which allows the measurement of absolute reflectance at each point on the surface from the corresponding pixel value. The camera is calibrated in advance by measuring a standard object of known reflectance under equal environmental light condition.
The specific calibration process is described as follows. We send iall to all LEDs simultaneously, and then measure reflectance at each point x. Then, the relationship between iall and the corresponding reflectance is stored as a lookup table. This process begins with iall = 1.0, and is repeated with different iall values ranging from 1.0 to 0.0. This lookup table is used as the function f′x.
3.3.2. Illuminance distribution of an LED w′k(x)
The mediating distribution parameter of each LED w′k(x) is measured as follows. The maximum value 1.0 is sent to the k-th LED, and the value 0.0 is sent to the other LEDs (i.e., ik = 1.0 and ij = 0.0 (j ≠ k)). The reflectance r(x) is measured by the camera at each point x on the screen. Then, using the inverse of function f′x, e′(x) is obtained from the measured reflectance value, which is identical to w′k(x). This process is repeated sequentially for each LED in the array.
4. High contrast projection display by reflectance modulation
An original high contrast image is split into two parts; first is for the visible projector and second is for the surface reflectance. This allows the original image to be reproduced by superimposing the split images. The flow of the splitting is depicted in Fig. 4.
The target luminance, which should be displayed at x, is denoted by l(x). Suppose the minimum and maximum luminance values are lm and lM respectively, the normalized target luminance L(x) can then be calculated as
Assuming that an ordinary 8-bit projector is used and the reflectance of the PhC is modulated by 8-bit pulse width modulation (PWM) of the UV LEDs, the normalized target reflectance value R(x) can then be computed as1]. Then, the target reflectance value r̂(x) cam be derived as Eq. (7), the input value ik for each UV LED can be obtained to realize the reflectance r̂(x) as accurately as possible.
Because the spatial resolution of the reflectance modulation by the array of UV LEDs is much lower than that of the incident illuminance modulation by the visible projector, the desired reflectance distribution r̂(x) computed by Eq. (11) cannot be correctly reproduced. Thus, after the input values for the UV LEDs are computed, the actual reflectance distribution r̃(x) is estimated using Eqs. (5) and (8) when these input values are sent to the LEDs. Then, a projection image is computed. Because the output luminance is the result of the multiplication of the reflectance and illuminance from the projector, the target illuminance p̂(x) is calculated by the following equation.10].
We conducted a proof-of-concept experiment using a prototype system. The first prototype system of the proposed principle consisted of a flat projection surface and a 2D LED array. In the experiment, we evaluated how the proposed principle increased contrast compared to a normal projection system under the condition where environmental light illuminated the screen.
5.1. Experimental setup
We mixed three types of PhCs so that the mixed material shows gray when exposed to UV light. The combination of PhCs was PSP-54 (20 %), PSP-33 (22.5 %), and PSP-73 (57.5 %). All the materials were produced by Yamada Chemical Co., Ltd. We painted the combined material on a 40×20 mm2 sheet of transparent acrylic that is the screen in this experiment. We used an Acer K10 projector (100 ANSI lumen) and a Point Grey Research Chameleon camera (1280×960 pixels). We constructed a 2D LED array with 32 (8 × 4) UV LEDs (Nitride Semiconductors Co., Ltd., NS365L-3RLQ) and modulated the UV intensity at 8 bit by the PWM. The screen was illuminated by approximately 120×60 projector pixels. Figure 5 shows the experimental setup.
The experimental system was calibrated as described in Section 3.3. First, the function f′x was measured, which represents the relationship between the reflectance of the surface and the input value to the LED array. An initial input value of 1.0 for iall was used to project UV light onto the surface. We then decreased the input value iall to 0.0. Each time the input value was changed, we measured the reflectance. Figure 6 shows the measured f′x.
We then measured UV illuminance distribution from each LED. The maximum value 1.0 was sent to each LED individualy and the reflectance of the surface was measured (Fig. 7). Then, we converted the reflectance to w′k(x) using the function f′x.
We measured the actual contrast of the projector in advance by projecting uniform white and black images onto the surface under environmental light and capturing the images using the camera. The measured contrast of the projector was 17:1. We also measured the actual contrast of the surface by sending the maximum and minimum values to all LEDs and capturing them using the camera under the same environmental light condition. The measured contrast of the surface was 4:1. Therefore, the theoretical contrast of the system could be computed as the multiplication of these values, resulting in a theoretical contrast of 68:1.
We then measured the actual contrast of the system. We sent the minimum value, i.e., 0.0, to all LEDs and projected a uniform white image on the surface to measure the brightest luminance. Then, we sent the maximum value, i.e., 1.0, to all LEDs and projected a uniform black image on the surface to measure the darkest luminance. The measured actual contrast of the system was 60:1, which was almost the same as the theoretical contrast. The contrast of the system was approximately 3.5 times higher than that of the projector and 15 times higher than that of the reflectance modulation. To evaluate the frequency response of the system, we used six fringe patterns of different frequencies as target images (Fig. 8(a)). The numbers of black and white fringe pairs compositing the target images were 1, 2, 4, 8, 16, and 32, respectively. Figure 8(b) shows the contrast ratios achieved both when the target fringe patterns were projected onto the surface (conventional condition) and when they were displayed on the surface with reflectance modulation by the proposed method (proposed condition). From the result, we found that the contrasts under both conditions decreased with increasing fringe frequency. We also found that the proposed method always provided better contrast, while the contrast increase ratio decreased with increase in fringe frequency (Fig. 8(c)).
Figure 9 shows the experimental results. We used four grayscale images as target luminance (three natural images and one simple gradation) (Fig. 9(a)). The natural images were identified as image 1, 2, and 3, and the gradation image was identified as image 4. Figure 9(b) shows the displayed luminance under the condition where the images were projected onto the surface without reflectance modulation (conventional condition). Figure 9(c) shows the displayed luminance under another condition where the reflectance of the surface was modulated according to the target image (proposed condition). Figures 9(d), 9(e), and 9(f) show the false color representations (blue: low luminance, red: high luminance) of Figs. 9(a), 9(b), and 9(c), respectively. Figure 10 shows a comparison of luminance values between a target and the corresponding projected results under the conventional and proposed conditions. From the results, we confirmed that the low intensities were better reproduced under the proposed condition than the conventional condition. Table 1 shows the actual contrast ratios achieved in the experimental results. We could confirm the contrast enhancement of up to 2.8 times. The contrast ratios under the proposed condition did not reach 60:1, because the target images were not spatially uniform colors but textured images. These contrasts were rather close to that achieved when the fringe pattern comprising two of black and white fringe pairs was the target image (see Fig. 8(b)).
We objectively evaluated the image qualities of the projected results using the structural similarity index (SSIM), which is a method for assessing perceptual quality of a distorted image compared to the original (Fig. 9(a) in this experiment) . Table 2 shows the results. The displayed images under the proposed condition provided better image qualities than the conventional condition.
We assessed the reproducibility of the proposed reflectance control technique, on which the final image quality highly depends. Figures 11(a), 11(b), and 11(c) show the target reflectance distributions r̂(x), estimated reflectance distributions r̃(x), and displayed results of reflectance distributions, respectively. Note that the white and black pixel colors are assigned to the maximum (1.0) and minimum (0.0) reflectance values, respectively. The mean squared errors between the estimated reflectance and displayed results were 0.023 (image 1), 0.024 (image 2), 0.026 (image 3), and 0.022 (image 4). From the results, we confirmed that the proposed reflectance control technique reproduces the target reflectance distributions with high precision.
We have confirmed from the experiment that the current prototype system achieved 3.5 times greater contrast than projection-only system. Although the contrast increase ratio appears small, it is comparable to that achieved in a previous high contrast projection system, which was 2.5 . It is our belief that we can improve the contrast increase ratio by placing UV illuminators in front of the surface (front illumination, as shown in Fig. 12(a)) rather than behind the surface (back illumination). We informally tested front UV illumination with a prototype, as shown in Fig. 12(b), and found that we could achieve the contrast increase ratio greater than 10. However, the prototype shown in Fig. 12(b) has a problem in the system configuration; the LEDs and circuits cover most of the screen and observers cannot view all of the displayed content. This system configuration issue can be solved by applying a UV projector  rather than UV LEDs by placing the UV projector close to a visible projector and projecting UV patterns onto a PhC painted screen to modulate the reflectance pattern on the screen. In such a configuration, the projector would not disturb the observers views. Another possibility to improve the contrast increase ratio is further development of more advanced photochromic materials.
The results of displaying fringe patterns showed that the contrast ratio decreased with increasing fringe frequency (Fig. 8(b)). Ordinary projection display systems generally share the same limitation that is caused by undesirable scattering both within the projector’s optical system and on the projection surface. Figure 8(c) showed that the contrast increase ratio also decreased with increasing fringe frequency. This happened because multiple projector pixels illuminated the area covered by each of the LEDs; i.e., the spatial resolution of reflectance modulation was much lower than that of projected image. We believe that this does not lead to a serious problem, because previous works such as  also applied the combination of low and high resolution modulators by taking into account the effect of veiling glare, i.e., the local reduction of contrast on the retina by scattering in the eye.
The calibration requires considerable time when a significant number of UV LEDs are used because our brute-force calibration method requires each individual LED to be turned on sequentially for the calibration of UV illuminance distribution. The number of LEDs in the current prototype, which was only 32, is obviously insufficient for large displays (e.g., TVs with LED backlight normally consist of thousands of LEDs). The efficiency of the calibration process can be improved by applying an adaptive method in which multiple LEDs can be switched on simultaneoously if the UV light from these LEDs do not overlap on screen. Such adaptive method is possibly developed based on an efficient light transport matrix acquisition technique .
The use of PhCs introduces two limitations in the current prototype. First, the PhC could display only greyscale reflectance distributions. A full color PhC printing technology  could improve results in terms of color representation. Second, we could switch different images displayed on the same surface in the experiment, which confirmed that the system can display dynamic content. However, the refresh rate of the PhC used in the experiment was very slow (less than 1 Hz). Recently, researchers have succeeded in synthesizing a PhC that has a quick response (nearly 200 ms) . Once further research and development of PhCs realize a material that can be controlled at common video rates, such as 30 Hz, we will be able to display moving images using the proposed system.
Two other limitations are due to the current UV illumination based on the UV-LED array. The first limitation is related to the scalability of the system. In AR applications, projection targets have various shapes. However, an LED array is not suitable to support diversity of screen shape because unique circuits must be developed for each screen shape. In addition, projection screens in a commercial office environment are generally of a rolled form that is also not suitable for aligning an LED array. Second, in the current prototype, the spatial resolution of reflectance modulation was much lower than the projected imagery. These limitations can be solved by using a UV projector, as used in a previous study .
6.2. Potential applications
The potential applications of the proposed system vary widely (Fig. 13). In a business presentation scenario where multiple people share projected slides or presentations on a white planar screen, presentations are normally given in a dark environment to maintain the contrast of the slides. However, once the proposed technique is applied, people could give presentations and listen while sharing projected slides under daily environmental lighting conditions without losing the contrast of their slides (Fig. 13(a)). Another practical application would be commercial advertising displays where a higher ambient light level would be expected and a lower refresh rate could be applied.
In immersive VR contexts, the inter-reflections of projected lights decrease the contrast when convex screens (e.g., corner, cube, cylindrical, and dome screens) are used. Once the proposed technique is applied, the inter-reflections can be minimized, and the contrast of the displayed content can be increased by modulating the reflectance of the screens adaptively (Fig. 13(b)).
The proposed technique is also useful in AR applications. Although AR technology is normally used in an ordinary situation where environmental light illuminates the projection target, the contrast of projected imagery becomes low under such environment light. If a UV projector and our technique are used together, we can achieve a high contrast representation on a 3D projection target fabricated from a 3D printer with a PhC on its surface (Fig. 13(c)). 3D high contrast representation is useful in several applications, such as industrial design for the assessment of a product prototype or in archeology for the realistic physical reproduction of digitally archived historic artifacts.
This paper has presented a high contrast projection display technique and its first prototype that modulates projected lights on the projection surface by controlling the surface reflectance spatially. This is achieved using a PhC and a UV-LED array. An experiment using the prototype system confirmed that the proposed technique achieved 3.5 times higher contrast than a normal projection system. We have discussed both limitations and potential applications of the proposed technique. In the future, we will apply the principle to 3D solid surfaces using a UV projector and find a more effective PhC to improve the image quality.
This work was supported by Grant-in-Aid for Scientific Research on Innovative Areas “Shit-sukan” (No. 22135003) from MEXT, Japan.
References and links
1. H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, L. Whitehead, M. Trentacoste, A. Ghosh, and A. Vorozcovs, “High dynamic range display systems,” ACM Trans. Graphics 23(3), 760–768 (2004). [CrossRef]
2. R. Hoskinson, B. Stoeber, W. Heidrich, and S. Fels, “Light reallocation for high contrast projection using an analog micromirror array,” ACM Trans. Graphics29(6), Article 165 (2010). [CrossRef]
3. Y. Kusakabe, M. Kanazawa, Y. Nojiri, M. Furuya, and M. Yoshimura, “A YC-separation-type projector: high dynamic range with double modulation,” J. Soc. Information Display 16(2), 383–391 (2008). [CrossRef]
4. O. Bimber and R. Raskar, Spatial augmented reality: merging real and virtual worlds (A. K. Peters Ltd., 2005).
5. R. Raskar, G. Welch, K. L. Low, and D. Bandyopadhyay, “Shader lamps: animating real objects with image-based illumination,” In Proceedings of the Eurographics Workshop on Rendering Techniques, 89–102 (2001).
6. O. Bimber, A. Grundhöfer, T. Zeidler, D. Danch, and P. Kapakos, “Compensating indirect scattering for immersive and semi-immersive projection displays,” In Proceedings of the IEEE Conference on Virtual Reality, 151–158 (IEEE, 2006). [CrossRef]
7. H. Habe, N. Saeki, and T. Matsuyama, “Inter-reflection compensation for immersive projection display,” In Proceedings of the IEEE International Workshop on Projector-Camera Systems, 2 pages (IEEE, 2007).
8. O. Bimber and D. Iwai, “Superimposing dynamic range,” ACM Trans. Graphics27(5), Article 150 (2008). [CrossRef]
9. S. Shimazu, D. Iwai, and K. Sato, “3D high dynamic range display system,” in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, 235–236 (IEEE, 2011).
10. O. Bimber, D. Iwai, G. Wetzstein, and A. Grundhöfer, “The visual computing of projector-camera systems,” Computer Graphics Forum 27(8), 2219–2254 (2008). [CrossRef]
11. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Perceptual image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13(4), 600–612 (2004). [CrossRef] [PubMed]
12. T. Hashida, Y. Kakehi, and T. Naemura, “Photochromic sculpture: volumetric color-forming pixels,” In Proceedings of ACM SIGGRAPH Studio Talks, Article 11 (2011). [CrossRef]
13. P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. Lensch, “Dual photography,” ACM Trans. Graphics 24(3), 745–755 (2005). [CrossRef]
14. S. Hirano, H. Takahashi, and I. Kawashima, “Photon-mode full-color rewritable image media using photochromic compounds,” Ricoh Tech. Report 29, 79–83 (2003).