Abstract

Display devices, or displays, such as those utilized extensively in cell phones, computer monitors, televisions, instrument panels, and electronic signs, are polarized light sources. Most displays are designed for direct viewing by human eyes, but polarization imaging of reflected light from a display can also provide valuable information. These indirect (reflected/scattered) photons, which are often not in direct field-of-view and mixed with photons from the ambient light, can be extracted to infer information about the content on the display devices. In this work, we apply Stokes algebra and Mueller calculus with the edge overlap technique to the problem of extracting indirect photons reflected/scattered from displays. Our method applies to recovering information from linearly and elliptically polarized displays that are reflected by transmissive surfaces, such as glass, and semi-diffuse opaque surfaces, such as marble tiles and wood furniture. The technique can further be improved by applying Wiener filtering.

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Removal of reflections in LWIR image with polarization characteristics

Ning Li, Yongqiang Zhao, Quan Pan, and Seong G. Kong
Opt. Express 26(13) 16488-16504 (2018)

Engineering of head-mounted projective displays

Hong Hua, Axelle Girardot, Chunyu Gao, and Jannick P. Rolland
Appl. Opt. 39(22) 3814-3824 (2000)

Real-time pseudocolor coding thermal ghost imaging

Deyang Duan and Yunjie Xia
J. Opt. Soc. Am. A 31(1) 183-187 (2014)

References

  • View by:
  • |
  • |
  • |

  1. M. Levoy and P. Hanrahan, “Light field rendering,” in 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42.
  2. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45, 5453–5469 (2006).
    [Crossref]
  3. B. M. Ratliff and J. S. Tyo, “Moving towards more intuitive display strategies for polarimetric image data,” Proc. SPIE 10407, 104070E (2017).
    [Crossref]
  4. Y. Y. Schechner, J. Shamir, and N. Kiryati, “Polarization and statistical analysis of scenes containing a semireflector,” J. Opt. Soc. Am. A 17, 276–284 (2000).
    [Crossref]
  5. Y. Ding, A. Ashok, and S. Pau, “Real-time robust direct and indirect photon separation with polarization imaging,” Opt. Express 25, 29432–29453 (2017).
    [Crossref]
  6. H. Farid and E. H. Adelson, “Separating reflections from images by use of independent component analysis,” J. Opt. Soc. Am. A 16, 2136–2145 (1999).
    [Crossref]
  7. A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
    [Crossref]
  8. N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014).
    [Crossref]
  9. Y. Diamant and Y. Y. Schechner, “Overcoming visual reverberations,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.
  10. Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.
  11. T.-P. Wu and C.-K. Tang, “Separating specular, diffuse, and subsurface scattering reflectances from photometric images,” in European Conference on Computer Vision (ECCV) (Springer, 2004), pp. 419–433.
  12. B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.
  13. X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.
  14. R. A. Chipman, “Mueller matrices,” in Handbook of Optics (McGraw-Hill, 2010), Vol. 1.
  15. E. Hecht, Optics (Pearson Education, 2016).
  16. S.-Y. Lu and R. A. Chipman, “Interpretation of Mueller matrices based on polar decomposition,” J. Opt. Soc. Am. A 13, 1106–1113 (1996).
    [Crossref]
  17. Thorlabs, “CMOS Cameras: USB 2.0 and USB 3.0,” (2017), retrieved https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=4024 .
  18. W. L. Hsu, J. Davis, K. Balakrishnan, M. Ibn-Elhaj, S. Kroto, N. Brock, and S. Pau, “Polarization microscope using a near infrared full-Stokes imaging polarimeter,” Opt. Express 23, 4357–4368 (2015).
    [Crossref]
  19. W. L. Hsu, G. Myhre, K. Balakrishnan, N. Brock, M. Ibn-Elhaj, and S. Pau, “Full-Stokes imaging polarimeter using an array of elliptical polarizer,” Opt. Express 22, 3063–3074 (2014).
    [Crossref]
  20. M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Cengage Learning, 2014).
  21. J. G. Nagy and D. P. O’Leary, “Restoring images degraded by spatially variant blur,” SIAM J. Sci. Comput. 19, 1063–1082 (1998).
    [Crossref]

2017 (2)

B. M. Ratliff and J. S. Tyo, “Moving towards more intuitive display strategies for polarimetric image data,” Proc. SPIE 10407, 104070E (2017).
[Crossref]

Y. Ding, A. Ashok, and S. Pau, “Real-time robust direct and indirect photon separation with polarization imaging,” Opt. Express 25, 29432–29453 (2017).
[Crossref]

2015 (1)

2014 (2)

N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014).
[Crossref]

W. L. Hsu, G. Myhre, K. Balakrishnan, N. Brock, M. Ibn-Elhaj, and S. Pau, “Full-Stokes imaging polarimeter using an array of elliptical polarizer,” Opt. Express 22, 3063–3074 (2014).
[Crossref]

2006 (1)

2005 (1)

A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
[Crossref]

2000 (1)

1999 (1)

1998 (1)

J. G. Nagy and D. P. O’Leary, “Restoring images degraded by spatially variant blur,” SIAM J. Sci. Comput. 19, 1063–1082 (1998).
[Crossref]

1996 (1)

Adelson, E. H.

Ashok, A.

Balakrishnan, K.

Boyle, R.

M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Cengage Learning, 2014).

Brock, N.

Bronstein, A. M.

A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
[Crossref]

Bronstein, M. M.

A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
[Crossref]

Chenault, D. B.

Chipman, R. A.

Davis, J.

Debevec, P.

B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.

Diamant, Y.

Y. Diamant and Y. Y. Schechner, “Overcoming visual reverberations,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

Ding, Y.

Durand, F.

Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.

Farid, H.

Freeman, W. T.

Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.

Ghosh, A.

B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.

Goldstein, D. L.

Greenberg, D. P.

X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.

Hanrahan, P.

M. Levoy and P. Hanrahan, “Light field rendering,” in 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42.

He, X. D.

X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.

Hecht, E.

E. Hecht, Optics (Pearson Education, 2016).

Hlavac, V.

M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Cengage Learning, 2014).

Hsu, W. L.

Ibn-Elhaj, M.

Kiryati, N.

Kong, N.

N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014).
[Crossref]

Krishnan, D.

Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.

Kroto, S.

Lamond, B.

B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.

Levoy, M.

M. Levoy and P. Hanrahan, “Light field rendering,” in 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42.

Lu, S.-Y.

Myhre, G.

Nagy, J. G.

J. G. Nagy and D. P. O’Leary, “Restoring images degraded by spatially variant blur,” SIAM J. Sci. Comput. 19, 1063–1082 (1998).
[Crossref]

O’Leary, D. P.

J. G. Nagy and D. P. O’Leary, “Restoring images degraded by spatially variant blur,” SIAM J. Sci. Comput. 19, 1063–1082 (1998).
[Crossref]

Pau, S.

Peers, P.

B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.

Ratliff, B. M.

B. M. Ratliff and J. S. Tyo, “Moving towards more intuitive display strategies for polarimetric image data,” Proc. SPIE 10407, 104070E (2017).
[Crossref]

Schechner, Y. Y.

Y. Y. Schechner, J. Shamir, and N. Kiryati, “Polarization and statistical analysis of scenes containing a semireflector,” J. Opt. Soc. Am. A 17, 276–284 (2000).
[Crossref]

Y. Diamant and Y. Y. Schechner, “Overcoming visual reverberations,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

Shamir, J.

Shaw, J. A.

Shih, Y.

Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.

Shin, J. S.

N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014).
[Crossref]

Sillion, F. X.

X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.

Sonka, M.

M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Cengage Learning, 2014).

Tai, Y. W.

N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014).
[Crossref]

Tang, C.-K.

T.-P. Wu and C.-K. Tang, “Separating specular, diffuse, and subsurface scattering reflectances from photometric images,” in European Conference on Computer Vision (ECCV) (Springer, 2004), pp. 419–433.

Torrance, K. E.

X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.

Tyo, J. S.

B. M. Ratliff and J. S. Tyo, “Moving towards more intuitive display strategies for polarimetric image data,” Proc. SPIE 10407, 104070E (2017).
[Crossref]

J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45, 5453–5469 (2006).
[Crossref]

Wu, T.-P.

T.-P. Wu and C.-K. Tang, “Separating specular, diffuse, and subsurface scattering reflectances from photometric images,” in European Conference on Computer Vision (ECCV) (Springer, 2004), pp. 419–433.

Zeevi, Y. Y.

A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
[Crossref]

Zibulevsky, M.

A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
[Crossref]

Appl. Opt. (1)

IEEE Trans. Patt. Anal. Mach. Intell. (1)

N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014).
[Crossref]

Int. J. Imaging Syst. Technol. (1)

A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005).
[Crossref]

J. Opt. Soc. Am. A (3)

Opt. Express (3)

Proc. SPIE (1)

B. M. Ratliff and J. S. Tyo, “Moving towards more intuitive display strategies for polarimetric image data,” Proc. SPIE 10407, 104070E (2017).
[Crossref]

SIAM J. Sci. Comput. (1)

J. G. Nagy and D. P. O’Leary, “Restoring images degraded by spatially variant blur,” SIAM J. Sci. Comput. 19, 1063–1082 (1998).
[Crossref]

Other (10)

M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Cengage Learning, 2014).

Thorlabs, “CMOS Cameras: USB 2.0 and USB 3.0,” (2017), retrieved https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=4024 .

M. Levoy and P. Hanrahan, “Light field rendering,” in 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42.

Y. Diamant and Y. Y. Schechner, “Overcoming visual reverberations,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.

T.-P. Wu and C.-K. Tang, “Separating specular, diffuse, and subsurface scattering reflectances from photometric images,” in European Conference on Computer Vision (ECCV) (Springer, 2004), pp. 419–433.

B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.

X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.

R. A. Chipman, “Mueller matrices,” in Handbook of Optics (McGraw-Hill, 2010), Vol. 1.

E. Hecht, Optics (Pearson Education, 2016).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (5)

Fig. 1.
Fig. 1. Schematic of the scene is shown. Light emitted from a display device (image of a camera man) propagates to the reflector (gray) at an incident angle of ϕ , is reflected by the reflector, passes through the polarization analyzer (blue), and is detected by an imaging sensor. The imaging lens is not shown. Local coordinates of the display device in x , y are rotated by an angle of χ into the incident local coordinates s in , p in on the reflector, and the exit local coordinates s out , p out on the reflector are rotated by an angle of θ into the local coordinates m , n of the sensor. The horizontal direction of the polarization analyzer (in yellow) aligns with m .
Fig. 2.
Fig. 2. Reflection of the monitor screen from a marble tile is shown. (a) Overlap of the image of the screen contents and the tile is denoted by a red rectangle. (b) Separated screen contents and (c) separated tile texture are shown. (d) Reflection-removed image emphasizing the continuity of irradiance over the boundary of the overlapping region is also shown.
Fig. 3.
Fig. 3. Reflection of the laptop screen (displaying ISO 12233 test chart) from a glass-covered picture is shown. (a) Overlap of the image of the screen contents and the picture is denoted by a red rectangle. (b) Separated screen contents and (c) separated picture are shown. (d) Reflection-removed image showing discontinuity of irradiance over the boundary of the overlapping region is also shown.
Fig. 4.
Fig. 4. Reflection of a cell phone screen (displaying a login screen with the username and the password) from a wood sample (overlapping region shown in red rectangle) is shown. (a) Image of the screen contents is reflected from the wood surface. (b) PSF is measured with a mirror as the reflection surface. (c) PSF is measured with the wood surface as the reflection surface. (d) MAD plot is shown comparing the overlapping/separated images before and after Wiener filtering with the mirror-reflected image. (e) Overlapping unfiltered image, (f) overlapping filtered image, (g) separated unfiltered image, (h) separated filtered image, and (i) mirror-reflected image are shown. Images (e)–(i) are rotated and flipped for a better view of the screen contents. (j) Separated wood surface texture image is shown. (k) Reflection-removed image shows a discontinuity of irradiance and color over the boundary of the overlapping region.
Fig. 5.
Fig. 5. Schematic shows the ghost image from multiple reflections. The incident beam splits into two beams on the air–coating interface. Beam 0 reflects off the interface toward the camera. Beam 1 transmits through the interface, reflects off the coating–substrate interface, and transmits toward the camera. Only the first-order ghost reflection (beam 1) is shown. Beams 0 and 1 are separated spatially by d at the air–coating interface. ϕ is angle of incidence, ϕ is angle of refraction, and t is the coating thickness.

Equations (15)

Equations on this page are rendered with MathJax. Learn more.

E src = λ 1 λ 2 A ( λ ) e i ω t R 45 ° · ( e i π · OPD QWP λ 0 0 e i π · OPD QWP λ ) · R 45 ° · ( 1 0 ) d λ = λ 1 λ 2 A ( λ ) e i ω t cos ( π · OPD QWP λ ) ( 1 i tan ( π · OPD QWP λ ) ) d λ ,
S = ( | E x | 2 + | E y | 2 | E x | 2 | E y | 2 2 Re ( E x E y * ) 2 Im ( E x E y * ) ) ,
S src = ( S 0 400    nm 700    nm A 2 ( λ ) cos ( π λ 0 2 λ ) d λ 0 400    nm 700    nm A 2 ( λ ) sin ( π λ 0 2 λ ) d λ ) ,
S cam = R θ · ( R ( ϕ ) · R χ · S src + T ( ϕ ) · S amb ) ,
R Θ = ( 1 0 0 0 0 cos ( 2 Θ ) sin ( 2 Θ ) 0 0 sin ( 2 Θ ) cos ( 2 Θ ) 0 0 0 0 1 ) ,
R ( ϕ ) = 1 2 ( R 12 s + R 12 p R 12 s R 12 p 0 0 R 12 s R 12 p R 12 s + R 12 p 0 0 0 0 2 R 12 s R 12 p 0 0 0 0 2 R 12 s R 12 p ) ,
T ( ϕ ) = a ( 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ) ,
T ( ϕ ) = ( 1 0 T P Δ m Δ ) ,
R ( ϕ ) = 1 2 ( R s w + R p w R s w R p w 0 0 R s w R p w R s w + R p w 0 0 0 0 R 33 w 0 0 0 0 R 33 w ) ,
T ( ϕ ) = 1 2 ( T s w + T p w T s w T p w 0 0 T s w T p w T s w + T p w 0 0 0 0 T 33 w 0 0 0 0 T 33 w ) ,
S i j cam = ( ( A 12 B 12 cos 2 χ ) S 0 ; i j + a i j · S amb C 12 sin ( 2 χ ) sin ( 2 θ ) S 0 ; i j + σ 1 cos 2 θ C 12 sin ( 2 χ ) cos ( 2 θ ) S 0 ; i j σ 1 sin 2 θ 0 ) ,
S i j cam = ( ( A 12 + B 12 s 1 cos 2 χ ) S 0 ; i j + a i j · S amb C 12 s 1 sin ( 2 χ ) sin ( 2 θ ) S 0 ; i j + σ 1 cos 2 θ C 12 s 1 sin ( 2 χ ) cos ( 2 θ ) S 0 ; i j σ 1 sin 2 θ C 12 S 0 ; i j ) ,
S i j cam = ( ( σ 2 σ 1 cos 2 χ ) S 0 ; i j + ( A t ; 12 2 + B t ; 12 2 ) S i j amb { [ ( σ 1 σ 2 cos 2 χ ) cos 2 θ + ( C t ; 12 2 + 1 ) C 12 sin 2 χ sin 2 θ ] S 0 ; i j + 2 A t ; 12 B t ; 12 cos ( 2 θ ) S i j amb } { [ ( σ 1 σ 2 cos 2 χ ) sin 2 θ + ( C t ; 12 2 + 1 ) C 12 sin 2 χ cos 2 θ ] S 0 ; i j 2 A t ; 12 B t ; 12 sin ( 2 θ ) S i j amb } 0 ) ,
EO ( ϕ ) = i , j Δ i j [ I Re ( ϕ ) ] · Δ i j [ I Tr / Sc ( ϕ ) ] ,
I filt = IFT [ OTF surf * | OTF surf | 2 + SNR 1 × OTF mirror × FT ( I unfilt ) ]

Metrics