Abstract

A profile preferentially partial occlusion removal method for integral imaging is presented. The profile of the occlusion always contains details with significant texture structure, and regions with significant texture structure often lead to reliable depth estimation. Taking the advantage of the significant texture structure, the profile of occlusion is preferentially dealt with, and then the entire occlusion region is determined via regional spreading according to the accurate profile. The details of occlusion can be accurately removed and the occluded scene is also retained to the maximum degree. In our method, elemental images are integrated into a four-dimensional light field to provide consistently reliable depth estimation and occlusion decisions among all elemental images. Experimental results show that the proposed method is efficient to deal with the details of the occlusion, and it is robust for the occlusions with different kinds of texture structure.

© 2016 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Three-dimensional profilometric reconstruction using flexible sensing integral imaging and occlusion removal

Xin Shen, Adam Markman, and Bahram Javidi
Appl. Opt. 56(9) D151-D157 (2017)

Occlusion removal method of partially occluded 3D object using sub-image block matching in computational integral imaging

Dong-Hak Shin, Byung-Gook Lee, and Joon-Jae Lee
Opt. Express 16(21) 16294-16304 (2008)

Mutual constraint using partial occlusion artifact removal for computational integral imaging reconstruction

Taekyung Ryu, Byunggook Lee, and Sukho Lee
Appl. Opt. 54(13) 4147-4153 (2015)

References

  • View by:
  • |
  • |
  • |

  1. S. Adrian and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
    [Crossref]
  2. J.-H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48(34), H77–H94 (2009).
    [Crossref] [PubMed]
  3. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [Invited],” Appl. Opt. 52(4), 546–560 (2013).
    [Crossref] [PubMed]
  4. B. Javidi, R. Ponce-Díaz, and S. H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. 31(8), 1106–1108 (2006).
    [Crossref] [PubMed]
  5. S.-H. Hong and B. Javidi, “Three-dimensional visualization of partially occluded objects using integral imaging,” J. Disp. Technol. 1(2), 354–359 (2005).
    [Crossref]
  6. H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. 26(3), 157–159 (2001).
    [Crossref] [PubMed]
  7. S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004).
    [Crossref] [PubMed]
  8. G. Saavedra, R. Martínez-Cuenca, M. Martínez-Corral, H. Navarro, M. Daneshpanah, and B. Javidi, “Digital slicing of 3D scenes by Fourier filtering of integral images,” Opt. Express 16(22), 17154–17160 (2008).
    [Crossref] [PubMed]
  9. J. H. Park and K. M. Jeong, “Frequency domain depth filtering of integral imaging,” Opt. Express 19(19), 18729–18741 (2011).
    [Crossref] [PubMed]
  10. J. Y. Jang, D. Shin, and E. S. Kim, “Optical three-dimensional refocusing from elemental images based on a sifting property of the periodic δ-function array in integral-imaging,” Opt. Express 22(2), 1533–1550 (2014).
    [Crossref] [PubMed]
  11. D. H. Shin, B. G. Lee, and J. J. Lee, “Occlusion removal method of partially occluded 3D object using sub-image block matching in computational integral imaging,” Opt. Express 16(21), 16294–16304 (2008).
    [Crossref] [PubMed]
  12. R. Taekyung, B. Lee, and S. Lee, “Mutual constraint using partial occlusion artifact removal for computational integral imaging reconstruction,” Appl. Opt. 54(13), 4147–4153 (2015).
    [Crossref]
  13. H. Yoo, “Depth extraction for 3D objects via windowing technique in computational integral imaging with a lenslet array,” Opt. Lasers Eng. 51(7), 912–915 (2013).
    [Crossref]
  14. H. Yoo, D. Shin, and M. Cho, “Improved depth extraction method of 3D objects using computational integral imaging reconstruction based on multiple windowing techniques,” Opt. Lasers Eng. 66, 105–111 (2015).
    [Crossref]
  15. B. G. Lee, H. H. Kang, and E. S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research. 1(2), 2–10 (2010).
    [Crossref]
  16. X. Xiao, M. Daneshpanah, and B. Javidi, “Occlusion removal using depth mapping in three-dimensional integral imaging,” J. Disp. Technol. 8(8), 483–490 (2012).
    [Crossref]
  17. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” ACM Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 43–54 (1996).
    [Crossref]
  18. E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology (1991).
  19. F. Meyer, “Color image segmentation,” International Conference on IET in Image Processing and its Applications, 303–306 (1992).
  20. Q. Yang, “Stereo matching using tree filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 37(4), 834–846 (2015).
    [Crossref] [PubMed]

2015 (3)

R. Taekyung, B. Lee, and S. Lee, “Mutual constraint using partial occlusion artifact removal for computational integral imaging reconstruction,” Appl. Opt. 54(13), 4147–4153 (2015).
[Crossref]

H. Yoo, D. Shin, and M. Cho, “Improved depth extraction method of 3D objects using computational integral imaging reconstruction based on multiple windowing techniques,” Opt. Lasers Eng. 66, 105–111 (2015).
[Crossref]

Q. Yang, “Stereo matching using tree filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 37(4), 834–846 (2015).
[Crossref] [PubMed]

2014 (1)

2013 (2)

H. Yoo, “Depth extraction for 3D objects via windowing technique in computational integral imaging with a lenslet array,” Opt. Lasers Eng. 51(7), 912–915 (2013).
[Crossref]

X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [Invited],” Appl. Opt. 52(4), 546–560 (2013).
[Crossref] [PubMed]

2012 (1)

X. Xiao, M. Daneshpanah, and B. Javidi, “Occlusion removal using depth mapping in three-dimensional integral imaging,” J. Disp. Technol. 8(8), 483–490 (2012).
[Crossref]

2011 (1)

2010 (1)

B. G. Lee, H. H. Kang, and E. S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research. 1(2), 2–10 (2010).
[Crossref]

2009 (1)

2008 (2)

2006 (2)

S. Adrian and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[Crossref]

B. Javidi, R. Ponce-Díaz, and S. H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. 31(8), 1106–1108 (2006).
[Crossref] [PubMed]

2005 (1)

S.-H. Hong and B. Javidi, “Three-dimensional visualization of partially occluded objects using integral imaging,” J. Disp. Technol. 1(2), 354–359 (2005).
[Crossref]

2004 (1)

2001 (1)

Adrian, S.

S. Adrian and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[Crossref]

Arimoto, H.

Cho, M.

H. Yoo, D. Shin, and M. Cho, “Improved depth extraction method of 3D objects using computational integral imaging reconstruction based on multiple windowing techniques,” Opt. Lasers Eng. 66, 105–111 (2015).
[Crossref]

Daneshpanah, M.

X. Xiao, M. Daneshpanah, and B. Javidi, “Occlusion removal using depth mapping in three-dimensional integral imaging,” J. Disp. Technol. 8(8), 483–490 (2012).
[Crossref]

G. Saavedra, R. Martínez-Cuenca, M. Martínez-Corral, H. Navarro, M. Daneshpanah, and B. Javidi, “Digital slicing of 3D scenes by Fourier filtering of integral images,” Opt. Express 16(22), 17154–17160 (2008).
[Crossref] [PubMed]

Hong, K.

Hong, S. H.

Hong, S.-H.

S.-H. Hong and B. Javidi, “Three-dimensional visualization of partially occluded objects using integral imaging,” J. Disp. Technol. 1(2), 354–359 (2005).
[Crossref]

S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004).
[Crossref] [PubMed]

Jang, J. Y.

Jang, J.-S.

Javidi, B.

X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [Invited],” Appl. Opt. 52(4), 546–560 (2013).
[Crossref] [PubMed]

X. Xiao, M. Daneshpanah, and B. Javidi, “Occlusion removal using depth mapping in three-dimensional integral imaging,” J. Disp. Technol. 8(8), 483–490 (2012).
[Crossref]

G. Saavedra, R. Martínez-Cuenca, M. Martínez-Corral, H. Navarro, M. Daneshpanah, and B. Javidi, “Digital slicing of 3D scenes by Fourier filtering of integral images,” Opt. Express 16(22), 17154–17160 (2008).
[Crossref] [PubMed]

B. Javidi, R. Ponce-Díaz, and S. H. Hong, “Three-dimensional recognition of occluded objects by using computational integral imaging,” Opt. Lett. 31(8), 1106–1108 (2006).
[Crossref] [PubMed]

S. Adrian and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[Crossref]

S.-H. Hong and B. Javidi, “Three-dimensional visualization of partially occluded objects using integral imaging,” J. Disp. Technol. 1(2), 354–359 (2005).
[Crossref]

S.-H. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004).
[Crossref] [PubMed]

H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. 26(3), 157–159 (2001).
[Crossref] [PubMed]

Jeong, K. M.

Kang, H. H.

B. G. Lee, H. H. Kang, and E. S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research. 1(2), 2–10 (2010).
[Crossref]

Kim, E. S.

J. Y. Jang, D. Shin, and E. S. Kim, “Optical three-dimensional refocusing from elemental images based on a sifting property of the periodic δ-function array in integral-imaging,” Opt. Express 22(2), 1533–1550 (2014).
[Crossref] [PubMed]

B. G. Lee, H. H. Kang, and E. S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research. 1(2), 2–10 (2010).
[Crossref]

Lee, B.

Lee, B. G.

B. G. Lee, H. H. Kang, and E. S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research. 1(2), 2–10 (2010).
[Crossref]

D. H. Shin, B. G. Lee, and J. J. Lee, “Occlusion removal method of partially occluded 3D object using sub-image block matching in computational integral imaging,” Opt. Express 16(21), 16294–16304 (2008).
[Crossref] [PubMed]

Lee, J. J.

Lee, S.

Martinez-Corral, M.

Martínez-Corral, M.

Martínez-Cuenca, R.

Navarro, H.

Park, J. H.

Park, J.-H.

Ponce-Díaz, R.

Saavedra, G.

Shin, D.

H. Yoo, D. Shin, and M. Cho, “Improved depth extraction method of 3D objects using computational integral imaging reconstruction based on multiple windowing techniques,” Opt. Lasers Eng. 66, 105–111 (2015).
[Crossref]

J. Y. Jang, D. Shin, and E. S. Kim, “Optical three-dimensional refocusing from elemental images based on a sifting property of the periodic δ-function array in integral-imaging,” Opt. Express 22(2), 1533–1550 (2014).
[Crossref] [PubMed]

Shin, D. H.

Stern, A.

Taekyung, R.

Xiao, X.

X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [Invited],” Appl. Opt. 52(4), 546–560 (2013).
[Crossref] [PubMed]

X. Xiao, M. Daneshpanah, and B. Javidi, “Occlusion removal using depth mapping in three-dimensional integral imaging,” J. Disp. Technol. 8(8), 483–490 (2012).
[Crossref]

Yang, Q.

Q. Yang, “Stereo matching using tree filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 37(4), 834–846 (2015).
[Crossref] [PubMed]

Yoo, H.

H. Yoo, D. Shin, and M. Cho, “Improved depth extraction method of 3D objects using computational integral imaging reconstruction based on multiple windowing techniques,” Opt. Lasers Eng. 66, 105–111 (2015).
[Crossref]

H. Yoo, “Depth extraction for 3D objects via windowing technique in computational integral imaging with a lenslet array,” Opt. Lasers Eng. 51(7), 912–915 (2013).
[Crossref]

3D Research. (1)

B. G. Lee, H. H. Kang, and E. S. Kim, “Occlusion removal method of partially occluded object using variance in computational integral imaging,” 3D Research. 1(2), 2–10 (2010).
[Crossref]

Appl. Opt. (3)

IEEE Trans. Pattern Anal. Mach. Intell. (1)

Q. Yang, “Stereo matching using tree filtering,” IEEE Trans. Pattern Anal. Mach. Intell. 37(4), 834–846 (2015).
[Crossref] [PubMed]

J. Disp. Technol. (2)

X. Xiao, M. Daneshpanah, and B. Javidi, “Occlusion removal using depth mapping in three-dimensional integral imaging,” J. Disp. Technol. 8(8), 483–490 (2012).
[Crossref]

S.-H. Hong and B. Javidi, “Three-dimensional visualization of partially occluded objects using integral imaging,” J. Disp. Technol. 1(2), 354–359 (2005).
[Crossref]

Opt. Express (5)

Opt. Lasers Eng. (2)

H. Yoo, “Depth extraction for 3D objects via windowing technique in computational integral imaging with a lenslet array,” Opt. Lasers Eng. 51(7), 912–915 (2013).
[Crossref]

H. Yoo, D. Shin, and M. Cho, “Improved depth extraction method of 3D objects using computational integral imaging reconstruction based on multiple windowing techniques,” Opt. Lasers Eng. 66, 105–111 (2015).
[Crossref]

Opt. Lett. (2)

Proc. IEEE (1)

S. Adrian and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006).
[Crossref]

Other (3)

S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” ACM Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 43–54 (1996).
[Crossref]

E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology (1991).

F. Meyer, “Color image segmentation,” International Conference on IET in Image Processing and its Applications, 303–306 (1992).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1 Relationship between integral imaging and the light field, (a) 2D pinhole array for integral imaging, (b) the corresponding light field.
Fig. 2
Fig. 2 Luminance consistent score distributions at different regions.
Fig. 3
Fig. 3 (a) The center image, (b) significant texture structure map, (c) depth map of vertical significant texture structure, (d) depth map of horizontal significant texture structure.
Fig. 4
Fig. 4 (a) local depth map in the 2D light field, (b) depth map completion after linear regression.
Fig. 5
Fig. 5 (a) Region spreading in Ι ( x , m ) , (b) region spreading in Ι ( y , n ) , (c) occlusion decision without region spreading in Ι ( x , y ) , (d) final occlusion decision after region spreading in Ι ( x , y ) .
Fig. 6
Fig. 6 (a) Occlusion with complex texture structure, (b) occlusion with smooth texture structure.
Fig. 7
Fig. 7 The occlusion decision.
Fig. 8
Fig. 8 (a) and (b) Original elemental images, (c) and (d) occlusion decision with the block match and the global regularization, (e) and (f) occlusion decision with variance based method, (g) and (h) occlusion decision with the proposed method.
Fig. 9
Fig. 9 The optical reconstructions compare with the proposed method, (a) the elemental image array, (b) the structure of micro-lens array, (c) optical reconstruction focused at occluded scene, (d) optical reconstruction focused at occlusion, (e) computational reconstruction focused at occluded scene, (f) computational reconstruction focused at occlusion, (g) computational occlusion removal with proposed method focused at occluded scene, (h) computational occlusion removal with proposed method focused at occlusion.
Fig. 10
Fig. 10 Computational reconstruction at the background (d = 1.9 pixels) (a) and (b) computational reconstruction without occlusion removal, (c) and (d) occlusion removal with the block match and global regularization, (e) and (f) occlusion removal with variance based method, (g) and (h) the proposed method.

Equations (9)

Equations on this page are rendered with MathJax. Learn more.

K = Δ n Δ y = s p d = f b z
E ( x , y , m , n , d ) = i = 1 X j = 1 Y Ι [ i , j , m + ( x i ) d , n + ( y j ) d ] · W ( V ( i , j ) ) i = 1 X j = 1 Y W ( V ( i , j ) )
V ( i , j ) = | Ι [ i , j , m + ( x i ) d , n + ( y j ) d ] Ι ( x , y , m , n ) |
C s ( x , y , m , n , d ) = i = 1 X j = 1 Y W ( | Ι [ i , j , m + ( x i ) d , n + ( y j ) d ] E ( x , y , m , n , d ) | )
W ( p ) = { 1 ( p / B ) 2 | p / B | 1 0 e l s e
G h = ( 1 0 + 1 2 0 + 2 1 0 + 1 ) G v = ( 1 2 1 0 0 0 + 1 + 2 + 1 )
m = ( x r x ) d + m r
R ( x , y , m , n , d ) = i = 1 X j = 1 Y Ι [ i , j , m + ( x i ) d , n + ( y j ) d ] X Y
R ( x , y , m , n , d ) = i = 1 X j = 1 Y Ι [ i , j , m + ( x i ) d , n + ( y j ) d ] * T [ i , j , m + ( x i ) d , n + ( y j ) d ] i = 1 X j = 1 Y T [ i , j , m + ( x i ) d , n + ( y j ) d ] T ( x , y , m , n ) = { 0 o c c l u s i o n 1 e l s e

Metrics