Abstract

Correlation image sensors have recently become popular low-cost devices for time-of-flight, or range cameras. They usually operate under the assumption of a single light path contributing to each pixel. We show that a more thorough analysis of the sensor data from correlation sensors can be used can be used to analyze the light transport in much more complex environments, including applications for imaging through scattering and turbid media. The key of our method is a new convolutional sparse coding approach for recovering transient (light-in-flight) images from correlation image sensors. This approach is enabled by an analysis of sparsity in complex transient images, and the derivation of a new physically-motivated model for transient images with drastically improved sparsity.

© 2014 Optical Society of America

Full Article  |  PDF Article
OSA Recommended Articles
Robust and accurate transient light transport decomposition via convolutional sparse coding

Xuemei Hu, Yue Deng, Xing Lin, Jinli Suo, Qionghai Dai, Christopher Barsi, and Ramesh Raskar
Opt. Lett. 39(11) 3177-3180 (2014)

Compressed imaging by sparse random convolution

Diego Marcos, Theo Lasser, Antonio López, and Aurélien Bourquard
Opt. Express 24(2) 1269-1290 (2016)

Hyperspectral video restoration using optical flow and sparse coding

Ajmal Mian and Richard Hartley
Opt. Express 20(10) 10658-10673 (2012)

References

  • View by:
  • |
  • |
  • |

  1. B. Das, K. Yoo, and R. Alfano, “Ultrafast time-gated imaging in thick tissues: a step toward optical mammography,” Opt. Lett. 18, 1092–1094 (1993).
    [Crossref] [PubMed]
  2. P. Han, G. Cho, and X.-C. Zhang, “Time-domain transillumination of biological tissues with terahertz pulses,” Opt. Lett. 25, 242–244 (2000).
    [Crossref]
  3. J. B. Schmidt, Z. D. Schaefer, T. R. Meyer, S. Roy, S. A. Danczyk, and J. R. Gord, “Ultrafast time-gated ballistic-photon imaging and shadowgraphy in optically dense rocket sprays,” Appl. Opt. 48, B137–B144 (2009).
    [Crossref] [PubMed]
  4. A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in “Computer Vision, 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 159–166.
  5. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
    [Crossref] [PubMed]
  6. R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
    [Crossref]
  7. R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
    [Crossref]
  8. M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
    [Crossref]
  9. F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).
  10. D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).
  11. D. L. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory 52, 1289–1306 (2006).
    [Crossref]
  12. E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm. Pure Appl. Math. 59, 1207–1223 (2006).
    [Crossref]
  13. Y. C. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applications (Cambridge University, 2012).
    [Crossref]
  14. A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).
  15. A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
    [Crossref] [PubMed]
  16. M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).
  17. A. Velten, R. Raskar, and M. Bawendi, “Picosecond camera for time-of-flight imaging,” in “Imaging and Applied Optics,” (Optical Society of America, 2011), p. IMB4.
  18. E. Grushka, “Characterization of exponentially modified gaussian peaks in chromatography,” Anal. Chem. 44, 1733–1738 (1972).
    [Crossref] [PubMed]
  19. P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.
  20. M. S. Lewicki and T. J. Sejnowski, “Coding time-varying signals using sparse, shift-invariant representations,” in “Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II,” (MIT, Cambridge, MA, USA, 1999), pp. 730–736.
  21. M. Mørup, M. N. Schmidt, and L. K. Hansen, “Shift invariant sparse coding of image and music data,” Tech. rep., Technical University of Denmark, Richard Petersens Plads bld. 321, 2800 Kgs. Lyngby, Denmark (2008).
  22. M. D. Zeiler and R. Fergus, “Learning image decompositions with hierarchical sparse coding,” Tech. Rep. TR2010-935, Courant Institute of Mathematical Science, New York University (2010).
  23. M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in “Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on,” (IEEE, 2010), pp. 2528–2535.
  24. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in “Adv. Neural Inf. Process. Syst.”, (2012), pp. 1097–1105.
  25. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
    [Crossref]
  26. H. Bristow, A. Eriksson, and S. Lucey, “Fast convolutional sparse coding,” in “Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on,” (IEEE, 2013), pp. 391–398.
  27. J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).
  28. R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. R. Stat. Soc. Ser. B Stat. Methodol. pp. 267–288 (1996).
  29. J. A. Tropp and A. C. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inform. Theory 53, 4655–4666 (2007).
    [Crossref]
  30. S. Rangan, “Generalized approximate message passing for estimation with random linear mixing,” in “Information Theory Proceedings (ISIT), 2011 IEEE International Symposium on,” (IEEE, 2011), pp. 2168–2172.
  31. F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

2014 (1)

2013 (2)

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).

2012 (1)

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

2011 (1)

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

2010 (1)

M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
[Crossref]

2009 (1)

2007 (1)

J. A. Tropp and A. C. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inform. Theory 53, 4655–4666 (2007).
[Crossref]

2006 (2)

D. L. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory 52, 1289–1306 (2006).
[Crossref]

E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm. Pure Appl. Math. 59, 1207–1223 (2006).
[Crossref]

2001 (1)

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[Crossref]

2000 (1)

1993 (1)

1972 (1)

E. Grushka, “Characterization of exponentially modified gaussian peaks in chromatography,” Anal. Chem. 44, 1733–1738 (1972).
[Crossref] [PubMed]

Alfano, R.

Barsi, C.

A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
[Crossref] [PubMed]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Bawendi, M.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

A. Velten, R. Raskar, and M. Bawendi, “Picosecond camera for time-of-flight imaging,” in “Imaging and Applied Optics,” (Optical Society of America, 2011), p. IMB4.

Bhandari, A.

A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
[Crossref] [PubMed]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Boyd, S.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

Bristow, H.

H. Bristow, A. Eriksson, and S. Lucey, “Fast convolutional sparse coding,” in “Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on,” (IEEE, 2013), pp. 391–398.

Buxbaum, B.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Candes, E. J.

E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm. Pure Appl. Math. 59, 1207–1223 (2006).
[Crossref]

Cho, G.

Chu, E.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

Dai, Q.

J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

Danczyk, S. A.

Das, B.

Datte, P.

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

Davis, J.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in “Computer Vision, 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 159–166.

Donoho, D. L.

D. L. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory 52, 1289–1306 (2006).
[Crossref]

Dorrington, A.

A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
[Crossref] [PubMed]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Eckart, M.

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

Eckstein, J.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

Eldar, Y. C.

Y. C. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applications (Cambridge University, 2012).
[Crossref]

Eriksson, A.

H. Bristow, A. Eriksson, and S. Lucey, “Fast convolutional sparse coding,” in “Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on,” (IEEE, 2013), pp. 391–398.

Feigin, M.

Fergus, R.

M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in “Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on,” (IEEE, 2010), pp. 2528–2535.

M. D. Zeiler and R. Fergus, “Learning image decompositions with hierarchical sparse coding,” Tech. Rep. TR2010-935, Courant Institute of Mathematical Science, New York University (2010).

Fischer, H.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Freedman, D.

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).

Gilbert, A. C.

J. A. Tropp and A. C. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inform. Theory 53, 4655–4666 (2007).
[Crossref]

Gord, J. R.

Gregson, J.

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).

Grushka, E.

E. Grushka, “Characterization of exponentially modified gaussian peaks in chromatography,” Anal. Chem. 44, 1733–1738 (1972).
[Crossref] [PubMed]

Gupta, O.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

Han, P.

Hansen, L. K.

M. Mørup, M. N. Schmidt, and L. K. Hansen, “Shift invariant sparse coding of image and music data,” Tech. rep., Technical University of Denmark, Richard Petersens Plads bld. 321, 2800 Kgs. Lyngby, Denmark (2008).

Heide, F.

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).

F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

Heidrich, W.

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).

F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

Heinol, H.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Hinton, G. E.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in “Adv. Neural Inf. Process. Syst.”, (2012), pp. 1097–1105.

Hullin, M. B.

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).

J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

Hutchison, T.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in “Computer Vision, 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 159–166.

Jackson, M.

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

Kadambi, A.

A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
[Crossref] [PubMed]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Khater, H.

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

Kirmani, A.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in “Computer Vision, 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 159–166.

Klein, R.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Koch, R.

M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
[Crossref]

Kolb, A.

M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
[Crossref]

Krishnan, D.

M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in “Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on,” (IEEE, 2010), pp. 2528–2535.

Krizhevsky, A.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in “Adv. Neural Inf. Process. Syst.”, (2012), pp. 1097–1105.

Krupka, E.

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).

Kutulakos, K. N.

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

Kutyniok, G.

Y. C. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applications (Cambridge University, 2012).
[Crossref]

Lange, R.

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[Crossref]

Leichter, I.

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).

Lewicki, M. S.

M. S. Lewicki and T. J. Sejnowski, “Coding time-varying signals using sparse, shift-invariant representations,” in “Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II,” (MIT, Cambridge, MA, USA, 1999), pp. 730–736.

Lin, J.

J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

Lindner, M.

M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
[Crossref]

Liu, Y.

J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

Lucey, S.

H. Bristow, A. Eriksson, and S. Lucey, “Fast convolutional sparse coding,” in “Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on,” (IEEE, 2013), pp. 391–398.

Manuel, A. M.

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

Meyer, T. R.

Mørup, M.

M. Mørup, M. N. Schmidt, and L. K. Hansen, “Shift invariant sparse coding of image and music data,” Tech. rep., Technical University of Denmark, Richard Petersens Plads bld. 321, 2800 Kgs. Lyngby, Denmark (2008).

Newton, M.

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

O’Toole, M.

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

Olk, J.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Parikh, N.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

Peleato, B.

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

Rangan, S.

S. Rangan, “Generalized approximate message passing for estimation with random linear mixing,” in “Information Theory Proceedings (ISIT), 2011 IEEE International Symposium on,” (IEEE, 2011), pp. 2168–2172.

Raskar, R.

A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
[Crossref] [PubMed]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

A. Velten, R. Raskar, and M. Bawendi, “Picosecond camera for time-of-flight imaging,” in “Imaging and Applied Optics,” (Optical Society of America, 2011), p. IMB4.

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in “Computer Vision, 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 159–166.

Romberg, J. K.

E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm. Pure Appl. Math. 59, 1207–1223 (2006).
[Crossref]

Roy, S.

Schaefer, Z. D.

Schiller, I.

M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
[Crossref]

Schmidt, J. B.

Schmidt, M.

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).

Schmidt, M. N.

M. Mørup, M. N. Schmidt, and L. K. Hansen, “Shift invariant sparse coding of image and music data,” Tech. rep., Technical University of Denmark, Richard Petersens Plads bld. 321, 2800 Kgs. Lyngby, Denmark (2008).

Schulte, J.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Schwarte, R.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Seitz, P.

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[Crossref]

Sejnowski, T. J.

M. S. Lewicki and T. J. Sejnowski, “Coding time-varying signals using sparse, shift-invariant representations,” in “Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II,” (MIT, Cambridge, MA, USA, 1999), pp. 730–736.

Smolin, Y.

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).

Streeter, L.

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Sutskever, I.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in “Adv. Neural Inf. Process. Syst.”, (2012), pp. 1097–1105.

Tao, T.

E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm. Pure Appl. Math. 59, 1207–1223 (2006).
[Crossref]

Taylor, G. W.

M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in “Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on,” (IEEE, 2010), pp. 2528–2535.

Tibshirani, R.

R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. R. Stat. Soc. Ser. B Stat. Methodol. pp. 267–288 (1996).

Tropp, J. A.

J. A. Tropp and A. C. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inform. Theory 53, 4655–4666 (2007).
[Crossref]

Veeraraghavan, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

Velten, A.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

A. Velten, R. Raskar, and M. Bawendi, “Picosecond camera for time-of-flight imaging,” in “Imaging and Applied Optics,” (Optical Society of America, 2011), p. IMB4.

Whyte, R.

A. Bhandari, A. Kadambi, R. Whyte, C. Barsi, M. Feigin, A. Dorrington, and R. Raskar, “Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization,” Opt. Lett. 39, 1705–1708 (2014).
[Crossref] [PubMed]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Willwacher, T.

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

Xiao, L.

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

Xu, Z.

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

Yoo, K.

Zeiler, M. D.

M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in “Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on,” (IEEE, 2010), pp. 2528–2535.

M. D. Zeiler and R. Fergus, “Learning image decompositions with hierarchical sparse coding,” Tech. Rep. TR2010-935, Courant Institute of Mathematical Science, New York University (2010).

Zhang, X.-C.

ACM Trans. Graph. (Proc. SIGGRAPH 2013) (1)

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich, “Low-budget transient imaging using photonic mixer devices,” ACM Trans. Graph. (Proc. SIGGRAPH 2013) 32, 45 (2013).

ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) (1)

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar, “Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles,” ACM Trans. Graph. (Proc. SIGGRAPH Asia 2013) 32, 167 (2013).

Anal. Chem. (1)

E. Grushka, “Characterization of exponentially modified gaussian peaks in chromatography,” Anal. Chem. 44, 1733–1738 (1972).
[Crossref] [PubMed]

Appl. Opt. (1)

Comm. Pure Appl. Math. (1)

E. J. Candes, J. K. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,” Comm. Pure Appl. Math. 59, 1207–1223 (2006).
[Crossref]

Computer Vision and Image Understanding (1)

M. Lindner, I. Schiller, A. Kolb, and R. Koch, “Time-of-flight sensor calibration for accurate range sensing,” Computer Vision and Image Understanding 114, 1318–1328 (2010).
[Crossref]

Foundations and Trends® in Machine Learning (1)

S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine Learning 3, 1–122 (2011).
[Crossref]

IEEE J. Quantum Electron. (1)

R. Lange and P. Seitz, “Solid-state time-of-flight range camera,” IEEE J. Quantum Electron. 37, 390–397 (2001).
[Crossref]

IEEE Trans. Inform. Theory (2)

D. L. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory 52, 1289–1306 (2006).
[Crossref]

J. A. Tropp and A. C. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,” IEEE Trans. Inform. Theory 53, 4655–4666 (2007).
[Crossref]

Nat. Commun. (1)

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3, 745 (2012).
[Crossref] [PubMed]

Opt. Lett. (3)

Other (17)

Y. C. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applications (Cambridge University, 2012).
[Crossref]

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for tof sensors,” arXiv preprint arXiv:1403.5919 (2014).

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte, “New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device,” in “Proc. SPIE,”, vol. 3100 (1997), vol. 3100, pp. 245–253.
[Crossref]

M. O’Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N. Kutulakos, “Temporal frequency probing for 5d transient analysis of global light transport,” ACM Trans. Graph. (Proc. SIGGRAPH)33 (2014).

A. Velten, R. Raskar, and M. Bawendi, “Picosecond camera for time-of-flight imaging,” in “Imaging and Applied Optics,” (Optical Society of America, 2011), p. IMB4.

S. Rangan, “Generalized approximate message passing for estimation with random linear mixing,” in “Information Theory Proceedings (ISIT), 2011 IEEE International Symposium on,” (IEEE, 2011), pp. 2168–2172.

F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin, “Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using transient imaging,” in “Computer Vision, 2009 IEEE 12th International Conference on,” (IEEE, 2009), pp. 159–166.

H. Bristow, A. Eriksson, and S. Lucey, “Fast convolutional sparse coding,” in “Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on,” (IEEE, 2013), pp. 391–398.

J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in “IEEE Conference on Computer Vision and Pattern Recognition (CVPR),” (2014).

R. Tibshirani, “Regression shrinkage and selection via the lasso,” J. R. Stat. Soc. Ser. B Stat. Methodol. pp. 267–288 (1996).

P. Datte, A. M. Manuel, M. Eckart, M. Jackson, H. Khater, and M. Newton, “Evaluating radiation induced noise effects on pixelated sensors for the national ignition facility,” (2013), vol. 8850, pp. 885003.

M. S. Lewicki and T. J. Sejnowski, “Coding time-varying signals using sparse, shift-invariant representations,” in “Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II,” (MIT, Cambridge, MA, USA, 1999), pp. 730–736.

M. Mørup, M. N. Schmidt, and L. K. Hansen, “Shift invariant sparse coding of image and music data,” Tech. rep., Technical University of Denmark, Richard Petersens Plads bld. 321, 2800 Kgs. Lyngby, Denmark (2008).

M. D. Zeiler and R. Fergus, “Learning image decompositions with hierarchical sparse coding,” Tech. Rep. TR2010-935, Courant Institute of Mathematical Science, New York University (2010).

M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in “Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on,” (IEEE, 2010), pp. 2528–2535.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in “Adv. Neural Inf. Process. Syst.”, (2012), pp. 1097–1105.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1
Fig. 1

Example of imaging in scattering media using our approach. Left: Original scene with objects submerged in water-filled glass tank. Center: 160 ml milk added. Right: Objects that are “invisible” due to strong scattering, like the structured object on the top right or parts of the circular object become detectable using our approach.

Fig. 2
Fig. 2

Time-of-flight depth imaging assuming a single light path.

Fig. 3
Fig. 3

Realistic scenes contain multiple light paths between source and sensor, and time-of-flight imaging has to account for them (left). Imaging in scattering media results in a continuum of paths of different lengths, resulting in a loss of sparsity in the time domain.

Fig. 4
Fig. 4

Sparsity of 40K signals of a transient image measured image (left). Sparsity after fitting to the convolutional basis proposed in this paper (right).

Fig. 5
Fig. 5

A few samples of the exponentially modified Gaussian signals located at μ = 50. Curves are normalized by adjusting amplitude a.

Fig. 6
Fig. 6

Example showing the effect of our sparse coding optimization on two pixels from the “Tomato” dataset. From left to right: (a) Proposed new model, (b) FFT and sequential models, (c) state-of-the-art sparse reconstruction (LASSO and OMP), (d) two state-of-the-art compressed sensing models (GAMP).

Fig. 7
Fig. 7

Camera prototype and setup: We use a modified PMDTechnologies CamBoard nano using an array of red laser diodes for illumination (left) as described by Heide et al. [31]. In our setup we image a tank filled with a scattering medium of different concentrations frontal with the cameras. The spatial dimensions and arrangement of the setup is shown in the center. The setup is shown on the right.

Fig. 8
Fig. 8

Qualitative results for the milk experiment described in Section 4.2. Experiments in each row from top down: 0 ml, 10 ml, 20 ml, 40 ml, 80 ml, 160 ml, 300 ml of milk in water. Each row shows the regular camera image (left), peak image for red camera reconstruction (right). Peak images are here a parabola fit through the two nearest neighbor points of the strongest peak, where the position is encoded as as hue and the intensity is encoded as value in the HSV color model.

Fig. 9
Fig. 9

Qualitative results for the plaster experiment described in Section 4.2. Experiments in each row from top down: 0 oz, 2 oz, 4 oz, 8 oz, 16 oz, 32 oz, 59 oz of plaster in water. Each row shows the regular camera image (left), peak image for red camera reconstruction (right). Peak images are here a parabola fit through the two nearest neighbor points of the strongest peak, where the position is encoded as as hue and the intensity is encoded as value in the HSV color model.

Fig. 10
Fig. 10

Error for three specific pixels shown on the left. The error of the position is shown for all 100 experiments using milk as scattering medium and all 50 experiments using plaster as scattering medium. The error is measured with respect to 0% concentration and corrected for the speed of light in water. Standard time-of-flight depth reconstruction (arctan solution) breaks down even at very low concentrations of scattering agent.

Equations (12)

Equations on this page are rendered with MathJax. Learn more.

g ( t ) = k = 0 g k cos ( ω T t ) s ( t ) = α p g ( t τ ) + I f ( t ) = k = 0 f k cos ( ω R t + ϕ ) .
p ( ϕ ) = f s = 𝔉 1 ( ( 𝔉 ( s ) ¯ ( ξ ) 𝔉 ( f ) ( ξ ) ) = α p 2 k f k g k cos ( ϕ ω T τ )
b ω i , ϕ i = 0 α ( τ ) 0 T g ω i ( t τ ) f ω i ( t + ϕ i ) d t = c ω i , ϕ i ( τ ) d τ , with α ( τ ) = 𝒫 α p δ ( | p | = τ ) d p ,
C = ( c ω i , ϕ i ( τ j ) ) i , j ,
b = Ci with i = [ α ( τ 1 ) , α ( τ 2 ) , , α ( τ M ) ] T .
i opt = argmin i i 1 subject to Ci b 2 2 < ε ,
h ( τ ; a , σ , ρ , μ ) = a exp ( 1 2 ( σ ρ ) 2 τ μ ρ ) ( 1 + erf ( ( τ μ ) σ 2 / ρ 2 σ ) ) ,
i ( τ ) = i = 1 n h ( τ ; u i ) ,
H = [ h ( s ; u 0 ) , , h ( s ; u n ) ]
v opt = argmin v 1 2 CHv b 2 2 + λ v 1
v opt = argmin v 1 2 C i = 0 k H i v i b 2 2 + λ v 1 subject to H i h ( s , 𝒞 ) i { 1 , , k }
v opt = argmin v 1 2 C 𝔉 1 ( i = 0 k H ^ i v ^ i ) b 2 2 + λ i = 0 k t i 1 subject to H i h ( s , 𝒞 ) i { 1 , , k } t i = v i i { 1 , , k }

Metrics