Abstract

We present a CMOS computational camera with on-chip compressive sensing technique. Through per-pixel programmable charge modulation, camera exposure is spatial-temporally encoded by a CMOS image sensor without utilization of superfluous optical modulators. Each sensor pixel incorporates a two-tap charge modulator and exposure code memory cells, and a proof-of-concept image sensor (128×128 pixels) is capable of per-frame spatial-temporal coded exposure in either full resolution or designated region of interest. After reconstruction, high-speed videos at various temporal resolutions are recovered, while the prototype camera operates at 10 fps. Comparing to previous works, this camera design provides a power-efficient solution for compressive sensing related applications.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure

Jie Zhang, Tao Xiong, Trac Tran, Sang Chin, and Ralph Etienne-Cummings
Opt. Express 24(8) 9013-9024 (2016)

Multi-exposure laser speckle contrast imaging using a video-rate multi-tap charge modulation image sensor

Panneer Selvam Sivakumar, Keiichiro Kagawa, Christian Crouzet, Bernard Choi, Keita Yasutomi, and Shoji Kawahito
Opt. Express 27(18) 26175-26191 (2019)

Single-event transient imaging with an ultra-high-speed temporally compressive multi-aperture CMOS image sensor

Futa Mochizuki, Keiichiro Kagawa, Shin-ichiro Okihara, Min-Woong Seo, Bo Zhang, Taishi Takasawa, Keita Yasutomi, and Shoji Kawahito
Opt. Express 24(4) 4155-4176 (2016)

References

  • View by:
  • |
  • |
  • |

  1. Z. Wang, L. Spinoulas, K. He, H. Chen, L. Tian, A. K. Katsaggelos, and O. Cossairt, “Compressive Holographic Video,” Opt. Lett. 32(10), 1229–1231 (2007).
    [Crossref]
  2. Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.
  3. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013).
    [Crossref]
  4. R. Koller, L. Schmid, N. Matsuda, T. Niederberger, L. Spinoulas, O. Cossairt, G. Schuster, and A. K. Katsaggelos, “High spatio-temporal resolution video with compressed sensing,” Opt. Express 23(12), 15992–16007 (2015).
    [Crossref]
  5. Q. Zhou, J. Ke, and E. Y. Lam, “Near-Infrared Temporal Compressive Imaging for Video,” Opt. Lett. 44(7), 1702–1705 (2019).
    [Crossref]
  6. T. Portz, L. Zhang, and H. Jiang, “Random Coded Sampling for High-Speed HDR Video,” in IEEE ICCP (IEEE, 2013), pp. 1–8.
  7. Y. Sun, X. Yuan, and S. Pang, “Compressive High-Speed Stereo Imaging,” Opt. Express 25(15), 18182–18190 (2017).
    [Crossref]
  8. F. Li, H. Chen, A. Pediredla, C. Yeh, K. He, A. Veeraraghavan, and O. Cossairt, “CS-ToF: High-resolution compressive time-of-flight imaging,” Opt. Express 25(25), 31096–31110 (2017).
    [Crossref]
  9. D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
    [Crossref]
  10. N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.
  11. M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.
  12. G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
    [Crossref]
  13. F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.
  14. J. Zhang, T. Xiong, T. Tran, S. Chin, and R. Etienne-Cummings, “Compact all-CMOS Sptiotemporal Compressive Sensing Video Camera with Pixel-Wise Coded Exposure,” Opt. Express 24(8), 9013–9024 (2016).
    [Crossref]
  15. Y. Luo and S. Mirabbasi, “A CMOS Pixel Design with Binary Space-Time Exposure Encoding for Computational Imaging,” IEEE CICC (IEEE2017), pp. 1–4.
  16. Y. Luo and S. Mirabbasi, “Always-On CMOS Image Sensor Pixel Design for Pixel-Wise Binary Coded Exposure,” IEEE ISCAS (IEEE2017), pp. 1–4.
  17. Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-Programmable CMOS Pixel with Selective Charge Storage and Code Memory for Computational Imaging,” IEEE Trans. Circuits and Systems I: Regular Papers 65(5), 1555–1566 (2018).
    [Crossref]
  18. D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise,” IEEE Trans. Inf. Theory 52(1), 6–18 (2006).
    [Crossref]
  19. E. J. Candes, J. Romberg, and T. Tao, “Stable Signal Recovery from Incomplete and Inaccurate Measurements,” Comm. Pure Appl. Math. 59(8), 1207–1223 (2006).
    [Crossref]
  20. M. Elad and M. Aharon, “Image Denoising via Learned Dictionaries and Sparse Representation,” IEEE CVPR (IEEE2006), pp. 895–900.
  21. M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.
  22. M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Processing 54(11), 4311–4322 (2006).
    [Crossref]
  23. A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos,” IEEE Trans. Pattern Anal. Mach. Intell 33(4), 671–686 (2011).
    [Crossref]

2019 (1)

2018 (1)

Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-Programmable CMOS Pixel with Selective Charge Storage and Code Memory for Computational Imaging,” IEEE Trans. Circuits and Systems I: Regular Papers 65(5), 1555–1566 (2018).
[Crossref]

2017 (2)

2016 (1)

2015 (1)

2014 (1)

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

2013 (1)

2012 (1)

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

2011 (1)

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos,” IEEE Trans. Pattern Anal. Mach. Intell 33(4), 671–686 (2011).
[Crossref]

2007 (1)

2006 (3)

D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise,” IEEE Trans. Inf. Theory 52(1), 6–18 (2006).
[Crossref]

E. J. Candes, J. Romberg, and T. Tao, “Stable Signal Recovery from Incomplete and Inaccurate Measurements,” Comm. Pure Appl. Math. 59(8), 1207–1223 (2006).
[Crossref]

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Processing 54(11), 4311–4322 (2006).
[Crossref]

Agranov, G.

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

Aharon, M.

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Processing 54(11), 4311–4322 (2006).
[Crossref]

M. Elad and M. Aharon, “Image Denoising via Learned Dictionaries and Sparse Representation,” IEEE CVPR (IEEE2006), pp. 895–900.

Baraniuk, R.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Baron, D.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Brady, D. J.

Bruckstein, A.

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Processing 54(11), 4311–4322 (2006).
[Crossref]

Candes, E. J.

E. J. Candes, J. Romberg, and T. Tao, “Stable Signal Recovery from Incomplete and Inaccurate Measurements,” Comm. Pure Appl. Math. 59(8), 1207–1223 (2006).
[Crossref]

Carin, L.

Chen, H.

Chin, S.

Cossairt, O.

Donoho, D. L.

D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise,” IEEE Trans. Inf. Theory 52(1), 6–18 (2006).
[Crossref]

Duarte, M.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Elad, M.

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Processing 54(11), 4311–4322 (2006).
[Crossref]

D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise,” IEEE Trans. Inf. Theory 52(1), 6–18 (2006).
[Crossref]

M. Elad and M. Aharon, “Image Denoising via Learned Dictionaries and Sparse Representation,” IEEE CVPR (IEEE2006), pp. 895–900.

Etienne-Cummings, R.

Genov, R.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

Gu, J.

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.

Gupta, M.

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.

Gusev, N.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

He, K.

Hitomi, Y.

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.

Ho, D.

Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-Programmable CMOS Pixel with Selective Charge Storage and Code Memory for Computational Imaging,” IEEE Trans. Circuits and Systems I: Regular Papers 65(5), 1555–1566 (2018).
[Crossref]

Horowitz, M.

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

Jiang, H.

T. Portz, L. Zhang, and H. Jiang, “Random Coded Sampling for High-Speed HDR Video,” in IEEE ICCP (IEEE, 2013), pp. 1–8.

Kagawa, K.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

Katic, N.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

Katsaggelos, A. K.

Ke, H.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

Ke, J.

Kelly, K.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Kittle, D.

Koller, R.

Kutulakos, K.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

Kutulakos, K. N.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

Lam, E. Y.

Laska, J.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Lee, H.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

Levoy, M.

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

Li, F.

Li, X.

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

Liao, X.

Liu, D.

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

Llull, P.

Luo, Y.

Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-Programmable CMOS Pixel with Selective Charge Storage and Code Memory for Computational Imaging,” IEEE Trans. Circuits and Systems I: Regular Papers 65(5), 1555–1566 (2018).
[Crossref]

Y. Luo and S. Mirabbasi, “Always-On CMOS Image Sensor Pixel Design for Pixel-Wise Binary Coded Exposure,” IEEE ISCAS (IEEE2017), pp. 1–4.

Y. Luo and S. Mirabbasi, “A CMOS Pixel Design with Binary Space-Time Exposure Encoding for Computational Imaging,” IEEE CICC (IEEE2017), pp. 1–4.

Matsuda, N.

Mirabbasi, S.

Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-Programmable CMOS Pixel with Selective Charge Storage and Code Memory for Computational Imaging,” IEEE Trans. Circuits and Systems I: Regular Papers 65(5), 1555–1566 (2018).
[Crossref]

Y. Luo and S. Mirabbasi, “Always-On CMOS Image Sensor Pixel Design for Pixel-Wise Binary Coded Exposure,” IEEE ISCAS (IEEE2017), pp. 1–4.

Y. Luo and S. Mirabbasi, “A CMOS Pixel Design with Binary Space-Time Exposure Encoding for Computational Imaging,” IEEE CICC (IEEE2017), pp. 1–4.

Mitsunaga, T.

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.

Mochizuki, F.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

Nayar, S. K.

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.

Niederberger, T.

O’Toole, M.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

Okihara, S.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

Pang, S.

Pediredla, A.

Portz, T.

T. Portz, L. Zhang, and H. Jiang, “Random Coded Sampling for High-Speed HDR Video,” in IEEE ICCP (IEEE, 2013), pp. 1–8.

Raskar, R.

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos,” IEEE Trans. Pattern Anal. Mach. Intell 33(4), 671–686 (2011).
[Crossref]

Reddy, D.

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos,” IEEE Trans. Pattern Anal. Mach. Intell 33(4), 671–686 (2011).
[Crossref]

Romberg, J.

E. J. Candes, J. Romberg, and T. Tao, “Stable Signal Recovery from Incomplete and Inaccurate Measurements,” Comm. Pure Appl. Math. 59(8), 1207–1223 (2006).
[Crossref]

Sapiro, G.

Sarhangnejad, N.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

Sarvotham, S.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Schmid, L.

Schuster, G.

Seo, M. W.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

Spinoulas, L.

Sun, Y.

Takasawa, T.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

Takhar, D.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Tao, T.

E. J. Candes, J. Romberg, and T. Tao, “Stable Signal Recovery from Incomplete and Inaccurate Measurements,” Comm. Pure Appl. Math. 59(8), 1207–1223 (2006).
[Crossref]

Temlyakov, V. N.

D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise,” IEEE Trans. Inf. Theory 52(1), 6–18 (2006).
[Crossref]

Tian, L.

Tran, T.

Veeraraghavan, A.

F. Li, H. Chen, A. Pediredla, C. Yeh, K. He, A. Veeraraghavan, and O. Cossairt, “CS-ToF: High-resolution compressive time-of-flight imaging,” Opt. Express 25(25), 31096–31110 (2017).
[Crossref]

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos,” IEEE Trans. Pattern Anal. Mach. Intell 33(4), 671–686 (2011).
[Crossref]

Wakin, M.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Wan, G.

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

Wang, Z.

Wei, M.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

Xia, Z.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

Xiong, T.

Yang, J.

Yeh, C.

Yuan, X.

Zhang, B.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

Zhang, J.

Zhang, L.

T. Portz, L. Zhang, and H. Jiang, “Random Coded Sampling for High-Speed HDR Video,” in IEEE ICCP (IEEE, 2013), pp. 1–8.

Zhou, Q.

Comm. Pure Appl. Math. (1)

E. J. Candes, J. Romberg, and T. Tao, “Stable Signal Recovery from Incomplete and Inaccurate Measurements,” Comm. Pure Appl. Math. 59(8), 1207–1223 (2006).
[Crossref]

IEEE J. Solid-State Circuits (1)

G. Wan, X. Li, G. Agranov, M. Levoy, and M. Horowitz, “CMOS Image Sensor with Multi-Bucket Pixels for Computational Photography,” IEEE J. Solid-State Circuits 47(4), 1031–1042 (2012).
[Crossref]

IEEE Trans. Circuits and Systems I: Regular Papers (1)

Y. Luo, D. Ho, and S. Mirabbasi, “Exposure-Programmable CMOS Pixel with Selective Charge Storage and Code Memory for Computational Imaging,” IEEE Trans. Circuits and Systems I: Regular Papers 65(5), 1555–1566 (2018).
[Crossref]

IEEE Trans. Inf. Theory (1)

D. L. Donoho, M. Elad, and V. N. Temlyakov, “Stable Recovery of Sparse Overcomplete Representations in the Presence of Noise,” IEEE Trans. Inf. Theory 52(1), 6–18 (2006).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell (1)

A. Veeraraghavan, D. Reddy, and R. Raskar, “Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos,” IEEE Trans. Pattern Anal. Mach. Intell 33(4), 671–686 (2011).
[Crossref]

IEEE Trans. Pattern Anal. Mach. Intell. (1)

D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Efficient space-time sampling with pixel-wise coded exposure for high-speed imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 248–260 (2014).
[Crossref]

IEEE Trans. Signal Processing (1)

M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Trans. Signal Processing 54(11), 4311–4322 (2006).
[Crossref]

Opt. Express (5)

Opt. Lett. (2)

Other (9)

T. Portz, L. Zhang, and H. Jiang, “Random Coded Sampling for High-Speed HDR Video,” in IEEE ICCP (IEEE, 2013), pp. 1–8.

Y. Hitomi, J. Gu, M. Gupta, T. Mitsunaga, and S. K. Nayar, “Video from a Single Coded Exposure Photograph using a Learned Over-Complete Dictionary,” in IEEE ICCV (IEEE, 2011), pp.287–294.

Y. Luo and S. Mirabbasi, “A CMOS Pixel Design with Binary Space-Time Exposure Encoding for Computational Imaging,” IEEE CICC (IEEE2017), pp. 1–4.

Y. Luo and S. Mirabbasi, “Always-On CMOS Image Sensor Pixel Design for Pixel-Wise Binary Coded Exposure,” IEEE ISCAS (IEEE2017), pp. 1–4.

N. Sarhangnejad, H. Lee, N. Katic, M. O’Toole, K. Kutulakos, and R. Genov, “CMOS Image Sensor Architecture for Primal-Dual Coding,” in IISW (IISS, 2017), pp. 356–359.

M. Wei, N. Sarhangnejad, Z. Xia, H. Ke, N. Gusev, R. Genov, and K. N. Kutulakos, “Coded Two-Bucket Cameras for Computer Vision,” ECCV (Springer, 2018), pp.1–8.

F. Mochizuki, K. Kagawa, S. Okihara, M. W. Seo, B. Zhang, and T. Takasawa, “Single-Shot 200Mfps 5×3-Aperture Compressive CMOS Imager,” in IEEE ISSCC (IEEE2015), pp. 116–118.

M. Elad and M. Aharon, “Image Denoising via Learned Dictionaries and Sparse Representation,” IEEE CVPR (IEEE2006), pp. 895–900.

M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, “Compressive Imaging for Video Representation and Coding,” in IEEE PCS (IEEE2006), pp. 1–7.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (10)

Fig. 1.
Fig. 1. Conceptual diagram of camera compressive sensing using spatial-temporal coded exposure.
Fig. 2.
Fig. 2. (a) The block diagram of proposed exposure-programmable pixel. (b) The time diagram of pixel coded exposure in a frame period.
Fig. 3.
Fig. 3. (a) Detailed pixel circuitry. (b) The block diagram of chip architecture.
Fig. 4.
Fig. 4. (a) Chip micrograph. (b) Fabricated CMOS image sensor. (c) Prototype computational camera system.
Fig. 5.
Fig. 5. Camera output images of (a) a non-intermittent exposure test, (b) a spatial-temporal coded exposure test using 128 column-grey-scale masks.
Fig. 6.
Fig. 6. Camera compressive sensing by single-shot spatial-temporal coded exposure. Comparing to a blurry image generated from non-intermittent exposure, a space-time volume of images is reconstructed from a captured coded image.
Fig. 7.
Fig. 7. High frame rate video synthesis using per-frame coded exposure. The prototype CS camera operates at steady frame rate while each output coded frame reconstructs a space-time volume as a part of the final high-speed video.
Fig. 8.
Fig. 8. High-speed video synthesis using region of interest (ROI) based per-frame coded exposure. The size of ROI is 70 pixels by 70 pixels.
Fig. 9.
Fig. 9. A comparison between high-speed videos generated by (a) non-intermittent exposure and (b) per-frame spatial-temporal coded exposure.
Fig. 10.
Fig. 10. (a) Power consumption of (a) the pixel array and peripheral modules during coded exposure. (b) The overall power consumption of the image sensor in a single shot.

Tables (1)

Tables Icon

Table 1. Performance Comparison to Related Works

Equations (7)

Equations on this page are rendered with MathJax. Learn more.

I ( y ) = N F ( y , n ) M ( y , n ) d n
F ( y , n ) = D α = α 1 D 1 + α 2 D 2 + α 3 D 3 + + α k D k
F ¯ ( y ) = arg min F ( y ) ( E d ( F ( y ) ) + β E r ( F ( y ) ) )
E d ( F ( y ) ) = | | I ( y ) I ( y ) | | 2 2
E r ( F ( y ) ) = N D | | α | | 1 d n
S N R g a i n = S N R C S S N R C o n v e n t i o n a l = ( N × P × F ) / ( λ × η s e n s o r ) P / η s e n s o r = N × F λ
P f r a m e = P r s t T r s t + X Y ( P c o d e + P d r a m ) N T c h o p + P r e a d T r e a d T

Metrics