Abstract

We show optical waves passing through a nanophotonic medium can perform artificial neural computing. Complex information is encoded in the wavefront of an input light. The medium transforms the wavefront to realize sophisticated computing tasks such as image recognition. At the output, the optical energy is concentrated in well-defined locations, which, for example, can be interpreted as the identity of the object in the image. These computing media can be as small as tens of wavelengths and offer ultra-high computing density. They exploit subwavelength scatterers to realize complex input/output mapping beyond the capabilities of traditional nanophotonic devices.

© 2019 Chinese Laser Press

Full Article  |  PDF Article
OSA Recommended Articles
Artificial neural networks using complex numbers and phase encoded weights

Howard E. Michel and Abdul Ahad S. Awwal
Appl. Opt. 49(10) B71-B82 (2010)

Design and optimization of optical passive elements using artificial neural networks

Ahmed M. Gabr, Chris Featherston, Chao Zhang, Cem Bonfil, Qi-Jun Zhang, and Tom J. Smy
J. Opt. Soc. Am. B 36(4) 999-1007 (2019)

Training of photonic neural networks through in situ backpropagation and gradient measurement

Tyler W. Hughes, Momchil Minkov, Yu Shi, and Shanhui Fan
Optica 5(7) 864-871 (2018)

References

  • View by:
  • |
  • |
  • |

  1. Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
    [Crossref]
  2. M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
    [Crossref]
  3. T. W. Hughes, M. Minkov, Y. Shi, and S. Fan, “Training of photonic neural networks through in situ backpropagation and gradient measurement,” Optica 5, 864–871 (2018).
    [Crossref]
  4. S. R. Skinner, E. C. Behrman, A. A. Cruz-Cabrera, and J. E. Steck, “Neural network implementation using self-lensing media,” Appl. Opt. 34, 4129–4135 (1995).
    [Crossref]
  5. P. R. Prucnal and B. J. Shastri, Neuromorphic Photonics (CRC Press, 2017).
  6. H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.
  7. J. Bueno, S. Maktoobi, L. Froehly, I. Fischer, M. Jacquot, L. Larger, and D. Brunner, “Reinforcement learning in a large-scale photonic recurrent neural network,” Optica 5, 756–760 (2018).
    [Crossref]
  8. X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
    [Crossref]
  9. M. Hermans and T. Van Vaerenbergh, “Towards trainable media: Using waves for neural network-style training,” arXiv:1510.03776 (2015).
  10. H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
    [Crossref]
  11. J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade, Photonic Crystals: Molding the Flow of Light (Princeton University, 2011).
  12. W. Cai and V. Shalaev, Optical Metamaterials: Fundamentals and Applications (Springer, 2009).
  13. B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
    [Crossref]
  14. A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
    [Crossref]
  15. L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
    [Crossref]
  16. V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010), pp. 807–814.
  17. E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).
  18. W. Shin, “MaxwellFDFD,” https://github.com/wsshin/maxwellfdfd .
  19. H. J. Caulfield, J. Kinser, and S. K. Rogers, “Optical neural networks,” Proc. IEEE 77, 1573–1583 (1989).
    [Crossref]
  20. L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.
  21. T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
    [Crossref]
  22. M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).
  23. C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
    [Crossref]
  24. J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res. 13, 281–305 (2012).
  25. J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems (2012), pp. 2951–2959.
  26. S. Saxena and J. Verbeek, “Convolutional neural fabrics,” in Advances in Neural Information Processing Systems (The MIT Press, 2016), pp. 4053–4061.
  27. G. D. Marshall, A. Politi, J. C. Matthews, P. Dekker, M. Ams, M. J. Withford, and J. L. O’Brien, “Laser written waveguide photonic quantum circuits,” Opt. Express 17, 12546–12554 (2009).
    [Crossref]
  28. D. A. Miller, “All linear optical devices are mode converters,” Opt. Express 20, 23985–23993 (2012).
    [Crossref]

2018 (4)

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
[Crossref]

J. Bueno, S. Maktoobi, L. Froehly, I. Fischer, M. Jacquot, L. Larger, and D. Brunner, “Reinforcement learning in a large-scale photonic recurrent neural network,” Optica 5, 756–760 (2018).
[Crossref]

T. W. Hughes, M. Minkov, Y. Shi, and S. Fan, “Training of photonic neural networks through in situ backpropagation and gradient measurement,” Optica 5, 864–871 (2018).
[Crossref]

2017 (3)

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
[Crossref]

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

2015 (2)

B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
[Crossref]

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

2012 (2)

D. A. Miller, “All linear optical devices are mode converters,” Opt. Express 20, 23985–23993 (2012).
[Crossref]

J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res. 13, 281–305 (2012).

2010 (1)

C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
[Crossref]

2009 (1)

2004 (1)

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

1995 (1)

1989 (1)

H. J. Caulfield, J. Kinser, and S. K. Rogers, “Optical neural networks,” Proc. IEEE 77, 1573–1583 (1989).
[Crossref]

Adams, R. P.

J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems (2012), pp. 2951–2959.

Ams, M.

Baehr-Jones, T.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Baek, J.-H.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Behrman, E. C.

Bengio, Y.

J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res. 13, 281–305 (2012).

M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).

Bergstra, J.

J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res. 13, 281–305 (2012).

Bienstman, P.

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

Brunner, D.

Bueno, J.

Burm, M.

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

Cai, W.

W. Cai and V. Shalaev, Optical Metamaterials: Fundamentals and Applications (Springer, 2009).

Caulfield, H. J.

H. J. Caulfield, J. Kinser, and S. K. Rogers, “Optical neural networks,” Proc. IEEE 77, 1573–1583 (1989).
[Crossref]

Chen, A.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

Chen, H. G.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Courbariaux, M.

M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).

Cruz-Cabrera, A. A.

Dambre, J.

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

Dekker, P.

Dubcek, T.

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

El-Yaniv, R.

M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).

Englund, D.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Fan, S.

T. W. Hughes, M. Minkov, Y. Shi, and S. Fan, “Training of photonic neural networks through in situ backpropagation and gradient measurement,” Optica 5, 864–871 (2018).
[Crossref]

T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
[Crossref]

Fischer, I.

Fox, M. D.

C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
[Crossref]

Froehly, L.

Gui, C.

C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
[Crossref]

Harris, N. C.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Hermans, M.

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

M. Hermans and T. Van Vaerenbergh, “Towards trainable media: Using waves for neural network-style training,” arXiv:1510.03776 (2015).

Hinton, G. E.

V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010), pp. 807–814.

Hochberg, M.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Hubara, I.

M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).

Hughes, T. W.

T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
[Crossref]

T. W. Hughes, M. Minkov, Y. Shi, and S. Fan, “Training of photonic neural networks through in situ backpropagation and gradient measurement,” Optica 5, 864–871 (2018).
[Crossref]

Jacquot, M.

Jarrahi, M.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Jayasuriya, S.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Jing, L.

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Joannopoulos, J. D.

J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade, Photonic Crystals: Molding the Flow of Light (Princeton University, 2011).

Johnson, S. G.

J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade, Photonic Crystals: Molding the Flow of Light (Princeton University, 2011).

Ju, Y.-G.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Khoram, E.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

Kim, S.-B.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Kim, S.-H.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Kinser, J.

H. J. Caulfield, J. Kinser, and S. K. Rogers, “Optical neural networks,” Proc. IEEE 77, 1573–1583 (1989).
[Crossref]

Kwon, S.-H.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Larger, L.

Larochelle, H.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems (2012), pp. 2951–2959.

LeCun, Y.

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Lee, Y.-H.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Li, C.

C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
[Crossref]

Lin, X.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Liu, D.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

Luo, Y.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Maktoobi, S.

Marshall, G. D.

Matthews, J. C.

Meade, R. D.

J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade, Photonic Crystals: Molding the Flow of Light (Princeton University, 2011).

Menon, R.

B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
[Crossref]

Miller, D. A.

Minkov, M.

T. W. Hughes, M. Minkov, Y. Shi, and S. Fan, “Training of photonic neural networks through in situ backpropagation and gradient measurement,” Optica 5, 864–871 (2018).
[Crossref]

T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
[Crossref]

Molnar, A.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Nair, V.

V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010), pp. 807–814.

O’Brien, J. L.

Ozcan, A.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Park, H.-G.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Petykiewicz, J.

A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
[Crossref]

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

Peurifoy, J.

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Piggott, A. Y.

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
[Crossref]

Politi, A.

Polson, R.

B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
[Crossref]

Prabhu, M.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Prucnal, P. R.

P. R. Prucnal and B. J. Shastri, Neuromorphic Photonics (CRC Press, 2017).

Rivenson, Y.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Rogers, S. K.

H. J. Caulfield, J. Kinser, and S. K. Rogers, “Optical neural networks,” Proc. IEEE 77, 1573–1583 (1989).
[Crossref]

Sapra, N. V.

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

Saxena, S.

S. Saxena and J. Verbeek, “Convolutional neural fabrics,” in Advances in Neural Information Processing Systems (The MIT Press, 2016), pp. 4053–4061.

Shalaev, V.

W. Cai and V. Shalaev, Optical Metamaterials: Fundamentals and Applications (Springer, 2009).

Shastri, B. J.

P. R. Prucnal and B. J. Shastri, Neuromorphic Photonics (CRC Press, 2017).

Shen, B.

B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
[Crossref]

Shen, Y.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Shi, Y.

Sivaramakrishnan, S.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Skinner, S. R.

Skirlo, S.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Snoek, J.

J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems (2012), pp. 2951–2959.

Soljacic, M.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Soudry, D.

M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).

Steck, J. E.

Stephen, J.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Su, L.

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
[Crossref]

Sun, X.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Tegmark, M.

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

Van Vaerenbergh, T.

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

M. Hermans and T. Van Vaerenbergh, “Towards trainable media: Using waves for neural network-style training,” arXiv:1510.03776 (2015).

Veeraraghavan, A.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Veli, M.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Verbeek, J.

S. Saxena and J. Verbeek, “Convolutional neural fabrics,” in Advances in Neural Information Processing Systems (The MIT Press, 2016), pp. 4053–4061.

Vuckovic, J.

A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
[Crossref]

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

Wang, P.

B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
[Crossref]

Wang, Q.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

Williamson, I. A.

T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
[Crossref]

Winn, J. N.

J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade, Photonic Crystals: Molding the Flow of Light (Princeton University, 2011).

Withford, M. J.

Xu, C.

C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
[Crossref]

Yang, J.

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

Yang, J.-K.

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Yardimci, N. T.

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

Ying, L.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

Yu, Z.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

Zhao, S.

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

ACS Photon. (2)

L. Su, A. Y. Piggott, N. V. Sapra, J. Petykiewicz, and J. Vuckovic, “Inverse design and demonstration of a compact on-chip narrowband three-channel wavelength demultiplexer,” ACS Photon. 5, 301–305 (2017).
[Crossref]

T. W. Hughes, M. Minkov, I. A. Williamson, and S. Fan, “Adjoint method and inverse design for nonlinear nanophotonic devices,” ACS Photon. 5, 4781–4787 (2018).
[Crossref]

Appl. Opt. (1)

IEEE Trans. Image Process. (1)

C. Li, C. Xu, C. Gui, and M. D. Fox, “Distance regularized level-set evolution and its application to image segmentation,” IEEE Trans. Image Process. 19, 1371–1378 (2010).
[Crossref]

J. Mach. Learn. Res. (1)

J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res. 13, 281–305 (2012).

Nat. Commun. (1)

M. Hermans, M. Burm, T. Van Vaerenbergh, J. Dambre, and P. Bienstman, “Trainable hardware for dynamical computing using error backpropagation through physical media,” Nat. Commun. 6, 6729 (2015).
[Crossref]

Nat. Photonics (2)

B. Shen, P. Wang, R. Polson, and R. Menon, “An integrated-nanophotonics polarization beamsplitter with 2.4 × 2.4 μm2 footprint,” Nat. Photonics 9, 378–382 (2015).
[Crossref]

Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, and M. Soljacic, “Deep learning with coherent nanophotonic circuits,” Nat. Photonics 11, 441–446 (2017).
[Crossref]

Opt. Express (2)

Optica (2)

Proc. IEEE (1)

H. J. Caulfield, J. Kinser, and S. K. Rogers, “Optical neural networks,” Proc. IEEE 77, 1573–1583 (1989).
[Crossref]

Sci. Rep. (1)

A. Y. Piggott, J. Petykiewicz, L. Su, and J. Vučković, “Fabrication constrained nanophotonic inverse design,” Sci. Rep. 7, 1786 (2017).
[Crossref]

Science (2)

X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All optical machine learning using diffractive deep neural networks,” Science 361, 1004–1008 (2018).
[Crossref]

H.-G. Park, S.-H. Kim, S.-H. Kwon, Y.-G. Ju, J.-K. Yang, J.-H. Baek, S.-B. Kim, and Y.-H. Lee, “Electrically driven single-cell photonic crystal laser,” Science 305, 1444–1447 (2004).
[Crossref]

Other (12)

J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade, Photonic Crystals: Molding the Flow of Light (Princeton University, 2011).

W. Cai and V. Shalaev, Optical Metamaterials: Fundamentals and Applications (Springer, 2009).

M. Hermans and T. Van Vaerenbergh, “Towards trainable media: Using waves for neural network-style training,” arXiv:1510.03776 (2015).

P. R. Prucnal and B. J. Shastri, Neuromorphic Photonics (CRC Press, 2017).

H. G. Chen, S. Jayasuriya, J. Yang, J. Stephen, S. Sivaramakrishnan, A. Veeraraghavan, and A. Molnar, “ASP vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 903–912.

L. Jing, Y. Shen, T. Dubcek, J. Peurifoy, S. Skirlo, Y. LeCun, M. Tegmark, and M. Soljačić, “Tunable efficient unitary neural networks (EUNN) and their application to RNNs,” in Proceedings of the 34th International Conference on Machine Learning (PMLR) (2017), pp. 1733–1741.

V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010), pp. 807–814.

E. Khoram, A. Chen, D. Liu, Q. Wang, Z. Yu, and L. Ying, “Nanophotonic media for artificial neural inference,” arXiv:1810.07815 (2018).

W. Shin, “MaxwellFDFD,” https://github.com/wsshin/maxwellfdfd .

M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1,” arXiv:1602.02830 (2016).

J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems (2012), pp. 2951–2959.

S. Saxena and J. Verbeek, “Convolutional neural fabrics,” in Advances in Neural Information Processing Systems (The MIT Press, 2016), pp. 4053–4061.

Supplementary Material (2)

NameDescription
» Visualization 1       This video shows the operation of a 2 dimensional implementation of the presented idea in the paper.
» Visualization 2       This video shows the operation of a 3 dimensional implementation of the presented idea in the paper.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (4)

Fig. 1.
Fig. 1. (a) Conventional ANN architecture where the information propagates only in the forward direction (depicted by the green line that goes through the nodes from input to output); (b) proposed NNM. Passive neural computing is performed by light passing through the nanostructured medium with both linear and nonlinear scatterers. (c) Full-wave simulation of light scattered by nanostructures, which spatially redistribute the optical energy in different directions. (d) The behavior of the implementation of such a nonlinear material in one dimension. The output intensity of light with wavelength λ, passing through the designed nonlinear material with a thickness of λ/2. It is a nonlinear function of the incident wave intensity. This material is used as nonlinear activation, as indicated by light blue color.
Fig. 2.
Fig. 2. (a) NNM trained to recognize handwritten digits. The input wave encodes the image as the intensity distribution. On the right side of the NNM, the optical energy concentrates to different locations depending on the image’s classification labels. (b) Two samples of the digit 2 and their optical fields inside the NNM. As can be seen, although the field distributions differ for the images of the same digit, they are classified as the same digit. (c) The same as (b) but for two samples of the digit 8. Also, in both (b) and (c), the boundaries of the trained medium have been shown with black borderlines (see Visualization 1).
Fig. 3.
Fig. 3. (a) Training starts by encoding an image as a vector of current source densities in the FDFD simulation. This step is followed by an iterative process to solve for the electric field in a nonlinear medium. Next, we use the ASM to calculate the gradient, which is then used to update the level-set function and consequently, the medium itself. Here we use mini-batch SGD (explained in the supplementary materials section of Ref. [17]). In training with mini-batches, we sum the cost functions calculated for different images in the same batch and compute the gradients. (b)–(d) show an NNM in training after 1, 33, and 66 training iterations, respectively. (After iteration 66, the medium has already seen each of the training samples at least once, since we are using batches of 100 images.) At each step, the boundary between the host material and the inclusions is shown, along with the field distribution for the same randomly selected digit 8. Also, the accuracy of the medium on the test set can be seen for that particular stage in training.
Fig. 4.
Fig. 4. (a) 3D NNM case. Different colors illustrate varying values of permittivity. The input image is projected onto the top surface. Computing is performed while the wave propagates through the 3D medium. The field distribution on the bottom surface is used to recognize the image. Full-wave simulation shows the optical energy is concentrated on the location with the correct class label, in this case 6. (b) The confusion matrix. The rows on the matrix show true labels of the images that have been presented as input, and the columns depict the labels that the medium has classified each input. Therefore, the diagonal elements show the number of correct classifications out of every 10 samples (see Visualization 2).

Equations (5)

Equations on this page are rendered with MathJax. Learn more.

L(r,E(r))E(r)=iωJ(r),
C=i=110yilog(hi)+(1yi)log(1hi).
dCdε(r)=2ω2Real{λ(r)E(r)}.
CE(r)+λ(r)(L(r,E(r))+L(r,E(r))E(r)E(r))+λ¯(r)(L¯(r,E(r))E(r)E¯(r))=0.
ε(r)={εSiO2φ(r)<0εAirφ(r)>0.

Metrics