Spatiotemporal diffractive deep neural networks

Junhe Zhou; Junhe Zhou; Haoqian Pu; Jiaxin Yan

doi:10.1364/OE.494999

1. Introduction

Space-time duality has been an everlasting topic in physics, which inspired ingenious contributions. The analogy of the space and time in classic optics has led to significant discoveries in optical research [1,2]. For examples, the well-known Kramers-Kronig relation, which arises from the causal temporal response of the material [3], has its spatial counterpart [4]; the spatial domain reflection and refraction can happen in the time domain as well [5].

The combination of the spatial and temporal concepts will lead to new physical phenomena. For example, a spatiotemporal localized wave packet could exist during the propagation [6], which resembles the non-diffractive Bessel beam in space. Such waves are different from the pure spatial beams or temporal pulses, because the spatial and temporal spectra are not separable [7]. The other recently discovered spatiotemporal optical concept is the spatiotemporal optical vortex (STOV), which extends a purely spatial concept, i.e., optical vortex, to the four-dimensional space-time [8–14], and opens a new frontier for the research of structured light.

While the booming studies in the area have witnessed abundant discoveries, tools for signal processing in the complex spatiotemporal domain have gained attention. It was already known that gratings could convert the frequency domain signal into the spatial domain signal, and a spatial light modulator (SLM) might provide spatiotemporal modulation afterwards [13,15,16]. H. E. Kondakci et.al. has proposed to generate the spatiotemporal diffraction free beams with a grating and a SLM [16] and S. Wang et. al. suggested to generate the STOV wave-packets based on a similar experimental setup [13]. M. Mounaix et.al. suggested multi-plane light conversion (MPLC) can be more robust and accurate in shaping the spatiotemporal waves [17]. However, it required a relatively complex spectral pulse shaper and 14 phase planes with the phases optimized through simulated annealing to accomplish the task. Alternatively, H. Wang et. al. suggested that a customized photonic crystal slab with distinctive spatiotemporal dispersive properties could be used for STOV wave-packet generation [14]. L. Chen et. al. suggested that metasurface could be used to shape the spatiotemporal waves [18]. Despite the pioneering researches [8–18], there is still a strong demand for a universal tool for spatiotemporal signal processing, which could generate an arbitrary spatiotemporal signal or multiplex/demultiplex different spatiotemporal symbols, e.g., the STOV wave-packets with different topological charges, with low complexity.

Neural networks are powerful signal processing tools and have witnessed rapid development with various applications [19]. Recently, an all-optical neural network, i.e., the diffractive deep neural network (D2NN), has emerged as one of the latest achievements in the field [20–30], which enables light speed all optical signal processing and recognition with ultra-low power consumption. The D2NN is different from the conventional neural network, which has the nonlinear activation part. The signal processing is accomplished through multiple layer diffraction. The concept of D2NN is generally implemented in the spatial domain, which consists of cascaded free space propagation and phase tuning elements to realize spatial optical wave transformation. We proposed a time domain D2NN, which used dispersive elements to mimic the time domain “free propagation” followed by the time domain phase modulators [28]. Recently, D2NN for spatiotemporal signal processing was proposed [29,30]. However, one should implement either temporal integration at the detector side or temporal buffers, which makes the configurations only suitable for low speed (in the order of ns [30]) spatiotemporal signal processing and not applicable for the high-speed spatiotemporal wave-packets, like the STOV wave-packets (in the order of 0.1 ps) processing.

In this work, we propose a novel architecture for signal processing in the spatiotemporal domain, which is referred to as the spatiotemporal diffractive deep neural network (STD2NN). The STD2NN is composed of the input and output gratings and multiple layers with each layer formed by a 4f system and a spatial light modulator (SLM) embedded in the middle. Firstly, signal encounters reflection by an input grating, which converts the frequency domain signal into the x-dimensional signal in the spatial domain. The signal therefore possesses the frequency information in the x-dimension and the spatial information in the y-dimension. It undergoes individual spatial and temporal propagation in the two dimensions of the 4f system with phase tuning in the middle by the SLM. To efficiently adjust the phases of the SLMs, training is accomplished via an all-optical backpropagation (BP) algorithm, which uses the forward propagating field and the backward propagating conjugated error field to calculate the phase gradient, which significantly speeds up the optimization process and allows real time phase adjustment. At the output, the signal is reflected by the output grating and the target spatiotemporal signal is generated. As a proof of concept, a spatiotemporal word “OPTICA” is generated by the STD2NN. Afterwards, a STOV wave-packet multiplexer based on the STD2NN is demonstrated, which converts the spatially separated Gaussian beams into the STOV wave-packets with different topological charges. Both cases illustrate the capability of the proposed STD2NN to generate and process an arbitrary spatiotemporal signal, which suggests that the STD2NN enables arbitrary pulse shaping in space-time simultaneously as a whole.

2. STD2NN architecture and all optical error back-propagation

The proposed STD2NN is composed of two gratings and multiple neuron layers, with each layer formed by a 4f pulse shaper with a SLM embedded in the middle. The fundamental structure of the STD2NN can be found in Fig. 1.

Fig. 1. Basic structure of the STD2NN.

Download Full Size | PDF

At the input the STD2NN, the signal S₀(x₀,y₀,ω) with the spatial variables x₀, y₀ and the frequency variable ω is reflected by a grating and is converted to U₀ as the input of the 4f system [13,15]:

(1)$${U_0}({{x_0},{y_0},\omega } )= \sqrt \beta {S_0}({\beta {x_0},{y_0},\omega } )\exp ({j\gamma \omega {x_0}} ),$$

where β=cos(θ_i)/cos(θ_d) with θ_i and θ_d as the incident angle and the diffraction angle, γ=k/(dcos(θ_d)) with k as the free space wavenumber and d as the grating line spacing. The implementation of diffractive gratings in input and output terminals is to uniquely map a specific frequency component to a specific point on the x axis and to turn the conventional spatial D2NN into the spatiotemporal D2NN.

The 4f pulse shaper is formed by two cylindrical lenses [13,15] which focus the beams in the x direction. While in the y direction, free space propagation occurs. Dropping the common phase term exp(2jkf), we have the field U₁^’ on the focal plane of the first lens before conducting spatiotemporal modulation as [13,15]:

(2)$$\begin{aligned} &{{\tilde{U}}_1}^\prime ({{x_1},{k_y},\omega } )= \frac{1}{{\sqrt {\lambda f} }}\exp \left( { - \frac{{jk_y^2f}}{k}} \right){{\tilde{\tilde{U}}}_0}\left( {\frac{{k{x_1}}}{f},{k_y},\omega } \right)\\ &= \frac{1}{{\sqrt {\lambda f\beta } }}\exp \left( { - \frac{{jk_y^2f}}{k}} \right){{\tilde{\tilde{S}}}_0}\left( {\frac{1}{\beta }\left( {\frac{{k{x_1}}}{f} - \gamma \omega } \right),{k_y},\omega } \right), \end{aligned}$$

where f is the focal length of the lens, λ is the free space wavelength, and k_y is the spatial frequency in the y direction. $\tilde{U}$ denotes the Fourier transform in the y dimension, and $\tilde{\tilde{U}}$ denotes the two-dimensional Fourier transform in the x-y dimensions. The derivation of Eq. (2) is shown in the Appendix A.

If the incident beam behaves as a plane wave in the x direction, e.g., it maintains a constant value 1 in the x direction, we have:

(3)$${\tilde{U}_1}^\prime ({{x_1},{k_y},\omega } )= \frac{{2\pi }}{{\sqrt {\lambda f\beta } }}\exp \left( { - \frac{{jk_y^2f}}{k}} \right)\delta \left( {\frac{1}{\beta }\left( {\frac{{k{x_1}}}{f} - \gamma \omega } \right)} \right){\tilde{S}_0}({{k_y},\omega } ),$$

where δ is the Dirac function. A direct match between the x dimension in the spatial domain and the frequency domain ω can be observed on the focal plane, i.e., the Fourier plane. It should be noted that when the beam width is large enough in the x direction, it can be approximated as the plane wave.

Therefore, we may have the SLM to conduct spatiotemporal modulation and obtain the field U₁ after the SLM as:

(4)$${U_1}({{x_1},{y_1},\omega } )= {U_1}^\prime ({{x_1},{y_1},\omega } )M({{x_1},{y_1}} ),$$

where M is the spatiotemporal phase modulation mask introduced by the SLM, which can be viewed as the controllable neurons within the first layer of the STD2NN.

The phase modulation mask is divided into numerous cells which are arranged as a two-dimensional array, with each cell to be considered as an individual neuron, whose phase is adjustable between 0 and 2π. The neuron array conducts the spatial modulation in the y dimension and the temporal modulation in the x dimension as explained above, and each layer has an array of neurons. Different from the traditional neural networks, diffractive deep neural networks do not require weight matrices.

The field further propagates through the second lens with the same focal length f and we have field U₁^’’ on its focal plane as:

(5)$${\tilde{U}_1}^{\prime \prime }({{x_2},{k_y},\omega } )= \frac{1}{{\sqrt {\lambda f\beta } }}\exp \left( { - \frac{{jk_y^2f}}{k}} \right){\tilde{\tilde{U}}_1}\left( {\frac{{k{x_2}}}{f},{k_y},\omega } \right),$$

In the space-frequency domain, we have:

(6)$${\tilde{\tilde{U}}_1}^{\prime \prime }({{k_x},{k_y},\omega } )= \sqrt {\frac{{\lambda f}}{\beta }} \exp \left( { - \frac{{jk_y^2f}}{k}} \right){\tilde{U}_1}\left( { - \frac{{f{k_x}}}{k},{k_y},\omega } \right).$$

The above results present the detailed mathematical model for the beam evolution within the first layer of the STD2NN. For the layers in the middle, 4f system is still implemented and the field varies according to Eq. (2)–(6) in a cascaded way. Such a configuration is to ensure that the different frequency components are mapped to the different points of the SLM on the x axis. If a fully diffractive approach is implemented, one SLM point on the x axis could have the mixed frequency components and makes the f-y dimensional modulation impossible.

The model established above can be used for the numerical calculation for the spatiotemporal signals. However, the SLM phase is usually discretized and one needs to find a method to tune the phase so that the target spatiotemporal signal can be generated. Therefore, we will introduce a discretized model, which will facilitate the derivation of the BP algorithm for SLM phase tuning.

From the analysis above, if the beam fulfils the plane approximation in the x direction, it can be viewed as a two-dimensional t-y (or ω-y) spatiotemporal (or spatial and frequential) symbol on the SLM plane while it passes through the STD2NN. Hence, we may discretize the system in space and frequency, and use a matrix to represent such an optical field after the SLM of each layer, with the rows indicating the frequency domain numbering and the columns indicating the spatial domain numbering. It is related to the optical field after the SLM of the previous layer as:

(7)$${{\mathbf U}_k} = ({{{\mathbf D}_k}^s{{\mathbf U}_{k - 1}}{{\mathbf D}_k}^f} )\odot {{\mathbf M}_k},$$

where matrix U_k stands for the optical field after the SLM of the k^th layer of the ST-D2NN, D_k^f and D_k^s are the frequency domain and spatial domain propagation operators for the x dimensional lenses and the y dimensional free space propagation, M_k is a phase operator introduced by the SLM, and ${\odot}$ denotes element-wise product. While U_k is a matrix, D_k^f and D_k^s are two square matrices with their sizes equaling the number of rows and the number of columns in U_k. M_k is a matrix with the same size as U_k, and the element of M_k on the l^th row and the m^th column is denoted as n_k,lm, which is considered as the neuron of the STD2NN, with n_k,lm= exp(jφ_k,lm). Assuming we have N layers of neurons for the STD2NN, we have:

(8)$${{\mathbf U}_N} = ({{{\mathbf D}_N}^s \cdots ({{{\mathbf D}_k}^s({ \cdots {{\mathbf U}_0} \cdots } ){{\mathbf D}_k}^f} )\odot {{\mathbf M}_k} \cdots {{\mathbf D}_N}^f} )\odot {{\mathbf M}_N},$$

where U₀ is the input signal and U_N is the signal after the SLM of the output layer. Finally, we have:

(9)$${{\mathbf U}_{N + 1}} = {{\mathbf D}_{N + 1}}^s{{\mathbf U}_N}{{\mathbf D}_{N + 1}}^f,$$

where U_N_+ 1 is the output beam after the reflection by the second grating, which is with the same physical parameters but placed in a symmetrical way with respect to the input grating, and D_N +₁^f and D_N +₁^s are the two propagation operators in the frequency and spatial domain. To optimize the neuron phases, we need to derive the BP algorithm. The loss function is defined as

(10)$$L = {|{{{\mathbf U}_{N + 1}} - {\mathbf T}} |^2} = Trace({{{({{{\mathbf U}_{N + 1}} - {\mathbf T}} )}^H}({{{\mathbf U}_{N + 1}} - {\mathbf T}} )} ),$$

where T is the target wave.

The gradient of the phase matrix φ_k with respect to the loss function can be formulated in the matrix form as (see Appendix A):

(11)$$\begin{aligned} &\frac{{\partial L}}{{\partial {\boldsymbol {\mathrm{\varphi}}_k}}} = 2\textrm{Re} ({j{\mathbf P}_k^b \odot {\mathbf P}_k^f} ),\\ &{\mathbf P}_k^f = ({{{\mathbf D}_k}^s({ \cdots {{\mathbf U}_0} \cdots } ){{\mathbf D}_k}^f} )\odot {{\mathbf M}_k},\\ &{\mathbf P}_k^b = ({{{\mathbf D}_{k + 1}}^{sT}({ \cdots {{\mathbf D}_N}^{sT}({{{\mathbf D}_{N + 1}}^{sT}{{({{{\mathbf U}_{N + 1}} - {\mathbf T}} )}^\ast }{{\mathbf D}_{N + 1}}{{^f}^T}} )\odot {{\mathbf M}_N}^T{{\mathbf D}_N}{{^f}^T} \cdots } )\odot {{\mathbf M}_{k + 1}}^T{{\mathbf D}_{k + 1}}{{^f}^T}} ), \end{aligned}$$

where ^T stands for the matrix transpose operation. The interpretation of the field P_k^f is quite straight forward as shown in Eq. (8), which represents the forward propagating optical field after the SLM of the kth layer. The field P_k^b is actually the backward propagating field after the SLM of the kth layer, which starts propagation from the output of the STD2NN with the input field as (U_N + 1-T)*. The reciprocal property of the optical element [31] suggests that matrices D_k^fT, D_k^fT and M_k^T are the related propagation matrices and phase matrices in case the signal is inversely injected into the STD2NN. In the proposed STD2NN, the matrices are symmetric and we have:

(12)$${{\mathbf D}_k}^{fT} = {{\mathbf D}_k}^f,{{\mathbf D}_k}^{sT} = {{\mathbf D}_k}^s,{{\mathbf M}_k}^T = {{\mathbf M}_k}.$$

The conjugated error field (U_N + 1-T)* can be obtained through the interference between the output field U_N +₁ and the target field T and the phase conjugation by the nonlinear process. The detailed process for the all-optical BP algorithm is shown in Fig. 2.

Fig. 2. Forward and backward propagation of the matrices ${\mathbf P}_k^f$ and ${\mathbf P}_k^b$.

Download Full Size | PDF

After the gradient of the phase in each layer is obtained, one may update it as:

(13)$${\boldsymbol {\mathrm{\varphi}}_k}^{new} = {\boldsymbol {\mathrm{\varphi}}_k}^{old} - \mu \frac{{\partial L}}{{\partial {\boldsymbol {\mathrm{\varphi}}_k}^{old}}},$$

where µ is the learning rate.

Initially, the pixel of the SLM can be set with a random phase, which converges after the iterations to produce very similar final output patterns. Without loss of generality, we have assumed the initial phase to be zero.

Although the gradient is formulated for a single target, it can be easily used for the multiple-target case by consecutive training of the STD2NN with different targets.

It is worth mentioning that since one dimension, i.e., the x dimension is considered to fulfill the plane wave assumption, only two dimensions are considered during the optical computing for the phase gradients. Therefore, it requires same computing resources as the 2D D2NN does.

3. Results and discussions

We designed a STD2NN with the above-mentioned structure. The input signal has the wavelength of 1550 nm. At the input, the signal is reflected by a grating with the incident angle of 50 degree and the diffraction angle of 9.44 degree. The grating has the diffraction pattern of 600 lines per mm. Then, the signal is injected into a cascaded pulse shaping system which is formed by seven layers. Each layer is composed of two lenses with the focal lengths of 0.005 m, and a phase-only SLM with 300 × 800 pixels. The pixel size is 8 µm × 8 µm. Since the phase-only SLM usually works in the reflection mode, one may reuse the lens on the reflective route and use only one lens for each layer in the experimental setup. Furthermore, multiple layers may jointly use one SLM (see Appendix A). The initial Gaussian beam waist of the input light is 50 µm × 50 µm in the x and y dimensions, which fits the output beam shape of a multimode fiber. In the temporal dimension, the beam has the pulse shape to be Gaussian as well. The STD2NN phase optimization is realized according to Eq. (13). The maximum pulse length for the STD2NN to modulate is dependent on the resolution of the frequency frame, which is related to the parameter γ and the pixel length. With the provided parameters in the manuscript, it can be evaluated that γ= 3.1447e-9. It can be computed that the frequency resolution is about 2THz per pixel. Hence, the longest pulse length should be 3 ps. Pulse longer than the length is not eligible for modulation, as the STD2NN is not able to resolve the finer frequency units. The plane wave assumption is adopted to compute the forward error matrix P_k^f and the backward error matrix P_k^b.

We test the capability of the STD2NN by using it to generate the word “OPTICA” in the spatiotemporal domain, which is shown in Fig. 1. For the OPTICA pattern, it is found that the pulse duration should be less than 0.05 ps to have more high frequency components, and the temporal beam waist is set as 0.038 ps. It should be noticed that all the beam amplitudes shown in the figures have been normalized so that the maximum amplitude is always unit one. While the x dimension is the plane wave as shown in Fig. 3(a1), the Fourier plane of the 4f system will conduct phase modulation on the frequency-spatial hybrid plane, i.e., the ω-y plane. Using the word “OPTICA” as the training target and after 200 iterations, we have the output as Fig. 3(b1) and Fig. 3(b2), which show that the word “OPTICA” has been generated in a perfect manner.

Fig. 3. The spatiotemporal word “OPTICA” generated by the STD2NN. (a1) The x-dimensional distribution function as 1 (the pure plane wave). (b1) The corresponding output amplitude of the STD2NN. (b2) The corresponding output phase of the STD2NN. (a2) The x-dimensional distribution function as the Gaussian function (the approximated plane wave). (c1) The corresponding output amplitude of the STD2NN. (c2) The corresponding output phase of the STD2NN.

Download Full Size | PDF

In practice, the beam has the beam waist of 50 µm in the x dimension, and is expanded by placing it on the focal plane of an x-dimensional cylindrical lens with the focal length of 0.2 m. Two y-dimensional cylindrical lenses form a 4f system and ensure the y-dimensional beam shape unaltered. The x-dimensional beam waist is expanded to about 2 mm as shown in Fig. 3(a2), which can be considered to fulfill the plane wave assumption. The output spatiotemporal amplitude and phase on the y-t plane when x = 0 mm are shown in Fig. 3(c1-c2). A detailed analysis for the spatiotemporal beam away from the center x = 0 mm is conducted to investigate the impact of noncompliance of the plane wave assumption, which shows that the plane wave assumption holds (see Appendix A).

The phase distributions, i.e., the neuron values, on the layers of the STD2NN to form the spatiotemporal word “OPTICA” are shown in Fig. 4(a-g), which suggest that only the phase in the center of the spatiotemporal modulation plane plays an important role for spatiotemporal beam shaping and therefore, one may reuse the phase modulation plane by placing a mirror and a SLM together so that the beam can be reflected between them and multiple 4f systems can be formed by inserting a cylindrical lens in the middle (see Appendix A). This is caused by the fact the waves have the power concentrating on the lower frequency and hence the SLM frequency axis only matters by a small range. The frequency range of the wave is determined by the pulse duration. If one needs to use more degree of freedom on each phase plane, a shorter pulse is expected to be implemented. The spatiotemporal beam amplitude and phase at the output of each layer are shown in Fig. 4(a1-g1) and Fig. 4(a2-g2), which demonstrate the evolutionary behavior of an input Gaussian beam converting to a clear spatiotemporal word “OPTICA”. It should be noted that one may require less diffractive layers and 4f systems to generate a single spatiotemporal image. However, the generation of multiple STOV wave-packets requires so many layers as to be shown later. In order to be consistent, we have implemented the same device to accomplish the two tasks.

Fig. 4. The spatiotemporal word “OPTICA” generated by the STD2NN. (a-g) The phase distributions of the neuron layers 1-7. (a1-g1) The spatiotemporal beam amplitudes at the output of the layers 1-7. (a2-g2) The spatiotemporal beam phases at the output of the layers 1-7.

Download Full Size | PDF

After demonstrating the generation of the word “OPTICA”, we considered a STOV wave-packet multiplexer formed by the same STD2NN. The input signal wavelength and the beam spatial distribution remain unchanged. The temporal waist is 0.19 ps. Two such beams are separated in the y dimension by 0.4 mm as shown in Fig. 5(a1-a2). After spatiotemporally processed by the STD2NN, the two beams are converted into two STOV wave-packets with the topological charges of 1 and 2. The related amplitudes and phases of the beams are shown in Fig. 5(b1-b2) and Fig. 5(d1-b2). It can be seen from the figure that the two STOV wave-packets appear in the same spatiotemporal region and it suggests multiplexing of the STOV wave-packets has been realized. The spiral wavefronts in Fig. 5(d1-d2) have demonstrated that the STOV does occur after the spatiotemporal transformation. The ideal case under the plane wave assumption is plotted in Fig. 5(c1-c2) and Fig. 5(e1-e2). Compared with the ideal case, the actual x-dimensional Gaussian shape beam does have a deteriorated performance, but the multiplexing phenomenon can still be clearly observed. The x dimensional sliced outputs of the STD2NN for STOV wave-packets are shown in Fig. 6, which clearly indicates that the amplitude and the phase maintain almost unaltered as the x dimensional position changes. Figures 5–6 suggest the STOV multiplexing can be achieved using the proposed device.

Fig. 5. The STOV wave-packets generated by the STD2NN. (a1) The initial spatiotemporal beam to generate STOV with l = 1. (b1) The STOV wave-packet amplitude with the topological charge of 1. (c1) The STOV wave-packet amplitude with the topological charge of 1 under the plane wave assumption. (d1) The STOV wave-packet phase with the topological charge of 1. (e1) The STOV wave-packet phase with the topological charge of 1 under the plane wave assumption. (a2) The initial spatiotemporal beam to generate STOV with l = 2. (b2) The STOV wave-packet amplitude with the topological charge of 2. (c2) The STOV wave-packet amplitude with the topological charge of 2 under the plane wave assumption. (d2) The STOV wave-packet phase with the topological charge of 2. (e2) The STOV wave-packet phase with the topological charge of 2 under the plane wave assumption.

Download Full Size | PDF

Fig. 6. The STOV wave-packets generated by the STD2NN sliced at different x positions. (a1-g1) The STOV wave-packet amplitudes with l = 1 located at x = -3, -2, -1, 0, 1, 2, and 3 mm. (a2-g2) The STOV wave-packet phases with l = 1 located at x = -3, -2, -1, 0, 1, 2, and 3 mm. (a3-g3) The STOV wave-packet amplitudes with l = 2 located at x = -3, -2, -1, 0, 1, 2, and 3 mm. (a4-g4) The STOV wave-packet phases with l = 2 located at x = -3, -2, -1, 0, 1, 2, and 3 mm.

Download Full Size | PDF

The phase distributions, i.e., the neuron values, on the layers of the STD2NN to achieve STOV wave-packet multiplexing are shown in Fig. 7(a-g). Again, the effective neurons concentrate in the middle of the spatiotemporal modulation plane and therefore, it reaffirms that the STD2NN composed by the cascaded 4f systems can be formed by a single SLM as discussed in the previous paragraph. Compared with Fig. 4, which has the temporal beam waist as 0.76 ps, the effective neuron stripe is narrower in Fig. 7, because corresponding temporal beam waist is 3.8 ps.

Fig. 7. The STOV wave-packets generated by the STD2NN at different layer outputs. (a-g) The phase distributions of the neuron layers 1-7. (a1-g1) The STOV wave-packet amplitudes with l = 1 at the output of the layers 1-7. (a2-g2) The STOV wave-packet phases with l = 1 at the output of the layers 1-7. (a3-g3) The STOV wave-packet amplitudes with l = 2 at the output of the layers 1-7. (a4-g4) The STOV wave-packet phases with l = 2 at the output of the layers 1-7.

Download Full Size | PDF

The evolutionary behavior of the STOV wave-packets during the conversion process is shown in Fig. 7(a1-g1), Fig. 7(a2-g2), Fig. 7(a3-g3), and Fig. 7(a4-g4). The amplitudes and phases of the beams at the output of the layers 1-7 for the evolutionary paradigm of the STOV wave-packet with l = 1 are shown in Fig. 7(a1-g1) and Fig. 7(a2-g2), while Fig. 7(a3-g3) and Fig. 7(a4-g4) demonstrate the amplitudes and phases for the evolutionary paradigm of the STOV wave-packet with l = 2. It can be seen from the figures that the beam evolves gradually from the initial shape without any STOV to the pure STOV wave-packet with the spiral wavefront holding the topological charge l = 1 or l = 2.

Finally, the proposed STD2NN is compared with the existing spatiotemporal pulse shaping system, which is formed by two gratings, a SLM and a 4f pulse shaper [11,13,14]. Both the proposed STD2NN and the traditional spatiotemporal beam shaping system are aimed to achieve STOV wave-packet multiplexing, i.e., to generate the STOV wave-packets with different topological charges by injecting the beams located at different y positions. With the same optimization target, the STD2NN has already demonstrated its capability in Figs. 6–7 to realize STOV wave-packet multiplexing and its output has the relative error with respect to target (defined as the integral of the loss function divided by the integral of the spatiotemporal power of the target) as about 35%, while the traditional spatiotemporal beam shaping system cannot achieve similar performance as shown in Fig. 8(a). With the target STOV wave-packets possessing the topological charges of l = 1 and l = 2, the amplitudes and phases of the spatiotemporal beams shaped by the traditional spatiotemporal beam shaping system are shown in Fig. 8(b1-b2) and Fig. 8(c1-c2). While the amplitudes differ remarkably from the results in Figs. 6–7, the phases in Fig. 8(b2) and Fig. 8(c2) are not in a spiral shape and do not possess any topological charges. Hence, we may conclude that the traditional spatiotemporal beam shaping system are not able to achieve STOV wave-packet multiplexing while the proposed STD2NN is capable of. (The Matlab code to generate the phase of the STD2NN for STOV multiplexing is added as we show in Code 1, Ref. [32] and the code for the propagation of the 4D beam is added as we show in Code 2, Ref. [33]).

Fig. 8. Comparison of the results by the proposed STD2NN and the traditional spatiotemporal beam shaping system with two gratings, a SLM and a 4f pulse shaper. (a) Relative error between the output and the target after 400 neuron phase optimizations iteration by the two methods. (b1) The amplitude of the spatiotemporal beam shaped by the traditional spatiotemporal beam shaping system with the target STOV topological charge of l = 1. (b2) The phase of the spatiotemporal beam shaped by the traditional spatiotemporal beam shaping system with the target STOV topological charge of l = 1. (c1) The amplitude of the spatiotemporal beam shaped by the traditional spatiotemporal beam shaping system with the target STOV topological charge of l = 2. (c2) The phase of the spatiotemporal beam shaped by the traditional spatiotemporal beam shaping system with the target STOV topological charge of l = 2.

Download Full Size | PDF

4. Conclusion

In summary, we have proposed a new framework for spatiotemporal signal processing, which is referred to as the STD2NN. It is realized by two gratings and several cascaded 4f pulse shaping systems with SLMs embedded in the middle. An all-optical BP algorithm is derived with the gradient of the loss function proportional to the inner product of the forward field and the backward conjugated error field. The word “OPTICA” is generated and a STOV wave-packet multiplexer is demonstrated based on the STD2NN. To realize the experimental demonstration, a femtosecond pulsed laser source is required. The proposed STD2NN is expected to play an important role in the field of spatiotemporal signal processing, which might enable novel optical beams, particularly space-time entangled beams to be generated and the potential applications, like information multiplexing in space-time, could be considered in the future. For example, one may multiplex the STOVs to carry different information and to constitute a new scheme of multiplexing.

Appendix A

Derivation of Eq. (2)

Ignoring the x dimension, the propagation of the beam in the y dimension for the distance of 2f follows the rules in the free space:

(14)$${U_1}^\prime ({y,\omega } )= IF{T_y}\left( {\exp \left( { - \frac{{jk_y^2f}}{k}} \right)F{T_y}({{U_0}({y,\omega } )} )} \right),$$

where FT_y and IFT_y stand for the y dimensional Fourier transform and inverse Fourier transform. Ignoring the y dimension, the beam in the x dimension is Fourier transformed by the lens:

(15)$$\begin{aligned} {U_1}^\prime ({x,\omega } )&= \frac{1}{{\sqrt {\lambda f} }}F{T_x}({{U_0}} )\left( {\frac{{kx}}{f},\omega } \right)\\ &= \frac{1}{{\sqrt {\lambda f} }}F{T_x}\left( {\sqrt \beta {S_0}({\beta x,\omega } )\exp ({j\gamma \omega x} )} \right)\left( {\frac{{kx}}{f},\omega } \right)\\ &= \frac{1}{{\sqrt {\beta \lambda f} }}F{T_x}({{S_0}} )\left( {\frac{1}{\beta }\left( {\frac{{kx}}{f} - \gamma \omega } \right),\omega } \right), \end{aligned}$$

where FT_x stands for the Fourier transform in the x dimension. Combining Eq. ((14)–(15)), we have Eq. (2).

Derivation of the BP algorithm formulas

In this part of the appendix, the detailed derivation for the BP algorithm is provided. Before moving forward, one has to obtain the gradient of the field with respect to the neuron:

(16)$$\begin{aligned} &\frac{{\partial {{\mathbf U}_{N + 1}}}}{{\partial {n_{k,lm}}}} = {{\mathbf D}_{N + 1}}^s\left( {\left( {{{\mathbf D}_N}^s \cdots ({{{\mathbf D}_k}^s({ \cdots {{\mathbf U}_0} \cdots } ){{\mathbf D}_k}^f} )\odot \frac{{\partial {{\mathbf M}_k}^s}}{{\partial {n_{k,lm}}}} \cdots {{\mathbf D}_N}^f} \right) \odot {{\mathbf M}_N}} \right){{\mathbf D}_{N + 1}}^f\\ &= {{\mathbf D}_{N + 1}}^s({({{{\mathbf D}_N}^s \cdots ({{{\mathbf D}_k}^s({ \cdots {{\mathbf U}_0} \cdots } ){{\mathbf D}_k}^f} )\odot {{\mathbf 1}_{lm}} \cdots {{\mathbf D}_N}^f} )\odot {{\mathbf M}_N}} ){{\mathbf D}_{N + 1}}^f. \end{aligned}$$

{{\mathbf 1}^{lm}}({i,j} )= \left\{ {\begin{array}{*{20}{c}} {1\quad \quad i = l\;and\;j = m}\\ {0\quad \quad otherwise\quad \quad } \end{array}} \right.

The gradient of the L function with respect to the neuron can be calculated as:

(17)$$\begin{aligned} &\frac{{\partial L}}{{\partial {n_{k,lm}}}} = Trace\left( {{{({{{\bf U}_{N + 1}} - {\bf T}} )}^{H}}\frac{{\partial {{\bf U}_{N + 1}}}}{{\partial {n_{k,lm}}}}} \right)\\ &= Trace(({{{\bf U}_{N + 1}} - {\bf T}} )^{H}({{\bf D}_{N + 1}}^{s}({{\bf D}_N}^{s} \cdots ({{\bf D}_k}^{s}(\cdots {{\bf U}_0} \cdots ){{\bf D}_k}^{f} )\odot {{\bf 1}^{lm}} \cdots {{\bf D}_N}^{f} )\odot {{\bf M}_N}{{\bf D}_{N + 1}}^{f} ) )\\ &= Trace({{\bf D}_{k + 1}}^{f}( \cdots {{\bf D}_N}^{f}({{{\bf D}_{N + 1}}^{f}{{({{{\bf U}_{N + 1}} - {\bf T}} )}^{H}}{{\bf D}_{N + 1}}^{s}} )\\&\quad\odot {{\bf M}_N}^{T}{{\bf D}_N}^{s} \cdots )\odot {{\bf M}_{k + 1}}^{T}{{\bf D}_{k + 1}}^{s}({({{{\bf D}_k}^{s}({ \cdots {{\bf U}_0} \cdots } ){{\bf D}_k}^{f}} )\odot {{\bf 1}^{lm}}} ) )\\ &= {\bf P}_k^{b}({l,m} ){\bf Q}^{f}_{k\, lm}({l,m} ),\\ &{\bf Q}_k^{f} = ({{{\bf D}_k}^{s}({ \cdots {{\bf U}_0} \cdots } ){{\bf D}_k}^{f}} ),\\ &{\bf P}_k^{b} = ({{{\bf D}_{k + 1}}^{{sT}}({ \cdots {{\bf D}_N}^{sT}({{{\bf D}_{N + 1}}^{sT}{{({{{\bf U}_{N + 1}} - {\bf T}} )}^{\ast} }{{\bf D}_{N + 1}}^{fT}} )\odot {{\bf M}_N}^{T}{{\bf D}_N}^{fT} \cdots } )\odot {{\bf M}_{k + 1}}^{T}{{\bf D}_{k + 1}}^{fT}} ). \end{aligned}$$

Here, we have treated n_k,lm and its complex conjugate as two independent variables and the derivative of U_N +₁H with respect to n_k,lm is zero in this case. From Eq. (17), it can be inferred that the derivative with respect to the neuron on the l^th row and the m^th column is related to the product of the related elements of the matrices Q_k^f and P_k^b. Therefore, we may formulate the fact in the matrix form:

(18)$$\frac{{\partial L}}{{\partial {{\mathbf n}_k}}} = {\mathbf P}_k^b \odot {\mathbf Q}_k^f.$$

Since neuron tuning is done by phase tuning, we have

(19)$$\begin{aligned} &\frac{{\partial L}}{{\partial {\varphi _{k,lm}}}} = j\frac{{\partial L}}{{\partial {n_{k,lm}}}}{n_{k,lm}} + c.c.\\ &= 2Re \left( {j\frac{{\partial L}}{{\partial {n_{k,lm}}}}{n_{k,lm}}} \right) \end{aligned}$$

where c.c. denotes complex conjugate. One may derive Eq. (11) afterwards.

Potential experimental setup of the STD2NN cascaded layers

For the proposed STD2NN, its cascaded 4f systems can be realized in a compact way. A SLM and a mirror can be placed together with a cylindrical lens between them, and each round trip between the mirror and the SLM forms a 4f system. The cascaded 4f systems can be potentially established experimentally as shown in Fig. 9, with multiple 4f systems jointly using one SLM and one cylindrical lens.

Fig. 9. The potential experimental setup for the cascaded 4f systems.

Download Full Size | PDF

Therefore, the proposed STD2NN can be realized with a mirror, a cylindrical lens and a SLM accompanied by two gratings at the input and output. Although the potential experimental setup is not easy to calibrate, it is, however, possible to be realized as shown in [34], which used a SLM and a mirror in the reflective way to generate an arbitrary spatial beam.

STOV beams located at different x positions with the word “OPTICA”

In the main text, the evolution of the STOV beam has been demonstrated. Since the input beam has the Gaussian distribution rather than the plane wave distribution in the x dimension, we have sliced the beam in the x dimension with x = -3 -2, -1, 0, 1, 2, and 3 mm. The related amplitudes and phases of the spatiotemporal beams are shown in Fig. 10. From Fig. 10, it can be concluded that the amplitude and the phase maintain almost unaltered as the x dimensional position changes.

Fig. 10. The spatiotemporal word “OPTICA” generated by the STD2NN sliced at different x positions. (a1-g1) The spatiotemporal beam amplitudes located at x = -3, -2, -1, 0, 1, 2, and 3 mm. (a2-g2) The spatiotemporal beam phases located at x = -3, -2, -1, 0, 1, 2, and 3 mm.

Download Full Size | PDF

Funding

National Natural Science Foundation of China (62375206); Science and Technology Commission of Shanghai Municipality (2021SHZDZX0100); Fundamental Research Funds for the Central Universities.

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. V. Torres-Company, J. Lancis, and P. Andrés, “Space-Time Analogies in Optics,” Prog. Opt. 56, 1–80 (2011). [CrossRef]

2. Y. Shen, Q. Zhan, L. G. Wright, et al., “Roadmap on spatiotemporal light fields,” J. Opt. 25(9), 093001 (2023). [CrossRef]

3. T. Harter, C. Füllner, J. N. Kemal, S. Ummethala, J. L. Steinmann, M. Brosi, J. L. Hesler, E. Bründermann, A.-S. Müller, W. Freude, S. Randel, and C. Koos, “Generalized Kramers–Kronig receiver for coherent terahertz communications,” Nat. Photonics 14(10), 601–606 (2020). [CrossRef]

4. S. A. R. Horsley, M. Artoni, and G. C. La Rocca, “Spatial Kramers–Kronig relations and the reflection of waves,” Nat. Photonics 9(7), 436–439 (2015). [CrossRef]

5. B. W. Plansinis, W. R. Donaldson, and G. P. Agrawal, “What is the Temporal Analog of Reflection and Refraction of Optical Beams?” Phys. Rev. Lett. 115(18), 183901 (2015). [CrossRef]

6. M. Yessenov, J. Free, Z. Chen, E. G. Johnson, M. P. J. Lavery, M. A. Alonso, and A. F. Abouraddy, “Space-time wave packets localized in all dimensions,” Nat. Commun. 13(1), 4573 (2022). [CrossRef]

7. M. Yessenov, L. A. Hall, K. L. Schepler, and A. F. Abouraddy, “Space-time wave packets,” Adv. Opt. Photonics 14(3), 455–570 (2022). [CrossRef]

8. K. Y. Bliokh and F. Nori, “Spatiotemporal vortex beams and angular momentum,” Phys. Rev. A 86(3), 033824 (2012). [CrossRef]

9. A. Chong, C. Wan, J. Chen, and Q. Zhan, “Generation of spatiotemporal optical vortices with controllable transverse orbital angular momentum,” Nat. Photonics 14(6), 350–354 (2020). [CrossRef]

10. S. W. Hancock, S. Zahedpour, A. Goffin, and H. M. Milchberg, “Free-space propagation of spatiotemporal optical vortices,” Optica 6(12), 1547–1553 (2019). [CrossRef]

11. K. Y. Bliokh, “Spatiotemporal vortex pulses: angular momenta and spin-orbit interaction,” Phys. Rev. Lett. 126(24), 243601 (2021). [CrossRef]

12. C. Wan, Q. Cao, J. Chen, A. Chong, and Q. Zhan, “Toroidal vortices of light,” Nat. Photonics 16(7), 519–522 (2022). [CrossRef]

13. S. Huang, P. Wang, X. Shen, and J. Liu, “Properties of the generation and propagation of spatiotemporal optical vortices,” Opt. Express 29(17), 26995–27003 (2021). [CrossRef]

14. H. Wang, C. Guo, W. Jin, A. Y. Song, and S. Fan, “Engineering arbitrarily oriented spatiotemporal optical vortices using transmission nodal lines,” Optica 8(7), 966–971 (2021). [CrossRef]

15. M. M. Wefers and K. A. Nelson, “Space-time profiles of shaped ultrafast optical waveforms,” IEEE J. Quantum Electron. 32(1), 161–172 (1996). [CrossRef]

16. H. E. Kondakci and A. F. Abouraddy, “Diffraction-free space-time light sheets,” Nat. Photonics 11(11), 733–740 (2017). [CrossRef]

17. M. Mounaix, N. K. Fontaine, D. T. Neilson, R. Ryf, H. Chen, J. C. Alvarado-Zacarias, and J. Carpenter, “Time reversed optical waves by arbitrary vector spatiotemporal field generation,” Nat. Commun. 11(1), 5813 (2020). [CrossRef]

18. L. Chen, W. Zhu, P. Huo, J. Song, H. J. Lezec, T. Xu, and A. Agrawal, “Synthesizing ultrafast optical pulses with arbitrary spatiotemporal control,” Sci. Adv. 8(43), eabq8314 (2022). [CrossRef]

19. E. Mjolsness and D. DeCoste, “Machine learning for science: state of the art and future prospects,” Science 293(5537), 2051–2055 (2001). [CrossRef]

20. X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, “All-optical machine learning using diffractive deep neural networks,” Science 361(6406), 1004–1008 (2018). [CrossRef]

21. T. Yan, J. Wu, T. Zhou, H. Xie, F. Xu, J. Fan, L. Fang, X. Lin, and Q. Dai, “Fourier-space diffractive deep neural network,” Phys. Rev. Lett. 123(2), 023901 (2019). [CrossRef]

22. J. Bueno, S. Maktoobi, L. Froehly, I. Fischer, M. Jacquot, L. Larger, and D. Brunner, “Reinforcement learning in a large-scale photonic recurrent neural network,” Optica 5(6), 756–760 (2018). [CrossRef]

23. M. S. S. Rahman, J. Li, D. Mengu, Y. Rivenson, and A. Ozcan, “Ensemble learning of diffractive optical networks,” Light Sci. Appl. 10(1), 14 (2021).

24. J. Li, D. Mengu, Y. Luo, Y. Rivenson, and A. Ozcan, “Class-specific differential detection in diffractive optical neural networks improves inference accuracy,” Adv. Photonics 1(1), 1 (2019). [CrossRef]

25. T. Zhou, X. Lin, J. Wu, Y. Chen, H. Xie, Y. Li, and Q. Dai, “Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit,” Nat. Photonics 15(5), 367–373 (2021). [CrossRef]

26. T. Zhou, L. Fang, T. Yan, J. Wu, Y. Li, J. Fan, and Q. Dai, “In situ optical backpropagation training of diffractive optical neural networks,” Photonics Res. 8(6), 940 (2020). [CrossRef]

27. Y. Luo, Y. Zhao, J. Li, E. Cetintas, Y. Rivenson, M. Jarrahi, and A. Ozcan, “Computational imaging without a computer: seeing through random diffusers at the speed of light,” eLight 2(1), 4 (2022). [CrossRef]

28. J. Zhou, Q. Hu, and H. Pu, “Nonlinear Fourier transform receiver based on a time domain diffractive deep neural network,” Opt. Express 30(21), 38576–38586 (2022). [CrossRef]

29. R. M. S. Sakib and A. Ozcan, “Time-Lapse Image Classification Using a Diffractive Neural Network,” Adv. Intell. Syst. 5(5), 2200387 (2023). [CrossRef]

30. T. Zhou, W. Wu, J. Zhang, S. Yu, and L. Fang, “Ultrafast dynamic machine vision with spatiotemporal photonic computing,” Sci. Adv. 9(23), eadg4391 (2023). [CrossRef]

31. R. J. Potton, “Reciprocity in optics,” Rep. Prog. Phys. 67(5), 717–754 (2004). [CrossRef]

32. J. Zhou, H. Pu, and J. Yan, “Algorithm 1,” figshare (2023), https://doi.org/10.6084/m9.figshare.24431458.

33. J. Zhou, H. Pu, and J. Yan, “Algorithm 2,” figshare (2023), https://doi.org/10.6084/m9.figshare.24431464.

34. N. K. Fontaine, R. Ryf, H. Chen, D. T. Neilson, K. Kim, and J. Carpenter, “Laguerre-Gaussian mode sorter,” Nat. Commun. 10(1), 1865 (2019). [CrossRef]

Name	Description
Code 1	This program calculates the spatial temporal diffractive deep neural network phase distribution with two STOV beams as the targets
Code 2	This program calculates 4D beam propagation in the spatial temporal diffractive deep neural network with the phase evaluated by Algorithm 1.

Spatiotemporal diffractive deep neural networks

Abstract

1. Introduction

2. STD2NN architecture and all optical error back-propagation

3. Results and discussions

4. Conclusion

Appendix A

Derivation of Eq. (2)

Derivation of the BP algorithm formulas

Potential experimental setup of the STD2NN cascaded layers

STOV beams located at different x positions with the word “OPTICA”

Funding

Disclosures

Data availability

References

Supplementary Material (2)

Data availability

Cited By

Figures (10)

Equations (20)

Optics Express