1. Introduction

Phase retrieval (PR) intends to reconstruct a complex-valued signal only from its Fourier intensity measurements. It is a key problem in crystallography [1], optical imaging [2,3], inverse scattering [4], diffraction imaging [5], etc. PR is also a crucial component of holographic imaging [6]. The investigation of PR methods was initiated in the 1970s. Numerous reconstruction approaches were developed by the optics research community [1,2,7]. Recently, the developments in modern optimization theories [8,9] and computational imaging [10,11] provided further understanding of the problem. From the mathematical perspective, the PR problem can be expressed as follows [9]:

In the last three decades, deep neural networks (DNN) have been widely studied and successfully applied to different applications [16]. They were also used in the PR problem [17–26]. The deep learning-based PR methods can work with only a single Fourier intensity measurement (which is the so-called single-shot PR). These methods can fit in some feedforward DNN structures to achieve real-time performance when processing with GPU. Nevertheless, the accuracy of these methods still has much room to improve. It is still a challenge to directly infer a complex-valued signal from its Fourier intensity due to the enormous discrepancy between a signal in the Fourier and spatial domains. To improve the accuracy, researchers also suggested the physics-driven method [21] in which the intensity measurement was used to inform a plug-and-play optimization process. However, the method is iterative and as time-consuming as the traditional optimization-based methods. Besides using the plug-and-play structure, researchers also introduced the physics information to the PR process by directly plugging in the HIO algorithm [2] running alternately with a DNN in an iterative procedure [26]. These physics-driven approaches have a common characteristic in that they apply physics information to a traditional optimization method and let it work iteratively with a network model. However, in this case, the system cannot be end-to-end trained. The estimation error of the optimization algorithm may have a special distribution that is unknown to the network model. The iterative process can thus be trapped at a local minimum and fails to give the best solution.

One common problem of these deep learning-based PR methods is that most of them are trained and tested with simulation data. Except for noises, they often ignored the other artifacts in practical Fourier intensity measurements. It makes their reported results unreliable. Most PR applications involve structured images, which have energy concentrated at low frequencies (in particular, at zero frequency). The dynamic range of the data in an intensity measurement is thus extremely large. Most general imaging devices nowadays have only a 12 to 16 bits dynamic range. It makes the low-frequency part of the intensity image severely saturated. As we have shown in Section 4.6, the saturation problem can significantly affect the performance of PR methods.

To rectify the abovementioned problems, we propose in this paper a novel physics-driven deep learning-based PR method dubbed PPRNet (PPRNet stands for Physics-driven Phase Retrieval Network). Similar to other deep learning-based PR methods, PPRNet requires only a single Fourier intensity measurement for each PR reconstruction. It, however, gives a much higher accuracy by using a physics-driven method that guides the network to follow the Fourier intensity measurement at different scales to reconstruct the images. The resulting network structure is still of feedforward type. Thus, it is much faster than the iterative physics-driven approaches. The whole network can be end-to-end trained to give the best result. PPRNet is enabled by a novel Hybrid Unwinding Block (HUB) embedded in a multi-scale convolutional neural network (CNN) structure. It directs the input feature map into two paths such that the global and local information of the feature map at different scales are separately processed with physics informed. The data are then combined with a channel attention method so that the significant features are collected for reconstructing the images. To evaluate the generality of the proposed PPRNet, we have conducted a series of simulations with 2 different datasets that contain complex-valued images with linearly correlated and uncorrelated magnitude and phase. All data in the intensity image were capped and quantized as 12-bit integers to simulate the saturation problem with quantization errors. We then adopted the defocusing method in [27–29] to mitigate the saturation problem in the intensity images. As shown in the simulation results, the proposed PPRNet significantly outperforms state-of-the-art (SOTA) deep learning-based PR methods. To understand the performance of the proposed PPRNet in practical applications, we constructed an optical setup for generating the intensity measurements of phase-only images obtained from 3 datasets. Similar to the simulation, the defocusing method was used to mitigate the saturation problem when preparing the datasets. They were then used for the training and testing of the proposed PPRNet. Based on these experimental data, we compared the proposed PPRNet with SOTA deep learning-based PR methods. A significant improvement in accuracy is noted. It also has lower complexity than other physics-driven PR methods.

To summarize, the contribution of this work is three-fold:

This paper is organized as follows: Section 2 provides a review of the traditional optimization-based algorithms and deep learning-based PR approaches. Section 3 introduces the proposed PPRNet. The simulation and experiment results with comparisons are shown in Sections 4 and 5, respectively. Finally, we conclude the paper in Section 6.

2. Related works

2.1 Optimization-based PR algorithms

In the early days, PR methods were mainly based on the iterative alternating minimization (AM) framework [1,2,7]. The estimated image $\tilde{{\textbf x}} \in {\mathrm{\mathbb{C}}^{N \times N}}$ is updated iteratively between the spatial and Fourier domain. Although the AM framework offers a portable solution, the AM-based algorithms are prone to stagnation and slow to converge (usually, more than 1000 iterations are needed). Besides, they are sensitive to initialization.

In recent years, the Wirtinger flow (WF) PR algorithm was developed with the advancement of modern optimization theories [8]. Different from the AM framework, WF solves the phase retrieval problem through gradient descent:

2.2 Deep Learning-based PR methods

In recent years, deep learning-based PR approaches have been widely studied since they provide non-iterative inferences compared with time-consuming optimization-based algorithms. Most of these approaches can work with only a single Fourier intensity measurement for each PR reconstruction without the need for any additional constraints on the measurement. This seemingly impossible task has a theoretical basis [30]. It is known that if the Fourier intensity measurement is oversampled by two or more times in each dimension, the original complex-valued signal can be uniquely reconstructed, except for trivial ambiguities. Although such a reconstruction problem is highly non-convex (it is the reason why the traditional optimization methods fail to perform), it is particularly suitable for deep learning-based methods due to their non-linear nature. Besides, they can also make use of the statistics acquired from a huge dataset to infer the solution. In general, deep learning-based PR methods can be split into two categories depending on whether the underlying physics is adopted in the networks.

For the first category, a feedforward network is used to estimate the target images directly from a Fourier intensity measurement [17,22–24,29,31–33]. Specifically, [24] proposed a two-branch CNN to reconstruct the magnitude and phase part from an oversampled Fourier intensity measurement, and [31] applied this network to the 3D crystal PR problem. The approach shows reasonably good performance for simple images with brief details. The performance when dealing with complex images is unknown. [22] applied the conditional generative adversarial network to reconstruct the images. The network is extremely large since multiple multi-layer perceptrons (MLP) are used for all stages of the network. [32] adopted the ResNet [34] structure for Fourier PR tasks. Only a simple dataset (MNIST) was used for testing, and the performance was only barely satisfactory. The error in the details was still rather large. [17] implemented a UNet structure [35] to reconstruct the phase-only images from Fresnel diffraction patterns. Its performance with Fraunhofer diffraction patterns (Fourier intensity) is unknown. As for [33], the authors adopted a multiple-resolution UNet structure and connected the hidden layers in the decoder to additional convolution layers to produce coarse outputs in an attempt to match the low-frequency components. Only the result of using 2 measurements is shown in the paper, and the blurring effect is quite significant, as shown in their result. Recently, our team also developed a feedforward DNN structure to tackle the PR problem [29]. It has an MLP front end for feature extraction and a residual attention-based reconstruction unit to generate the phase images. Although it outperforms most of the existing methods, its performance when dealing with more complex images still has room for further improvement.

To improve the quality of the reconstructed images, the second category of deep learning-based PR methods implicitly or explicitly utilizes the underlying physics in the models [19,21,25,26,36,37]. For instance, [36] made use of physics information to perform a learnable spectral initialization [8]. It is followed by a double branch UNet for reconstruction. The approach requires an additional masking scheme to impose constraints on the measurement. The reconstructed images are rather noisy, even for simple images. [23] proposed to use MLP of different sizes in a cascaded network. The intensity measurement is applied to each MLP to assist the training and inferencing. The network size is very large due to the use of multiple MLPs. And the approach fails to reconstruct the details in the images. There are other iterative approaches similar to traditional optimization-based methods. For instance, [26] suggested an iterative method with a 3-step structure: HIO initialization, iterative update between UNet and HIO, and final refinement by UNet. [21] proposed to embed a pre-trained DnCNN [38] into a plug-and-play iterative algorithm for refining the estimated images at each iteration. These iterative physics-driven approaches are usually quite time-consuming. Besides, they cannot be end-to-end trained, which often affects the overall performance.

3. Proposed approach

3.1 Defocus-based Fourier phase retrieval system

A Fourier phase retrieval system reconstructs an image ${\textbf x} \in {\mathrm{\mathbb{C}}^{N \times N}}$ from its Fourier intensity measurement ${\bf {\cal X}} \in {\mathrm{\mathbb{R}}^{M \times M}}.$ However, as mentioned in Section 2, the saturation problem often happens when using standard imaging devices to directly capture Fourier intensity images. It is due to the large dynamic range of Fourier intensity data. Besides, we can also find many zero-valued data in an intensity measurement. Most of them come from the high-frequency parts of the measurement. They have originally very small values but are quantized to zero, resulting in the so-called dead pixels. As a result, a Fourier intensity measurement often contains many errors in both the low-frequency and high-frequency regions.

Researchers have suggested a few solutions to the problem. One of them is the defocusing method [27–29] which is adopted in this study. Specifically, we reduce the dynamic range of the intensity measurement by convolving it with a defocus kernel ${\textbf D},$ described mathematically as follows:

 

Fig. 1. The optical path of the defocus-based PR system.

Download Full Size | PDF

3.2 Overall architecture

Figure 2 illustrates the overall architecture of the proposed PPRNet, which is designed to take a Fourier intensity measurement ${\bf {\cal X}} \in {\mathrm{\mathbb{R}}^{M \times M \times 1}}$ as input and the estimated real and imaginary images ${\textbf x} \in {\mathrm{\mathbb{R}}^{N \times N \times 2}}$ as its output. As mentioned above, the oversampling requirement needs to be fulfilled for unique reconstruction. So, in our experiment, M and N are set as 762 and 128, respectively. For the proposed structure, an initial guess $\hat{{\textbf x}} \in {\mathrm{\mathbb{R}}^{N \times N \times C}}$ is generated through the Init Block, where C is the number of channels ($C$ is set as 64 in our experiment). In the Init Block, a random image is first generated and refined by a HUB, which is a learnable physics-informed network structure (will be described in Section 3.4). The multi-scale structure has a contracting and expanding path, similar to [35]. To extract the essential features, the size of the feature map is gradually reduced by a number of DS Blocks in the contracting path. Each DS Block has three Convolution Blocks (ConvB) and each ConvB contains a convolution layer with $3 \times 3$ filters, plus an Instance Norm layer and a LeakyReLU activation function. The convolutional layers of the 3 ConvBs have strides 2, 1, and 1 such that the dimension of a feature vector is downsampled to half after passing through a DS Block.

 

Fig. 2. Architecture of the proposed PPRNet. The most significant output feature map after each HUB is shown at the bottom of the figure. The sizes of the feature maps are adjusted to be the same for display.

Download Full Size | PDF

In the expanding path, the US Blocks gradually enlarge the feature maps to the original size using the nearest neighbor interpolation function. Most importantly, the introduction of HUBs at various scales in the expanding path allows the reconstruction to be guided by the underlying physics. Note that we only have HUBs in the expanding path but not the contracting path since the role of HUB is to guide the reconstruction but not the encoding process. Finally, the PP Block, which contains two convolutional layers, reconstructs the real and imaginary images from the obtained feature maps. The proposed multi-scale structure contains skip connections to allow the expanding path to employ the fine-grained features discovered in the contracting path, much like the conventional multiscale structure does. Examples of the feature maps at various scales are shown in Fig. 2. Note that only the most important feature map for each scale is displayed. As the scale increases, it can be observed that the quality of the feature maps gradually improves.

3.3 Initialization

In classic optimization-based algorithms, initialization is an important step as it gives the optimization process a proper starting point. The optimization can avoid becoming stuck in unwanted saddle points with proper initialization [8]. Deep learning models are typically sensitive to the quality of the inputs. When initialization is done incorrectly, more noise is introduced, which lowers the performance of the deep learning model. To enhance the initialization process, we suggest incorporating it into the learnable network structure. First, we randomly generate a spatial image ${\tilde{{\textbf x}}_{init}} \sim \mathrm{{\cal N}}({0,1} )\in {\mathrm{\mathbb{R}}^{N \times N}}$ and feed it into a network that contains a 1 × 1 convolutional layer to convert the data into $C = 64$ channels. Note that the 1 × 1 convolution is commonly used in deep neural networks for feature size conversion. After that, they are delivered to a HUB to produce the first guess for the multi-scale network. The HUB provides a superior initial estimation of the target image than the conventional techniques that just depend on the existing physics knowledge. It is because it is physics-informed and trained end-to-end with the other components of the network. Figure 2 presents a case in point. It can be observed that the Init Block reconstructs the object’s overall shape. It is up to the following multi-scale structure to improve the estimation further.

3.4 Hybrid unwinding block (HUB)

HUB is the core building block of PPRNet. Figure 3 shows its structure. The input feature maps $\tilde{{\textbf x}} \in {\mathrm{\mathbb{R}}^{H \times W \times C}}$ are first divided into two processing branches (where H and W are set as N in our experiments). ${\textbf u} \in {\mathrm{\mathbb{R}}^{H \times W \times 2}}$ are the feature maps’ first two channels and are used by the Physics-driven Unwinding Block (PUB) as input. The Feature Refinement Block (FRB) then processes the remaining channels ${\textbf v} \in {\mathrm{\mathbb{R}}^{H \times W \times ({C - 2} )}}$ to give the local features. The outputs of PUB and FRB, designated as $\tilde{{\textbf u}}$ and $\tilde{{\textbf v}}$, respectively, concatenate with the input feature maps and are sent to the Feature Fusion Block (FFB). FFB makes use of the channel attention method to re-weigh these feature maps to extract the significant features for sending to the next stage.

 

Fig. 3. Structure of the Hybrid Unwinding Block (HUB).

Download Full Size | PDF

3.4.1 Physics-driven unwinding block (PUB)

Utilizing the intensity measurement as prior knowledge to constrain the optimization is a common strategy of conventional physics-driven PR approaches. However, since the constraint must be repeatedly applied to produce the desired effect, it frequently results in an iterative process. We suggest PUB, which converts the iterative procedure into a feedforward network operation, as a solution to the issue. The structure of PUB is depicted in Fig. 4. It is seen that a PUB consists of K unwinding cascading layers, where $K = 5$ in our experiment. For the $({k + 1} )$ tnwinding layer, ${{\textbf u}_k} \in {\mathrm{\mathbb{R}}^{H \times W \times 2}}$ forms the real and imaginary parts of the input feature map. They are first transformed to the Fourier domain as follows:

Then, we update $|{{{\textbf U}_k}} |$ with the intensity measurement in order to constrain the estimation. Note that HUB is utilized at various scales throughout the expanding path. We need to first convert the intensity measurement to the appropriate scale to apply it to the PUB of that scale. The updated ${\textbf U}_k^{\prime}$ is then transformed back to the spatial domain, creating an updated image ${\textbf u}_k^{\prime}$. The entire process can be stated as:

 

Fig. 4. Structure of the Physics-driven Unwinding Block (PUB).

Download Full Size | PDF

3.4.2 Feature refinement block (FRB) and feature fusion block (FFB)

Both global and local elements can be found in images. The PUB shown above seeks to constrain the reconstruction of the image using the Fourier intensity measurement. However, the Fourier transform can only reveal an image's overall frequency information. The local features in the image cannot be properly informed by the magnitude constraint alone. To improve the local processing power, we suggest a Feature Refinement Block (FRB) in addition to PUB to enrich the detailed structures corresponding to the local features. FRB has a shallow structure that contains 3 ConvBs. They can learn detailed representations such as edges, corners, etc., from the input feature maps. They supplement the global information from PUB to provide comprehensive representations for the reconstruction process. The output feature maps of PUB and FRB are concatenated with the input feature maps to form the resulting feature maps of size $({H,W,2C} ).$ However, these 2C channels of feature maps obviously have different significance to the reconstruction. To adaptively re-weight these channels based on their significance, we use a Feature Fusion Block (FFB), which is effectively a channel attention network [39] as illustrated in Fig. 5. We refer to [39] for the details of the operation of the channel attention network. After that, the weighted feature maps are integrated to form the output of HUB.

 

Fig. 5. Structure of the Feature Fusion Block (FFB).

Download Full Size | PDF

3.4.3 Visualize the effect of hybrid unwinding block

To gain a deeper insight into the operation of HUB, we show in Fig. 6 the feature maps generated by different components of the HUBs of the first two scales when inferencing an example image. The ground truth and the final estimated image are shown at the top. Let us start with the input feature maps for the HUB at Scale 2. The feature maps contain 256 channels, with the first two fed to PUB and the rest fed to FRB. The outputs of PUB and FRB are shown. For better visualization of the FRB branch, the average feature maps of all channels are presented. It can be seen that PUB tends to give the global structure of the image, and FRB tends to give the details. These feature maps are then combined by FFB using the channel attention method. For FFB, we show the two output channels with the largest weights (denoted as Max) and two with the smallest weights (denoted as Min). The feature maps given by the channels with the largest weights have already contained the basic structure of the ground truth. They are upsampled by the US Block and sent to Scale 1 for further enhancement. At Scale 1, the feature maps given by PUB are already very close to the ground truth. They are further processed to give the final output. To summarize, it can be seen in Fig. 6 that the FFB outputs continuously improve from the lower scale to the upper scale. The physics information applied to the PUBs at different scales has played an important role in this enhancement process. It guides the network to reconstruct the image in the right direction not only at the training stage but also at the inferencing stage.

 

Fig. 6. Visualization of the feature maps generated by different components of the HUBs of the first two scales. The example image is from the COCO dataset.

Download Full Size | PDF

3.5 Loss function

The phase retrieval challenge is treated as a supervised learning problem in this study. Given a Fourier intensity measurement ${\bf {\cal X}}$, there is a corresponding complex-valued image x that serves as the ground-truth reference. As a result, we can calculate the difference between the estimated image $\hat{{\textbf x}}$ and the ground truth image x. The loss function that we employed consists of two components: the pixel-wise loss ${\mathrm{{\cal L}}_{pixel}}$, which reduces the pixel-wise difference between the estimated and ground-truth images; and the total variation (TV) loss ${\mathrm{{\cal L}}_{TV}},$ which employs a smoothness prior to regularize the estimated image while preserving its edges and textures. The formulation of the total loss function $\mathrm{{\cal L}}$ is:

The TV norm sums all local horizontal and vertical gradients. By promoting spatial smoothness in the estimated image, TV regularization can help reduce noise and improve the consistency of the reconstructed image. ${\mathrm{{\cal L}}_{TV}}$ is defined as:

4. Simulation

4.1 Datasets

We performed extensive simulations to test the efficacy of the proposed PPRNet. To prepare the training and testing samples, we first gathered images from two publicly accessible datasets: the Real-world Affective Faces (RAF) dataset [41] and the Fashion-MNIST dataset [42]. The images were resized to $128 \times 128$ pixels and made grayscale. Then, we created two complex-valued image datasets (i.e., each target image has both the magnitude and phase parts) using two different combinations of these images. The Fashion-MNIST dataset's images were used to create the first dataset, of which the images have linearly correlated magnitude and phase components. Each image ${{\boldsymbol x}_{raw}}$ that was taken from the dataset was scaled to [0, 1] and used as the final image's magnitude component ${{\boldsymbol x}_{mag}}$. We then used an exponential function to obtain the phase component, i.e., ${{\boldsymbol x}_{phase}} = exp({2\pi i{{\boldsymbol x}_{raw}}} )$. Finally, we combined the magnitude and phase components by ${\boldsymbol x} = {{\boldsymbol x}_{mag}} \circ {{\boldsymbol x}_{phase}}$, where ◦ denotes the element-wise multiplication. To build our training and test set, we used the first 25,000 and 1,000 images from the Fashion-MNIST's training and testing datasets, respectively. The second dataset we constructed contains images with uncorrelated magnitude and phase parts. We utilized the same method as before to rescale the data, but to produce the magnitude and phase components using images from two distinct datasets, RAF and Fashion-MNIST. The training set contained 12,771 images, whereas the testing set contained 1,000 images.

4.2 Defocused intensity measurements

To simulate the defocused intensity measurements as discussed in Section 3.1, we multiplied all testing images with a defocus kernel, which was generated by a built-in function of Holoeye SLM control software corresponding to the defocusing distance of 30 mm. The same defocus kernel was used in the experiments on the optical platform, as will be discussed later. Note that other defocusing distance is also possible. Our experiment shows that a similar performance can be obtained with the defocusing distance set from 15 mm to 50 mm. Then, 2-D FFT was performed on these images, and the magnitudes of the resulting images were extracted to become the intensity measurements. The resolution of the measurements is 762 × 762 pixels. To simulate the saturation and quantization errors, we capped the measurements with the maximum limit of 4,095 (12-bit) and quantized the numbers to the integer format.

4.3 Training details and evaluation metrics

We used the Adam optimizer [43] to train our network. The batch size was set to 24, and the learning rate was set to ${10^{ - 4}}$. On a computer with two NVIDIA RTX3090 GPUs and the PyTorch deep learning platform, the network was trained for 160 epochs. The weighting factor $\gamma $ in the loss function (7) was empirically fixed to 0.1.

We evaluate the quality of the reconstructed images using the Peak Signal-to-Noise Ratio (PSNR, higher the better), Structural Similarity (SSIM, higher the better), and Mean Absolute Error (MAE, lower the better). They are commonly used in the field of image restoration to gauge the accuracy of estimates. To make sure that no negative values are used in the PSNR calculation, the phase parts of all images are shifted by π. For comparison, the average PSNR, SSIM, and MAE for the 1,000 testing images of both datasets are used.

4.4 Compared methods

Since the application domains of the optimization-based and learning-based approaches can be quite different, we only compare the performance of PPRNet with SOTA learning-based PR methods, namely PRCGAN [22], HIO-UNet [26], LenlessNet [17], NNPhase [24], and MCNN [33]. Note that some of these approaches are designed under much trivial input and output requirements as compared with those in this paper. For instance, NNPhase and PRCGAN assume the input intensity measurement is very small (only $64 \times 64$ and $28 \times 28$ pixels, respectively, as compared to $762 \times 762$ in the simulation environment). And PRCGAN assumes the target image only has real-valued data (as compared to having both real and imaginary data in the simulation environment). To allow these networks to adapt to the simulation environment, a small modification to these networks is required. Specifically, we modified the PRCGAN and NNPhase networks such that they could accept input of 128 × 128 pixels. The input data's dimension was then changed from $762 \times 762$ to $128 \times 128$ pixels by placing a pre-processing block in front of the original network. It has two convolutional layers with a 5 × 5 kernel and strides 3 and 4, respectively. Additionally, we modified HIO-UNet and LenlessNet's output to let them have two channels for real-imaginary representation, and PRCGAN's output to have two channels for magnitude-phase representation. These adjustments enabled these networks to function in the simulation environment.

4.5 Simulation results

Although the defocused intensity measurement is not the exact intensity measurement of the image, PR methods using the defocused intensity measurements usually perform better than using the original measurements with saturated and dead pixels. Some examples are given in Fig. 7. In the figure, we first show the performance of HIO [2] and prDeep [21], which represent the optimization-based and deep learning-based PR methods. They were implemented following the setting in their original papers (i.e., without saturation and dead pixels). The results are similar to those reported in their original papers, as shown in the second column of Fig. 7 and Table 1. Then, we introduced the saturation problem and dead pixels to the intensity measurements by capping the image data and quantizing them as 12-bit integers. It can be seen that the performances of these methods drop substantially when the imperfections in the measurements are taken into account. Finally, we used the defocusing method as mentioned above to mitigate the problem. As shown in the fourth column of Fig. 7 and Table 1, both approaches can significantly benefit from using the defocusing method. An improvement of up to 15 dB can be achieved. For this reason, we followed [27–29] so that all comparing methods were trained and tested with the defocused intensity measurements.

 

Fig. 7. Qualitative evaluation results of HIO [2] and prDeep [21] with and without using the defocused kernel.

Download Full Size | PDF

Table 1. Quantitative evaluation results of HIO [2] and prDeep [21] with and without using the defocused kernel. The results are the averages of 6 natural images used in [21].

View Table | View all tables in this article

The qualitative and quantitative comparison results are shown in Fig. 8 and Table 2. To save space, we do not include the qualitative results of LenslessNet in the comparison due to its subpar performance. Besides, we also include the result of HIO [2] just as a reference. As shown in Fig. 8(b), the proposed PPRNet outperforms all compared learning-based approaches for both datasets. Specifically, only PPRNet can reconstruct the details in the images, such as the “Lee” characters and the plaid on the shirt. The images given by other methods are of poor quality. On the other hand, as shown in Fig. 8(a), all competing algorithms struggle with the dataset containing uncorrelated magnitude and phase components. It is particularly the case for PRCGAN and HIO-UNet since they were originally intended to only output real-valued images. As seen in Fig. 8(a), there is a clear distortion in their magnitude images. The output images for techniques like NNPhase and MCNN are somewhat hazy and different from the intended images. These approaches also cannot reconstruct the details in the phase images. When compared to the aforementioned methods, the proposed PPRNet performs the best. The reconstructed magnitude and phase images closely follow the target images, especially the phase images. Occasionally, we can spot some artifacts in the magnitude images, but they are not serious.

 

Fig. 8. Qualitative simulation results of different phase retrieval methods on two datasets with images having (a) uncorrelated and (b) linearly correlated magnitude and phase components. For each image, the Fourier intensity measurements with pixel values ranging from 0 to 4,095 are shown in the first column. The ground truth of each target image is shown in the second column, with the magnitude part shown on top and the phase part shown right below it. The colormaps of the images can also be found. The third to eighth columns show the reconstructed magnitude and phase images through different methods. Zoom in for a better view.

Download Full Size | PDF

Table 2. Quantitative simulation results (average PSNR/SSIM/MAE) of different phase retrieval methods on two datasets.

View Table | View all tables in this article

In Table 2, the quantitative comparison is displayed. The proposed PPRNet outperforms all compared learning-based approaches measured using various metrics. Specifically, the proposed PPRNet achieves average PSNR and SSIM improvements of at least 5.818 dB and 0.146, respectively, for the magnitude portion. The increases in PSNR and SSIM gains for the phase component can be as high as 9.192 dB and 0.106, respectively. The aforementioned simulation results demonstrate that the proposed PPRNet outperforms SOTA learning-based PR techniques. We will show in Section 5 that the same conclusion can be drawn when testing these methods on an optical platform.

An ablation analysis of various design parameters of PPRNet is given in the supplementary document.

5. Experiment on optical platform

To understand the performance of the proposed PPRNet when applied to practical applications, we constructed an optical platform for collecting realistic intensity measurements for training and testing the network. The platform largely follows the optical path in Fig. 1. A snapshot of it is shown in Fig. 9. It comprises a Thorlabs 10 mW HeNe laser with wavelength $\lambda $ = 632.8 nm and a 12-bit 1920 × 1200 Kiralux CMOS camera with a pixel pitch 5.86 µm. We utilized a 1920 × 1080 Holoeye Pluto phase-only SLM with a pixel pitch ${\delta _{SLM}}$= 8 µm to generate the target images. The SLM can impose different phase shifts to the coherent light that gets through it. It effectively synthesizes the objects in phase imaging applications [12,17]. Since the ground truth is known, we can easily evaluate the accuracy of the reconstructed phase images given by different approaches. Another advantage of using the SLM is, similar to that in the simulation, we can directly multiply the defocus kernel with the images to generate the defocusing effect.

 

Fig. 9. Optical setup largely follows the optical path in Fig. 1.

Download Full Size | PDF

For the training and testing datasets, we used the same RAF and Fashion-MNIST datasets as in the simulation. Besides, we added the COCO dataset [43] (used the first 50,000 and 2,000 images of the training and testing dataset, respectively), which contains more complex images for challenging the PR approaches. We used the same approach as in the simulation to generate the phase part of the images based on the images collected from these datasets. The magnitude part was assumed as constant. Then these phase images were multiplied with the defocus kernel and loaded to the SLM. Following the optical path, the images captured by the camera were the defocused Fourier intensity measurements of these phase images. They were used for the training and testing of different PR approaches. The resolutions of these intensity measurements and phase images were the same as in the simulation, i.e., $762 \times 762$ and $128 \times 128$ pixels, respectively. When training the models, we first initialized the models with the ones we used in the simulation, as discussed in Section 4. Then we fine-tuned all the pre-trained models by re-training them for 120 epochs with the experimental measurements as the new training inputs.

Similar to the simulation, we compare the proposed PPRNet with the same set of learning-based PR methods. These methods were modified as in the simulation to suit the experiment environment. We evaluate the performance of all competing methods using the same metrics as in the simulation, i.e., PSNR, SSIM, and MAE. The quantitative and qualitative evaluation results are shown in Table 3 and Fig. 10, respectively. To save space, we do not include the qualitative results of ResNet, UNet, LearnInitNet, and CPR-FS in Fig. 10 due to their inferior performances. On the other hand, we include the result of HIO [2] just as a reference. Compared with SOTA deep learning-based PR methods, the proposed PPRNet can achieve PSNR and SSIM gains of at least 7.45 dB and 0.08, respectively, on the Fashion-MNIST dataset. It also performs the best on the RAF and COCO datasets. It can be seen in Fig. 10 that the reconstructed images of PPRNet can provide more details than other methods with fewer artifacts. It is particularly the case for the COCO dataset. Since the images are more complex, other approaches even cannot reconstruct the contours of the objects in the images. PPRNet can recover not only the contours but provide more textures of the original images. It is benefited from the physics information of the underlying image that guides the reconstruction process. Compared with other physics-driven methods, such as prDeep [21] and HIO-UNet [26], PPRNet not only provides better-quality images but also has a lower complexity, measured through the required GMAC operations. GMAC stands for Giga Multiply-Accumulation, which is a commonly used evaluation metric of the complexity of deep learning models. Specifically, the proposed PPRNet requires to perform 35.27 GMACs, while [21] and [26] require 1828.01 and 142.87 GMACs, respectively. The proposed PPRNet obviously has a much lower complexity. It is expected since PPRNet has a feedforward architecture that does not require iterative estimation. Besides, PPRNet is end-to-end trained. It is different from [21] and [26] which have an optimizer operated outside the network model. Thus, it does not have the potential mismatch problem between the optimizer and the network. Besides, prDeep is only successful for real-valued images, as claimed by the authors [21]. The result is unsatisfactory when it is used to retrieve phase images. The above experimental results have verified the performance of the proposed PPRNet when used in a practical phase retrieval system. It consistently outperforms SOTA deep learning-based PR methods both quantitatively and qualitatively. The proposed PPRNet also has a modest network size. In terms of the number of parameters, the proposed PPRNet is higher than only 3 out of 11 approaches in the list in Table 3.

 

Fig. 10. Experimental results of different phase retrieval methods on (a) RAF dataset [40], (b) Fashion-MNIST dataset [41], and (c) COCO dataset [43]. The first column and the second column show the Fourier intensity measurements (pixel values: 0–4,095) and the corresponding phase parts of ground-truth images, respectively. The values of the Fourier measurements are scaled for better visualization. The other columns show the reconstruction images through different methods. Except for the Fourier intensity measurements (the first column), the colormap of the rest columns ranges from 0 to 2π, with color bars shown in the second column.

Download Full Size | PDF

Table 3. Quantitative comparison (average PSNR/SSIM/MAE/Parameters/Complexity) with SOTA learning-based methods for phase retrieval on three datasets. The training and testing samples are collected with the optical system shown in Fig. 9. Best performances are marked in bold. The second and third-best performances are colored in blue and green, respectively.

View Table | View all tables in this article

6. Conclusion

In this paper, we proposed a deep learning-based PR method, namely PPRNet. Similar to other deep learning-based PR approaches, PPRNet requires only a single Fourier intensity measurement as its input and does not need an extra masking scheme for the PR system to constrain the measurement data. The main novelty of PPRNet is the introduction of physics information to the PR process. Unlike the traditional physics-driven PR methods that often end up in a time-consuming iterative procedure, the proposed PPRNet has a non-iterative feedforward structure but can still effectively utilize the intensity measurement to guide the image reconstruction process. It is enabled by the novel HUB embedded in the network’s input and expanding path. It separately processes the global and local information of the feature maps with the aid of the intensity measurement and combines them with a channel attention method. Our simulation and experiment results have verified the effectiveness of PPRNet. In particular, our experiment results were obtained from an optical platform. They demonstrate the performance of PPRNet when applied to practical phase retrieval systems. From the simulation and experiment results, it can be concluded that the proposed PPRNet consistently outperforms SOTA deep learning-based PR methods, proving it a promising solution to practical PR applications. Nevertheless, at the moment, there is still room for PPRNet to improve further when handling images with complex scenes. It is one of the ongoing works in our group.