Open Access
Add to BibSonomy
Abstract
As the core task of the reconstruction in conventional ptychography (CP) and Fourier ptychographic microscopy (FPM), the meticulous design of ptychographical iterative engine (PIE) largely affects the performance of reconstruction algorithms. Compared to traditional PIE algorithms, the paradigm of combining with machine learning to cross a local optimum has recently achieved significant progress. Nevertheless, existing designed engines still suffer drawbacks such as excessive hyper-parameters, heavy tuning work and lack of compatibility, which greatly limit their practical applications. In this work, we present a complete set of alternative schemes comprised of a kind of new perspective, a uniform design template, and a fusion framework, to naturally integrate Fourier ptychography (FP) with machine learning concepts. The new perspective, Dynamic Physics, is taken as the preferred tool to analyze a path (algorithm) at the physical level; the uniform design template, T-FP, clarifies the physical significance and optimization part in a path; the fusion framework follows two workable guidelines that are specially designed to keep convergence and make later localized modification for a new path, and further establishes a link between FP iterations and the gradient update in machine learning. Our scheme is compatible with both traditional FP paths and machine learning concepts. By combining ideas in both fields, we offer two design examples, MaFP and AdamFP. Results for both simulations and experiments show that designed algorithms following our scheme obtain better, faster (converge at the early stage after a few iterations) and more stable recovery with only minimal tuning hyper-parameters, demonstrating the effectiveness and superiority of our scheme.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Quantitative phase imaging (QPI) [1] has become a universal computational imaging technology due to the advantages of robustness and label-free imaging and has been applied in the fields of cells and tissues observation. With the developments of QPI, the conventional ptychographic (CP) imaging [2,3] and Fourier ptychographic microscopy(FPM) [4] as the implementation techniques of QPI receive substantial attention of the scholars. The CP imaging utilizes aperture overlap and probe scanning to quantitatively recover the original intensity and phase information and has become a powerful tool with a broad range of applications [5–9]. Unlike CP imaging, FPM uses a LED array as the light source and a low-magnification objective to achieve aperture scanning and synthesis in the frequency domain. Furthermore, FPM broadens the space-bandwidth product (SBP) and equivalent numerical aperture (NA) of an optical system [10,11]. In recent years, both of the CP and FPM techniques have been widely studied in terms of imaging mode [12–17], error correction [18–21] and iterative optimization [22–26], etc.
Although the hardware composition of the systems of CP and FPM are different, the CP and FPM share a similar core in the field of computational imaging- ptychographical iterative engine (PIE) [2,15,22,25]. Originally, the PIE and basic FP based on GS framework were designed to obtain intensity and phase information of the target in the field of view (FOV). But in actual circumstances, these algorithms are implemented on a series of assumptions such as, ignoring the thickness of sample, partially coherent light, assuming an ideal (0-1) low-pass filter for the pupil, neglecting noise, and etc [21,24,27–29], which greatly limit the application of CP and FPM [30,31].
Later, ePIE [22] and EPRY [23] were proposed, which improved the robustness of PIE and further relaxd the required conditions in practical hardware. The designs of ePIE and EPRY place the pupil function and sample spectrum in the same key position. Employing a symmetric-designed route and the alternative projection method [32], the sample spectrum and pupil function are recovered simultaneously and accurately. Nowadays, ePIE and EPRY are still two of the most widely used algorithms in CP and FPM studies [19,21,31,33]. However, these two algorithms still have some shortcomings: Firstly, since the designs of ePIE and EPRY are based on the traditional gradient descent method [34], it is easy to fall into a local optimal solution [35], which also exsits in almost current traditional PIE and FP algorithms. Secondly, to ensure convergence, the step length adopts the maximum value as the denominator term, which also reduces the reconstruction speed [25]. Thirdly, the algorithms are susceptible to noise.
To further improve the performance of PIE algorithms, rPIE and mPIE were proposed in [25]. For rPIE, it combines naturally PIE and ePIE in the form of a convex combination to enhance the optimization-seeking ability. For mPIE, it is the first successful attempt to combine ptychographic theory with the concepts of machine learning. The mPIE adds a first-order momentum acceleration module [36] on the rPIE and becomes the best algorithm at present [10]. Compared with other PIE algorithms, mPIE has made great achievements in convergence speed and reconstruction performance but at the expense of extra hyper-parameters and heavy tuning work. Moreover, During the tuning process, large parameter settings easily results in the failure of mPIE reconstruction in the early stage.
Despite these difficulties in mPIE, mPIE introduces a new field, machine learning, into the design of future algorithms. Inspired by it, in this article, we propose a complete set of alternative scheme to integrate FP with advanced machine learning concepts.
First, we revisit the path design of previous excellent algorithms in depth with a new perspective - Dynamic Physics, by treating the iterative design of the algorithm as a dynamic physical motion process. In this view, we suggest that the overall design of the iterative algorithm is determined not only by the position change in each iteration, but also by the inner factors of speed and duration designs. During the process of revisiting, we provide a guiding template for FP design, T-FP, which specifies the optimization objectives and the physical significance in a path.
Second, we further give an alternative framework that combines FP paths with the ideas of machine learning. In the framework, we design two workable guidelines to keep a new path convergence and make later localized modification. One is the Same-order criteria, which is applied to ensure the convergence of a new path. The other is the Background criteria, which is utilized to guide some local modifications for FP with machine learning. Furthermore, with Dynamic Physics and T-FP, we construct a link to combine FP iterations with the gradient update of machine learning in the framework.
Finally, we provide two design examples to show the compatibility of our scheme, MaFP and AdamFP (Momentum acceleration [36] Fourier ptychography and Adaptive momentum estimation [37] Fourier ptychography). MaFP reduces the tuning parameters of mPIE to two and achieves comparable performance in the simulations and experiments. AdamFP is not only a fully adaptive design for duration and direction with only one tuning parameter, but also overcome the shortcomings of rPIE-based paths. The simulations and experiments also show that AdamFP always find and stabilize an optimal solution quickly (at the early of stage) and behaves much more robust to noise than those of other paths, whose stable performance outperformed all existing paths.
2. Principle
2.1 Overview of the basic FP model
Conventional FPM implementation can be divided into two independent steps: image acquisition and image reconstruction [4]. During image acquisition, a LED array with $M \times N$ units is placed at the horizontal plane below the sample to supply illumination at various oblique angles. Assuming that the distance $h$ from the light source to the thin sample is sufficiently large, the illumination from a single LED unit (in row $m$ and column $n$) can be regarded as an ideal plane wave with central wavelength $\lambda$ and wave vector $k_{m,n}=(k_{x,m,n},k_{y,m,n})$,
During image reconstruction, each of the intensity measurement meets the support constraint in the frequency domain, to update the sub-spectrum within each aperture. Starting with an initial estimated sample spectrum $O_{0}$ and pupil function $P$, an iterative reconstruction framework is crafted to realize image recovery. Specifically, when updating the information within a sub-aperture during one iteration, the sub-spectrum is picked up and converted to the complex amplitude $\varphi$ in the spatial domain:
The above is the theoretical framework of basic FP. For later subsection, focused on Eq. (7), we will review the progressive construction of typical reconstruction algorithms in FP in depth with a new perspective.
2.2 Overview of evolving reconstruction algorithms in FP
The design of PIE is normally derived from a well-designed loss function (error evaluation function) with gradient descent. This design paradigm concentrates most of the effort on the design of the loss function. On the basis of the previous work, A. Maiden et al. [25] offers three different perspectives to reexamine the PIE family of algorithms at the mathematical level.
In this article, we provide a novel perspective at the physical level on analyzing and designing iterative updating formulas in FP, named Dynamic Physics, which regards the update (optimization) process as a physical process of dynamic motion. The iterative process of a reconstruction algorithms will be given some new physical significance. Aided by the simplest gradient descent method in machine learning, one update process is described as:
where $O$ is the objective to update, with current state $O^{0}$ and updated state $O^{1}$. ${\nabla _O}J(O^{0})$ denotes the gradient from objective function $J(O)$, and $\eta$ is the learning rate. From the perspective of Dynamic Physics, the above update process indicated in Eq. (9) can be interpreted as one move: the objective $O$ starts from position $O^{0}$, moves with speed ${\nabla _O}J(O^{0})$ and duration $\eta$, and finally reaches position $O^{1}$.Next, we continue to revisit evolving reconstruction algorithms in FP through Dynamic Physics. During this process, we focus on clarifying the speed (the magnitude and direction) and duration for each move, not just the position change ($O^{0}$ to $O^{1}$), which is necessary in this article for how to propose a new design for FP.
In basic FP, since $P$ can be considered as an ideal low-pass circular filter, Eq. (7) can be reformulated as:
Fig. 1. The schematic diagram of the view of Dynamic physics in FP. The content in the upper of the picture is the principle analysis and the bottom is the supplementary explanation. The cyan box indicates a spectrum update within the sub-aperture $P_{m,n}$. The brown box indicates the decomposition of basic FP and T-FP in our view. Where $v_1$ marked by the red arrow is the direction of sub-aperture movement, $v_2$ marked by the yellow arrow is the self-updating direction, and $\overline {t}$ is the duration in a single movement.
In the Dynamic Physics, FPM has two unique characteristics: i) once the intensity measurement has been captured and the initial estimates $O_{0}$ and $P$ have been set, the magnitude of moving speed would be determined as $\Phi _{m,n}^{i}(k) - \varphi _{m,n}^{i}(k)$ (more strictly, $\frac {\Phi _{m,n}^{i}(k) - \varphi _{m,n}^{i}(k)}{\left | P_{(m,n)-1}(k+k_{m,n}) \right |}$); ii) the update direction of pupil function $P(k)$, which is the direction of sub-aperture movement $P(k + {k_{m,n}})$, is the main component of moving speed (in basic FP, it is the only direction for $P$). In most cases, the update order of sub-apertures is determined in advance.
Obviously, duration $1$ is far from a wise choice. Due to the difference between the bright and the dark fields, using a fixed and long duration may lead to an unpredictable result. Specifically, some practical uncertainties such as noise, system errors, and etc., could make the recovery of basic FP less reliable.
To relax the conditions in practical FPM systems, the EPRY algorithm recovers both the sample spectrum and the pupil function through alternative projection without the priori knowledge of the aberration. The update process of EPRY is expressed as:
In the Dynamic Physics, compared to basic FP, the initial position and magnitude of the speed remain the same, while the direction evolves from $P(k+k_{m,n})$ to $P_{(m,n) - 1}^{*}(k + {k_{m,n}})$, and the duration changes from $1$ to $\frac {\alpha \left |P^{*}(k+k_{m,n})\right |^{2}}{{\left | {{P_{(m,n) - 1}}(k + {k_{m,n}})} \right |_{\max }^{2}}}$. In the Eq. (11), the sample spectrum $O$ and the pupil function $P$ move meanwhile. The direction of move for each individual in the aperture is modified from a static $P$ to a dynamic $P^{*}$. That is, the update of $P$ involves not only the pre-designed aperture update direction (static direction) but also the self-update direction (dynamic direction), which enhances the merit-seeking ability.
Furthermore, $\frac {\alpha \left | P(k+k_{m,n})\right |^{2}}{{\left | {{P_{(m,n) - 1}}(k + {k_{m,n}})} \right |_{\max }^{2}}}$ is selected as a dynamic adjustment factor for duration. It is a normalized multiplicative term that satisfies the practical physical situation and whose value is positively correlated to $P$. That is, when $P$ is small, the luminous flux of this area is little. Hence, the pixel is easily disturbed by environmental factors such as noise and errors. In the case of inaccurate speed, it is a better option for a single move to produce a small shift with a small duration, and vice versa.
On the basis of the above advantages and the pattern of EPRY, we provide a template of FP (T-FP) expressed as:
Nevertheless, there are still some disadvantages for EPRY. First, the design of move path in EPRY is actually based on the gradient descent method, which always faces the risk of falling into a local optimization. Second, there is no significant difference between the basic FP and EPRY in terms of duration design. Moreover, the selection of denominator $\left | {{P_{(m,n) - 1}}(k + {k_{m,n}})} \right |_{\max }^{2}$ in Eq. (11) reduces the duration, which may slow the convergence in some cases. Finally, comparing the Eq. (11) and Eq. (12), only the numerator term $P^{*}(k+k_{m,n})$ guides the directions of speed and duration, whereas the denominator term ${\left | {{P_{(m,n) - 1}}(k + {k_{m,n}})} \right |_{\max }^{2}}$ in the Eq. (11) regulates the overall scale. Furthermore, only the numerator $\left |P^{*}(k+k_{m,n})\right |^{2}$ in the template is limited to characterize the direction, so it becomes a key task to build a suitable denominator term ($f$) to generalize directions while balancing the scale.
To further improve the PIE, mPIE has been developed. First, the design of the denominator is implemented as a convex combination of PIE and ePIE, which is named as rPIE. Specifically, the denominators in T-FP are designed as:
Second, the idea of machine learning is introduced into PIE. To help the motion subject to cross a local pit, additional cumulative momentum is directly added to the current position after a fixed number of movement $T$. The accumulated momentum during the $T$ movements is:
In [41,42], the experiments have demonstrated that incorporating the momentum acceleration concept improved the convergence speed and performance of PIE. However, some details need to be discussed in depth. First, the good results comes at the price of more hyper-parameters, which often requires the designer or user to solve a manual-tuning problem. Unfortunately, these fixed hyper-parameters $(T,\eta _{obj},\eta _{p},\alpha,\beta,\gamma _{obj},\gamma _{p})$ are not universally applicable to various situations. Second, the periodic use of the momentum acceleration module prevents the motion subject from falling into local pits. By the derivation for the momentum $v$ in Eq. (14,15), the core of momentum maps is re-stacking of the paths during the previous $T$ movements in the exponential form of $(\gamma \eta )$. Although it follows the forward directions of the previous movements and provides enough energy to cross a pit, it may affect the original paths of normal FP in turn. Once the momentum is attached to the current position, the momentum acceleration is equivalent to an increase in duration of each path within the previous $T$ movements, which is catastrophic for FP in some cases and makes troubles in manual tuning. Finally, from the view of Dynamic Physics, momentum acceleration module in mPIE is achieved by the update of speed determined by the change of position, which is inconsistent with the physics view of motion.
2.3 Fusion framework of FP and machine learning
In this section, we present a complete framework that integrates FP with the idea of machine learning, as indicated in Fig. 2. In the framework, on the basis of the T-FP and perspective of Dynamic Physics, two workable guidelines are designed to keep convergence and make later localized modification:
- i) Same-order criteria. The convergence in path programming requires that in the alternating updates for $O$ and $P$, each motion of one should keep the overall order of the other to be 0. That is, when updating $O$ in one movement, the order of $P$ keeps 0 and vice versa.
- ii) Background criteria. The ideas of FP and machine learning are from two different domains, in which the unique characteristics are kept for later localized modification.
Fig. 2. The flowchart of the fusion framework. Flowchart shows the overall process in a single path. In the flowchart, the black texts mark the existing theoretical part in FP, and the red texts mark the theoretical part in this article.
Here, we offer two path designs named MaFP and AdamFP, which integrate FP with current popularly used machine learning ideas-Ma (Momentum acceleration [36]) and Adam (Adaptive momentum estimation [37]). Ma is a method commonly adopted for accelerating the training of the neural network parameters. Adam is a parametric training method that provides an adaptive learning rate, which is widely employed in the field of computer vision and natural language processing. The flowcharts of MaFP and AdamFP are shown in Fig. 3.
As illustrated in T-FP, $\Phi _{m,n}^{i}(k) - \varphi _{m,n}^{i}(k)$ has offered the magnitude of speed and $P_{(m,n) - 1}^{*}(k + {k_{m,n}})$ has provided the self-update (First-order linear) and aperture update directions. They are both fixed and should not be modified any more because of the nature in FP. However, the first mission to be concerned with designing a single path is the speed. Therefore, the main task is to design a new nonlinear direction and scale modulation terms. Inspired by Adam in machine learning, we design an alternative solution.
For AdamFP, the first-order moment begins at position $t=0$ and guides the direction of a single path, while training the ability to cross a local pit:
The second-order moment balances the scale of duration and ensures the convergence:
To prevent the moments from biasing towards zero in the early stage, we make a bias correction for them:
In Eq. (16–18), the experience values of $\beta _1$ and $\beta _2$ for $O$ are recommended to be 0.9 and 0.9 (or 0.99), respectively.
Next, to combine with T-FP, we construct a proper direction ${\vec {g}}$ of ${g_{t}}$. Especially, when we specify the task in design from the view of Dynamic Physics, one of the solutions is:
where $g$ denotes a function relationship, $\vec {\cdot }$ indicates the direction operator and $f$ has been defined in Eq. (12). Furthermore, we can also demonstrate its rationality from the perspective of updating the gradient. Previously proposed paths of the basic FP, PIE and EPRY are designed to serve the loss function via gradient descent, which deduce an iterative form of $O$. However, our purpose is to design a single path, the service-subject is $O$ (Position) rather than the loss function and the target is $P_{(m,n) - 1}^{*}(k + {k_{m,n)}} (\Phi _{m,n}^{i}(k) - \varphi _{m,n}^{i}(k))$, which is a fixed speed in FP. Hence in T-FP, the value of gradient is the reciprocal of denominator term at position $t$. Undoubtedly, $g$ acts as a bridge that connects the machine learning with the FP framework.As instructed in T-FP, the current optimal design of $f$ is the denominator term in rPIE of Eq. (13). Then, the corresponding construction of $\vec {g}$ and the update of $O$ are given, respectively:
However, it conflicts with the first criterion in the framework. Worse yet, the term $\frac {{{{\hat m}_{p,(m,n)}}}}{{\sqrt {{{\hat v}_{p,(m,n)}}} + eps}}$ is not actually pure zero-order of $\vec {g}_t$ and the balance of external second-order terms is invalid. Moreover, we find that the problem reflects not only on the appearance of order imbalance, but also on the neglect of context in machine learning, which conflicts with the second criterion. Specifically, the processed sample in its field is often standardized and dimensionless, so the order balance is not required when updating a new increment. Hence, we provide an alternative construction on the direction of $g_t$:
This pattern brings two benefits in path design. First, $g_t$ has thoroughly become a normalized zero-order term for $P$. Second, the numerator term of $g_t$ is ${\left | {{P_{(m,n) - 1}}(k + {k_{m,n}})} \right |}^{2}$, which is superior to that of rPIE indicated in Eq. (20). Then, $\frac {{{{ \hat m}_{p,(m,n)}}}}{{\sqrt {{{\hat v}_{p,(m,n)}}} + eps}}$ in Eq. (21) has become a zero-order term for $P$. Now, we assign the modulus of current position to Eq. (22) and the $g_t$ is represented as:
According to the first criterion, we need to add a second-order term on $P$ outside to keep the order zero in one movement. Considering the nature of FP, the main directions in T-FP need to be maintained. The Eq. (21) is reconstructed as:
Similar to the analysis of $O$, the path of pupil function $P$ is derived as:
To summarize, the duration in AdamFP is adaptive, which solves the problem of a fixed duration during the whole path to adapt to the fast updating in the early stage and slow updating in the late stage of FP. However, in practice, the noise is really existing and it accumulates with the renewal of momentum continuously, which gradually dominates the path as the subject plateaus. Depending on the selected duration, there are two possible consequences in the end:(1) The subject ’reconciles’ with the noise and image degrades to some extent; (2) The outcome is completely degraded.
To reduce this effect, we apply a duration decay strategy to constrain the magnitude in different stages. Specifically, in Eq. (24,25), we employ two hyper-parameters, $\gamma _{obj}^{i}$ and $\gamma _{p}^{i}$, in the path to regulate the duration of each movement. The uniform form of $\gamma ^{i}$ is given by:
where $\eta$ here is a decay factor that requires manual tuning and the recommended range of values is 0.9 to 1, which depends on the difficulty of reconstruction and the degree of noise or other errors. $[\cdot ]$ indicates the Gaussian rounding operator.The above is overall path programming with the conception of AdamFP. Next, integrating the momentum acceleration in machine learning, we develop another path, MaFP, with T-FP and the same structure of $g_{t}$.
In AdamFP, the construction for $g_{t}$ in Eq. (23) has actually satisfied the form of T-FP as well as the first criteria. Then, as long as we assign the original direction of speed in FP to $g_{t}$ (the second criteria), the momentum acceleration module in MaFP is designed as the first-order moment in AdamFP. The expression for $g_t$ in the path is give by:
For Eq. (27), the $g_t$ acts directly the role of increment in FP. Moreover, the equation returns to the design of rPIE, which leads the same goal as mPIE but via different roads. Since there is no design for scale balance, we can only artificially adjust their overall updates. In MaFP, the hyper-parameters $\gamma _{obj}$, $\gamma _{p}$ in T-FP are kept in a single path and the update of MaFP is expressed as:
where the values of $\gamma _{obj}$ and $\gamma _{p}$ is not fixed and often set as 0.2 and 0.2, respectively. In MaFP, the decay coefficients of first-order moment for $O$ and $P$ are recommended as 0.9.As seen from the previous analysis through Dynamic Physics, T-FP is more important than the designed path itself. In addition, we have showed the strong compatibility of our design pattern with traditional FP paths or advanced machine learning concepts. Next, we give the related simulations and experiments to demonstrate the effectiveness and superiority of our design concept. Meanwhile, we compare our results with those of algorithms mentioned before.
3. Results
In this section, we perform not only a set of simulations but also the experiments under a public dataset. The system parameters of the optical setup and the experimental parameters are set in Table 1.
Table 1. The parameters of the simulations and experiments
3.1 Simulations
In the simulations, two normalized images with 512$\times$512 pixels named ’Baboon’ and ’Aerial’ respectively are well-chosen as the high-resolution intensity and phase maps, which contain rich layers and details, as shown in Fig. 4(a) and Fig. 4(b). During the imaging process, assuming the system noise is Gaussian noise with the mean of 0 and standard deviation of $5\times 10^{-5}$. The noisy low-resolution image of the central LED is shown in Fig. 4(c) .
Fig. 4. The sample in the simulations. (a) the intensity map of sample. (b) the phase map of sample. (c) the noisy low-resolution image.
According to the perspective of update function in [25], we set two groups of simulation parameters on $\alpha$ and $\beta$ to show the stability and optimization capacity of various algorithms under the same benchmark. These two hyper-parameters represent the preservation of main directions in momentum updates, which are held as $\alpha =0.15, \beta =1.5$ (recommend in mPIE) and $\alpha =0.01, \beta =0.01$ (manually set), respectively. For AdamFP simulations, the decay factor $\eta$ is 0.9. To represent the outcome of each iteration and reduce the span of values in evaluation, the structural similarity (SSIM) is employed as the metric.
In the first group of $\alpha =0.15, \beta =1.5$, the hyper-parameters in mPIE are set as the suggested values of $(T,\eta _{obj},\eta _{p},\alpha,\beta,\gamma _{obj},\gamma _{p})=(50,0.6,0.85,0.15,1.5,0.15,0.25)$. The simulation results of the first group are indicated in Fig. 5, of which (a) presents the results of reconstruction process during 200 iterations and (b)-(f) represent the reconstruction results of those algorithms corresponding to the colors in (a). In the early stage, the SSIMs of AdamFP, MaFP and mPIE are superior to those of traditional gradient descent algorithms of rPIE and EPRY by momentum crossing the local optimums. AdamFP obtains an optimal solution with the fastest speed of merit seeking. mPIE approximates the optimal solution quickly due to the cumulative effect of momentum ($T=50$) but at the cost of persistent oscillations and degeneracy in the medium stage. MaFP approximates the optimal solution smoothly but fluctuating slightly throughout the iterative process. Compared to EPRY, rPIE finds a local optimum superior to EPRY because of the introduction of a new direction. In the mid-to-late stage, mPIE and rPIE are hardly to ’settle’ with noise and gradually deviate from the optimum to severe degradation. EPRY falls into a local optimum and tends to stabilization. By contrast, AdamFP and MaFP eventually stabilize near the optimal solution, which are more robust to the noise.
Fig. 5. Simulation results in the first group. (a) is SSIM over 200 iterations of EPRY, rPIE, mPIE, MaFP and AdamFP algorithms. (b)-(f) are the recovered results of those algorithms corresponding to the color in (a), where the first row and second row show the distribution of intensity and phase,respectively.
In the second group of $\alpha =0.01, \beta =0.01$, these suggested hyper-parameters in mPIE are no longer adapted to this case. As described in Section 2.2, a large cumulative duration $(\gamma \eta )$ is easy to create a failure in reconstruction. After many tests, we summarize a simple tuning rule in mPIE, which reduces the value of $(\gamma _{obj},\gamma _{p},\eta _{obj},\eta _{p})$ when mPIE does not converge or obtain an obviously incorrect outcome. Here, we provide a set of optimal parameters of $(T,\eta _{obj},\eta _{p},\alpha,\beta,\gamma _{obj},\gamma _{p})=(15,0.1,0.1,0.01,0.01,0.01,0.1)$ for comparison. The reconstruction of MaFP uses the same ($\gamma _{obj},\gamma _{p}$) as mPIE. The simulation results of the first group are indicated in Fig. 6, of which (a) presents the results of reconstruction process during 200 iterations and (b)-(e) represent the reconstruction results of EPRY, mPIE, MaFP and AdamFP corresponding to the colors in (a). In this case, rPIE fails to reconstruct and two first-order moment-based algorithms of mPIE and MaFP show the degradation as a result of loss in main directions support. For EPRY, the merit-seeking process is extremely slow. But, EPRY can find at least one locally optimal solution without the reconstruction failure. However, AdamFP continues to perform as well as in the first group, which reconstructs with the fastest speed of merit seeking and converges stably.
Fig. 6. Simulation results in the second group. (a) is SSIM over 200 iterations of EPRY, rPIE, mPIE, MaFP and AdamFP algorithms. (b)-(e) are the recovered results of those algorithms corresponding to the color in (a), where the first row and second row show the distribution of intensity and phase,respectively.
From the two groups of simulations, the selection for speed and duration is important to design a new path. Although AdamFP is adaptive to directions and duration, the different selections of $(\alpha,\beta )$ in MaFP and mPIE still lead the algorithms converging to different endpoints. Here, we recommend $(0.15,1.5)$ as a set of good directions for MaFP, in which only two tuning-parameters is required to obtain results rivaling those of mPIE.
Furthermore, we add five groups of simulations with different levels of noise in the imaging process to validate the stability of our paths against noise. The added Gaussian noises have the mean value of $0$ and standard deviations from $2\times 10^{-5}$ to $1\times 10^{-4}$ with an interval of $2\times 10^{-5}$. All involved paths are iterated 200 times to obtain overall evaluation of performance.
In these simulations, the directional hyper-parameters are still taken as the recommended parameters $(\alpha,\beta )=(0.15,1.5)$. For mPIE, the tuning parameters are $(T,\eta _{obj},\eta _{p},\alpha,\beta,\gamma _{obj},\gamma _{p})=(15,0.1,0.1,0.15,1.5,0.1,0.1)$. For MaFP, the duration-adjusted parameters are taken as $(\gamma _{obj},\gamma _{p})=(0.1,0.1)$, kept the same as mPIE. For AdamFP, the duration decay parameter is taken as $\eta =0.95$ under the condition that the standard deviation of the Gaussian noise is $2\times 10^{-5}$ and $\eta =0.9$ for the rest of the conditions.
Fig. 7. Simulation results of the intensity recovery at different stages of reconstruction in evaluation. (a), (b), (c) and (d) plot the evaluated results for 15, 50, 100 and 200 iterations of the involved paths at different levels of noise, respectively. In each subplot, the horizontal coordinate indicates the standard deviation of the added Gaussian noise, and the scale is exhibited in the parenthesis.
Figure 7 plots the comparative results of the intensity recovery at different stages of reconstruction in evaluation. Obviously, the paths optimized with machine learning perform better than the traditional paths. For EPRY, in a low-level noisy environment, it can yield an acceptable recovery. However, as the level of noise increases, EPRY falls into a local optimum solution. For rPIE, due to the introduction of the new direction and the setting of the large overall duration, it is always possible to obtain an expected recovery at the early stage, but at the cost of difficulty in resisting the noise afterwards, resulting in a severe degradation of the recovery. For mPIE and MaFP, they obtain similar performance. Thanks to the usage of small duration-tuning hyper-parameters and the momentum acceleration modules, the paths of mPIE and MaFP avoid the consequence of severe degradation in rPIE and gain the ability to cross local optima that EPRY does not have. From (c) and (d), it can be seen that the performance of mPIE and MaFP in the middle and late stages is no longer significantly different from that of AdamFP. For AdamFP, it always acquires an optimal recovery in the early stage of reconstruction and stabilizes in its vicinity. These conclusions demonstrate the stability of our designed paths in the scheme.
3.2 Experiments
To further validate the effectiveness of our path design, we perform a uniform test for the algorithms mentioned above on a public dataset in [11]. For AdamFP experiments, the decay factor $\eta$ is 0.96. The experiments are conducted in two groups and involved algorithms are iterated 100 times.
In the first group, for the named ’IHC’ data, the targets are numerous and stacked on each other, which causes the path highly susceptible to noise. In this group, $(\alpha,\beta )$ are selected as the suggested values of $(0.15,1.5)$. Other tuning parameters of mPIE are $(T,\eta _{obj},\eta _{p},\gamma _{obj},\gamma _{p})=(15,0.1,0.1,0.1,0.1)$. In MaFP, the parameters $(\gamma _{obj},\gamma _{p})$ to control the duration are set the same as $(0.1,0.1)$. Figure 8 illustrates the experimental results of algorithms for blue channel ($475nm$). Traditional path design of EPRY and rPIE has bad performance due to defects of conventional gradient descent and uniform duration setting, resulting in paths vulnerable to noise accumulation and local optimal impact. By contrast, path design of mPIE and MaFP, with the first-order momentum optimization achieves good and similar performance at the cost of requiring an small magnitude of uniform modulation for duration to prevent path collapse at the early stage. The path design of AdamFP achieves an optimum performance among the five algorithms. The accumulation effect of noise is bound to affect the path to some extent without adaptive selection for duration, which is consistent with the analysis in Section 2.3.
Fig. 8. Performance of the different paths on the ’IHC’ data in the blue channel. (a) is the central low-resolution image, where the region of interest is marked with a red box. (b)-(f) are the recovered results with EPRY, rPIE, mPIE, MaFP, and AdamFP, where the red box marks the region of interest with the same field of view as in (a). Among (b)-(f), the first and third rows display the overall results of intensity and phase, respectively. The second row offers the results of intensity recovery in the field of view marked by the red box.
Furthermore, Fig. 9 presents the comparison of results with AdamFP, MaFP and mPIE at different stages of reconstruction. It can be judged that after 50 iterations of mPIE and MaFP with the first-order momentum optimization, the contours of cells become clear and algorithms gradually converges. For the iterations of AdamFP in Fig. 9, it achieves the fastest merit-seeking convergence before 15 iterations.
Fig. 9. Comparison of results with AdamFP, MaFP and mPIE at different stages. The colored boxes mark the estimated timing for convergence.
In the second group, the central wavelength of the illumination is $630nm$. For the named ’blood_aberration’ data, the targets are disturbed by aberrations, of which pre-calibration has been done before reconstruction. Then, we test the performance when the $(\alpha,\beta )$ are set as $(0.15,0.15)$. Here, we modify the duration parameters $(\gamma _{obj},\gamma _{p})$ of mPIE to $(0.3,0.3)$ and keep other parameters unchanged. Figure 10 illustrates the performance of the different paths at the different stages. Due to the destruction of main directions, the capability of aberration correction is gradually lost with iterations for all paths designed based on rPIE, even if the aberration is pre-calibrated initially. The timing of aberration reflected from the performance in the three paths of rPIE, mPIE and MaFP is 15 iterations in rPIE, 50 iterations in mPIE and 100 iterations in MaFP, respectively. Correspondingly, the reason for this phenomenon is the adopted different unified duration $(\gamma _{obj},\gamma _{p})$ in paths, those are $(1,1)$ in rPIE, $(0.3,0.3)$ in mPIE, $(0.2,0.2)$ in MaFP. In this case, the aberration will certainly accumulate with the iterations, which is due to the selection for duration that determines how quickly the aberration is reflected in the results. By contrast, AdamFP achieves the convergence before 15 iterations without aberrations and the main directions are kept meanwhile, which is consistent with the path design for AdamFP.
Fig. 10. Performance of the different paths on the ’blood_aberration’ data at the different stages. The colored boxes mark the estimated timing for the appearance of aberration effect. (a) is the central low-resolution image. From top to bottom, (b)-(e) show the results of rPIE, mPIE, MaFP and AdamFP at 15 iterations, 30 iterations, 50 iterations and 100 iterations, respectively.
4. Conclusion
In this article, we analyze the path design process of the algorithms in FP with a new perspective, and introduce how to integrate a kind of machine learning concepts into current FP algorithms and design two new paths named MaFP and AdamFP, which achieve better performance compared with the state-of-the-art algorithms. Furthermore, relevant simulations and experiments have demonstrated the effectiveness and superiority of our design paths. Finally, we present a complete set of alternative scheme to integrate FP with machine learning for future path design.
In the scheme, we focus on the speed and duration in the path design with a new perspective of Dynamic Physics. Besides, we further design a uniform template of T-FP to clarify the physical significance and optimization part in the path design. Based on the above works, we give a fusion framework to integrate FP with machine learning. It not only has strong compatibility with machine learning concepts and FP paths, but also achieves better performance with fewer tuning parameters compared with mPIE. However, there are some details to be discussed. Firstly, how to determine the optimal combination of direction $(\alpha,\beta )$ and optimal selection for duration decay $\gamma ^{i}$. In this article, we only give the recommended values. Combining with another pattern [43] may be a promising solution to this problem. Secondly, high-speed FP [44,45] has been a hot topic of research in the field of FP. One of the directions in future work is to achieve FP reconstruction with less required data with the tools of machine learning and deep learning [46]. We believe that there will be a deeper integration and application in the fields of machine learning and FP in the future.
Funding
National Natural Science Foundation of China (61527802); China Postdoctoral Science Foundation (BX20200051).
Acknowledgment
The authors acknowledge Guoan Zheng’s group for providing a open source code and dataset and the authors of mPIE for providing three perspectives that gave us inspirations.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are available in [11].
References
1. Y. Park, C. Depeursinge, and G. Popescu, “Quantitative phase imaging in biomedicine,” Nat. Photonics 12(10), 578–589 (2018). [CrossRef]
2. J. M. Rodenburg and H. M. L. Faulkner, “A phase retrieval algorithm for shifting illumination,” Appl. Phys. Lett. 85(20), 4795–4797 (2004). [CrossRef]
3. H. M. L. Faulkner and J. M. Rodenburg, “Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm,” Phys. Rev. Lett. 93(2), 023903 (2004). [CrossRef]
4. G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nat. Photonics 7(9), 739–745 (2013). [CrossRef]
5. J. Marrison, L. Räty, P. Marriott, and P. O’Toole, “Ptychography – a label free, high-contrast imaging technique for live cells using quantitative phase information,” Sci. Rep. 3(1), 2369 (2013). [CrossRef]
6. S. McDermott, P. Li, G. Williams, and A. Maiden, “Characterizing a spatial light modulator using ptychography,” Opt. Lett. 42(3), 371 (2017). [CrossRef]
7. S. Sala, D. J. Batey, A. Prakash, S. Ahmed, C. Rau, and P. Thibault, “Ptychographic X-ray computed tomography at a high-brilliance X-ray source,” Opt. Express 27(2), 533 (2019). [CrossRef]
8. B. J. Daurer, S. Sala, M. F. Hantke, H. K. N. Reddy, J. Bielecki, Z. Shen, C. Nettelblad, M. Svenda, T. Ekeberg, G. A. Carini, P. Hart, T. Osipov, A. Aquila, N. D. Loh, F. R. N. C. Maia, and P. Thibault, “Ptychographic wavefront characterization for single-particle imaging at x-ray lasers,” Optica 8(4), 551 (2021). [CrossRef]
9. M. Rosenfeld, G. Weinberg, D. Doktofsky, Y. Li, L. Tian, and O. Katz, “Acousto-optic ptychography,” Optica 8(6), 936 (2021). [CrossRef]
10. A. Pan, C. Zuo, and B. Yao, “High-resolution and large field-of-view Fourier ptychographic microscopy and its applications in biomedicine,” Rep. Prog. Phys. 83(9), 096101 (2020). [CrossRef]
11. G. Zheng, C. Shen, S. Jiang, P. Song, and C. Yang, “Concept, implementations and applications of Fourier ptychography,” Nat. Rev. Phys. 3(3), 207–223 (2021). [CrossRef]
12. L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for Fourier Ptychography with an LED array microscope,” Biomed. Opt. Express 5(7), 2376 (2014). [CrossRef]
13. R. Horstmeyer, J. Chung, X. Ou, G. Zheng, and C. Yang, “Diffraction tomography with Fourier ptychography,” Optica 3(8), 827 (2016). [CrossRef]
14. X. Ou, J. Chung, R. Horstmeyer, and C. Yang, “Aperture scanning Fourier ptychographic microscopy,” Biomed. Opt. Express 7(8), 3140 (2016). [CrossRef]
15. S. McDermott and A. Maiden, “Near-field ptychographic microscope for quantitative phase imaging,” Opt. Express 26(19), 25471 (2018). [CrossRef]
16. H. Öztürk, H. Yan, Y. He, M. Ge, Z. Dong, M. Lin, E. Nazaretski, I. K. Robinson, Y. S. Chu, and X. Huang, “Multi-slice ptychography with large numerical aperture multilayer Laue lenses,” Optica 5(5), 601 (2018). [CrossRef]
17. Y. Zhu, M. Sun, X. Chen, H. Li, Q. Mu, D. Li, and L. Xuan, “Single full-FOV reconstruction Fourier ptychographic microscopy,” Biomed. Opt. Express 11(12), 7175 (2020). [CrossRef]
18. A. Maiden, M. Humphry, M. Sarahan, B. Kraus, and J. Rodenburg, “An annealing algorithm to correct positioning errors in ptychography,” Ultramicroscopy 120, 64–72 (2012). [CrossRef]
19. J. Liu, Y. Li, W. Wang, H. Zhang, Y. Wang, J. Tan, and C. Liu, “Stable and robust frequency domain position compensation strategy for Fourier ptychographic microscopy,” Opt. Express 25(23), 28053 (2017). [CrossRef]
20. R. Eckert, Z. F. Phillips, and L. Waller, “Efficient illumination angle self-calibration in Fourier ptychography,” Appl. Opt. 57(19), 5434 (2018). [CrossRef]
21. C. Zheng, S. Zhang, G. Zhou, Y. Hu, and Q. Hao, “Robust Fourier ptychographic microscopy via a physics-based defocusing strategy for calibrating angle-varied LED illumination,” Biomed. Opt. Express 13(3), 1581 (2022). [CrossRef]
22. A. M. Maiden and J. M. Rodenburg, “An improved ptychographical phase retrieval algorithm for diffractive imaging,” Ultramicroscopy 109(10), 1256–1262 (2009). [CrossRef]
23. X. Ou, G. Zheng, and C. Yang, “Embedded pupil function recovery for Fourier ptychographic microscopy,” Opt. Express 22(5), 4960 (2014). [CrossRef]
24. C. Zuo, J. Sun, and Q. Chen, “Adaptive step-size strategy for noise-robust Fourier ptychographic microscopy,” Opt. Express 24(18), 20724 (2016). [CrossRef]
25. A. Maiden, D. Johnson, and P. Li, “Further improvements to the ptychographical iterative engine,” Optica 4(7), 736 (2017). [CrossRef]
26. J. Zhang, T. Xu, Z. Shen, Y. Qiao, and Y. Zhang, “Fourier ptychographic microscopy reconstruction with multiscale deep residual network,” Opt. Express 27(6), 8612 (2019). [CrossRef]
27. L. Hou, H. Wang, J. Wang, and M. Xu, “Background-noise reduction for fourier ptychographic microscopy based on an improved thresholding method,” Curr. Opt. Photonics 2, 165–171 (2018).
28. M. Zhang, L. Zhang, D. Yang, H. Liu, and Y. Liang, “Symmetrical illumination based extending depth of field in Fourier ptychographic microscopy,” Opt. Express 27(3), 3583 (2019). [CrossRef]
29. R. Claveau, P. Manescu, M. Elmi, V. Pawar, M. Shaw, and D. Fernandez-Reyes, “Digital refocusing and extended depth of field reconstruction in Fourier ptychographic microscopy,” Biomed. Opt. Express 11(1), 215 (2020). [CrossRef]
30. S. Dong, K. Guo, P. Nanda, R. Shiradkar, and G. Zheng, “FPscope: a field-portable high-resolution microscope using a cellphone lens,” Biomed. Opt. Express 5(10), 3305 (2014). [CrossRef]
31. T. Aidukas, R. Eckert, A. R. Harvey, L. Waller, and P. C. Konda, “Low-cost, sub-micron resolution, wide-field computational microscopy using opensource hardware,” Sci. Rep. 9(1), 7457 (2019). [CrossRef]
32. V. Elser, “Phase retrieval by iterated projections,” J. Opt. Soc. Am. A 20(1), 40 (2003). [CrossRef]
33. Y. Zhou, J. Wu, J. Suo, X. Han, G. Zheng, and Q. Dai, “Single-shot lensless imaging via simultaneous multi-angle LED illumination,” Opt. Express 26(17), 21418 (2018). [CrossRef]
34. S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv:1609.04747 [cs] (2017).
35. L.-H. Yeh, J. Dong, J. Zhong, L. Tian, M. Chen, G. Tang, M. Soltanolkotabi, and L. Waller, “Experimental robustness of Fourier ptychography phase retrieval algorithms,” Opt. Express 23(26), 33214 (2015). [CrossRef]
36. I. Sutskever, J. Martens, and G. Dahl, “On the importance of initialization and momentum in deep learning,” in International conference on machine learning, (2013) pp. 1139–1147.
37. D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,” arXiv:1412.6980 [cs] (2017).
38. H. Zhang, S. Jiang, J. Liao, J. Deng, J. Liu, Y. Zhang, and G. Zheng, “Near-field Fourier ptychography: super-resolution phase retrieval via speckle illumination,” Opt. Express 27(5), 7498 (2019). [CrossRef]
39. Y. Huang, S. Jiang, R. Wang, P. Song, J. Zhang, G. Zheng, X. Ji, and Y. Zhang, “Ptychography-based high-throughput lensless on-chip microscopy via incremental proximal algorithms,” Opt. Express 29(23), 37892 (2021). [CrossRef]
40. P. Li, N. W. Phillips, S. Leake, M. Allain, F. Hofmann, and V. Chamard, “Revealing nano-scale lattice distortions in implanted material with 3D Bragg ptychography,” Nat. Commun. 12(1), 7059 (2021). [CrossRef]
41. X. Wei and P. Urbach, “Ptychography with multiple wavelength illumination,” Opt. Express 27(25), 36767 (2019). [CrossRef]
42. L. Liu, J. Li, X. Wang, J. Zhang, J. Yu, and L. Cao, “Momentum Acceleration Fourier Ptychographic Microscopy,” in 2021 International Conference on Electronic Information Engineering and Computer Science (EIECS), (IEEE, Changchun, China, 2021), 780–783.
43. S. Jiang, K. Guo, J. Liao, and G. Zheng, “Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow,” Biomed. Opt. Express 9(7), 3306 (2018). [CrossRef]
44. J. Sun, Q. Chen, J. Zhang, Y. Fan, and C. Zuo, “Single-shot quantitative phase microscopy based on color-multiplexed Fourier ptychography,” Opt. Lett. 43(14), 3365 (2018). [CrossRef]
45. J. Sun, C. Zuo, J. Zhang, Y. Fan, and Q. Chen, “High-speed Fourier ptychographic microscopy based on programmable annular illuminations,” Sci. Rep. 8(1), 7669 (2018). [CrossRef]
46. Y. Xue, S. Cheng, Y. Li, and L. Tian, “Reliable deep-learning-based phase imaging with uncertainty quantification,” Optica 6(5), 618 (2019). [CrossRef]
