1. Introduction

Due to the advantages of non-contact, low cost and high accuracy, 3D shape measurement methods based on fringe projection profilometry (FPP) [1–4] have been widely used in computer vision, biomedical engineering, and other fields [5–7]. In some applications, such as industrial online detection, it is important to perform 3D shape measurement for dynamic scenes in low latency, which brings challenges to traditional FPP techniques for their accuracy and real-time capability.

The conventional 3D reconstruction process of FPP consists of four key steps: fringe pattern modulation, phase demodulation, phase unwrapping, and 3D calibration. To tackle the demands of dynamic scenes, these steps need to be adapted and optimized for improved accuracy and real-time performance. Fourier transform profilometry (FTP) [8,9] and phase-shifting profilometry (PSP) [10,11] are the most common approaches to demodulate wrapped phase from deformed fringe patterns. PSP-based methods utilize the least square algorithm (LSA) and usually require at least three frames to demodulate the phase value. Such kind of methods are robust to surface variation and non-uniform reflection but sensitive to motion-induced error in the measurement of dynamic scenes [12]. FTP-based methods adopt a properly designed bandpass filter to separate the desired components from the frequency domain of the fringe pattern. The influence of moving objects’ motion can be avoided radically since only one frame fringe is required. However, due to the influence of filter operations, it is difficult for FTP-based methods to perform high-accuracy measurement on complex surfaces [13].

In FPP, to eliminate the ambiguity of the demodulated wrapped phase, phase unwrapping should be carried out, which can mainly be divided into spatial phase unwrapping (SPU) [13–16], and temporal phase unwrapping (TPU) [17–19], etc. SPU methods usually determine the phase unwrapping path by analyzing the adjacent region of one pixel, or optimize an absolute phase by minimizing the difference of phase gradients. They are usually implemented based on a single wrapped phase, and are efficient for measurement. However, constrained by the Itoh condition [20], they are not suitable to deal with locally isolated discontinuous regions. On the other hand, TPU methods can offer robust pixel-wise results with the aid of additional wrapped phases. With the increase of required frames for wrapped phase calculation, TPU methods have to make compromises with time resolution [18].

In order to overcome the difficulties of traditional methods in measuring dynamic scenes, researchers have developed various kinds of strategies [3,21–26] to reduce the number of required projected fringe patterns. One intuitive approach is to utilize the redundant information to replace the original phase-shifting fringe patterns with fewer images, such as Zuo’s method (3 + 2) [22] shares the same background intensity within two sets of phase-shifting fringe patterns, his other method (2 + 2) [23] uses ramp patterns with a linear variation of intensity to compute phase. However, the motion-induced errors occurred in phase demodulation still exist, since the height of the same point on the tested object may change between frames. Another approach is to modulate multiple fringe patterns by different carrier frequencies along the orthogonal direction, and composite them into one pattern image [25,26]. The corresponding wrapped phases could be separated from the spectrum of the composited fringe pattern by FTP and unwrapped by TPU. Although frequency multiplexing can significantly improve the efficiency of FPP, the spectrum aliasing problem occurred in FTP will make it unreliable to apply this approach in high-accuracy measurement.

Compared with traditional methods, many recently developed deep learning-based FPP methods have achieved excellent performance in various aspects, such as fringe enhancement [27], phase unwrapping [28–31], high dynamic range measurement [32–34] and fringe analysis [35–38]. For dynamic scene measurement with fewer patterns, Sam Van der Jeught et al. [39] constructed an end-to-end convolutional neural network (CNN) that transforms one fringe pattern to depth directly. This one-stage way can predict depth without any additional intermediate processing steps, but is difficult to learn the accurate mapping function between the fringe intensity and height distribution, especially in the case of complex surface. Nguyen et al. [40] designed a fringe-to-fringe CNN that separated out multiple phase-shifted grayscale fringe patterns from a composite color coding pattern, which are further used for PSP and TPU. Li et al. utilized a dual-frequency composite approach [41] and frequency multiplexing approach [42], in which the numerator and denominator of LSA and a coarse phase map can be predicted from a single composite fringe pattern, and the unwrapped phase can be obtained by TPU. The processing of such multi-stage way mainly follows the traditional FPP and the deep learning technique is incorporated in the phase demodulation step for higher accuracy. However, because of the large amount parameters of neural networks and the complexity of network structures, it is time-consuming to perform one interface and hard to achieve real-time processing for these methods.

Based on the neural architecture search (NAS) technique [43,44], a modified dual frequency heterodyne fringe projection profilometry is proposed for real-time high-accuracy 3D measurement in this paper. For phase demodulation, U-shaped architecture is selected as the backbone network, and the down- and up-sampling layers are replaced by corresponding cell architectures, which are searched from the space that contains suitable primitive operations. For phase unwrapping, one pre-computed phase of a reference plane is employed to enable a heterodyne method of dual high frequency patterns. After proper training for the searched architecture, the wrapped phase can be recovered from two sinusoidal fringe patterns rapidly and robustly with fewer amount of parameters in the neural network. By reusing patterns from adjacent frames in dynamic measurement, each newly captured fringe pattern can update one corresponding 3D result. Experimental results show that our method can achieve a reconstruction rate of up to 58 frame per second (fps) and realize real-time high-accuracy measurement for moving objects.

2. Principle

2.1 Basic principle of fringe projection profilometry (FPP)

A typical monocular FPP system consists of a camera and a projector. Fringe patterns are projected onto a target object by the projector and the distorted fringe patterns that contained the wanted depth information of the object’s surface are captured by the camera. A sinusoidal fringe pattern can be represented as:

While traditional FPP techniques have demonstrated their efficacy in 3D shape measurement, they often encounter challenges in achieving both high-accuracy and real-time processing. On the other hand, although deep learning-based FPP have shown great potential in achieving high-accuracy reconstruction with few fringe patterns, the manual design of neural architectures is often heuristic and time-consuming. Therefore, we seek to advance 3D shape measurement by combining traditional FPP with NAS-assisted deep learning, offering real-time processing capabilities and enhanced accuracy in phase demodulation.

2.2 Basic principle of neural architecture search (NAS)

Since the invention of deep learning techniques, numerous successful neural architectures have been introduced in the field of CNN, such as ResNet [46], UNet [47], and so on. However, the design of the neural architecture is heuristic in most cases, and heavily relies on the prior knowledge of the designer. As a result, NAS was developed to find the optimal architecture automatically with minimal human intervention.

The NAS involves three key components [44]: the search space, search strategy, and performance estimation strategy. These components collaborate with each other, as illustrated in Fig. 1(a), to accomplish the task of searching optimal architecture. The search process is similar to the parameter optimization in usual CNN training, except that it seeks optimized architecture rather than hyperparameter. The search strategy details how to explore the search space, which typically includes predefined primitive operations and neural architecture hyperparameters, to construct a candidate architecture. The candidate architecture found by the search strategy is then quantitatively analyzed by a performance evaluation strategy to obtain a score on the target task. The information on the performance is then fed back to the search strategy to adjust inner parameters for the next search. The whole process is repeated until the given termination condition is satisfied. Once the final optimal architecture is searched, it will be retrained as a normal neural network and applied to practical inference.

 

Fig. 1. (a) General flowchart of NAS. (b) Flowchart of NAS with cell-based search space and gradient descent search strategy.

Download Full Size | PDF

To search for a proper architecture, one direct approach is to start from scratch and employ reinforcement learning or recurrent neural networks [46] to discretely search among all possible combinations of candidate operations, which means a very large search space and inefficient adjustment of neural architecture. Rather than search all necessary components of the neural architecture, we adopt the cell-based search space of differentiable architecture search (DARTS) [48]. The best cell architectures are searched first and the overall neural architecture is constructed by repeatedly stacking the cells on the backbone network. In this way, the existing excellent neural architecture can be utilized to greatly save the corresponding search cost.

As shown in Fig. 1(b), every cell in the cell-based search space can be regarded as a directed acyclic graph (DAG) consisting of a topologically ordered sequence of nodes. There are two input nodes and one output node in each cell, where the input node is the output of two previous cells and the results of all intermediate nodes are concatenated to generate the cell output. Each intermediate node has a latent representation x_i to be learned. Each edge in DAG is a mixed operation ${\bar{o}^{({i,j} )}}(x )$ of candidate primitive set that transforms x_i to compose x_j, which can be expressed as a weighted sum of all possible operations $o(x )$:

NAS explores a wide range of architectures and discovers patterns and configurations that may not have been considered by human designers, which has the potential to find more reliable and robust architectures for fringe analysis and 3D reconstruction. Besides, during the continuous optimization of the architecture, NAS eliminates unnecessary operations and identifies more efficient configurations, which enables the identification of complex and effective combinations of operations that accurately capture and represent features of fringe patterns. In summary, NAS can revolutionize fringe analysis by automating the search for optimal neural architectures. By leveraging its ability to explore vast search spaces, refine architectures, and optimize computational efficiency, NAS can offer improved reliability, accuracy, and speed with limited model size, ultimately enhancing the effectiveness of fringe analysis and reconstruction processes. For this reason, NAS is the first choice of our neural architecture in this research work.

2.3 NAS-optimized dual frequency real-time 3D reconstruction method

As discussed before, traditional methods face challenges in achieving high-accuracy 3D reconstruction with a limited number of fringe patterns. Meanwhile, the inference times of most deep learning-based methods are struggling to meet the requirements of real-time 3D measurement in dynamic scenes, especially when dealing with high-resolution images. To address this problem, an efficient phase demodulation using NAS-optimized lightweight neural network was proposed for accurate fringe analysis, which couples the characteristics of high-resolution, high information utilization and low computational density, and a modified dual frequency heterodyne phase unwrapping strategy is used to increase the fringe frequencies, which offers more information for the feature extraction in the neural network. The diagram of the proposed NAS-based real-time 3D reconstruction method is shown in Fig. 2.

 

Fig. 2. Proposed NAS-optimized dual frequency real-time 3D reconstruction method. Steps 1 to 4 represent NAS-assisted phase demodulation, wrapped phase calculation, reference plane-assisted dual-frequency heterodyne phase unwrapping and 3D shapes reconstruction, respectively.

Download Full Size | PDF

Step 1: To feed the low-frequency and high-frequency distorted fringe patterns to a well-trained NAS-optimized lightweight network respectively, and obtain the numerator M and denominator D of the corresponding frequency of the arctangent function in Eq. (4). As introduced in section 2.1, the traditional dual-frequency heterodyne method has to construct a beat unit frequency, which limits its application in high-precision real-time 3D reconstruction. More specifically, it exhibits poor noise resistance in high frequency which makes it prone to phase unwrapping failures. Besides, existing neural networks for fringe analysis often demonstrate lower accuracy when dealing with single-frame low-frequency fringe, which is primarily due to the spectral overlapping that makes it difficult to extract reliable information. Therefore, two frequencies with a heterodyne larger than 1 are selected for higher frequency capability and phase unwrapping robustness, where the unwrapping process of the equivalent phase is handled with the phase of a reference plane. In this paper, we choose f_l = 56 and f_h = 64 to realize phase unwrapping.

Step 2: To calculate the wrapped phase ${\phi _h}$ and ${\phi _l}$ according to Eq. (3).

Step 3: To calculate the unwrapped phase ${\mathrm{\Phi }_h}$ and ${\mathrm{\Phi }_l}$ assisted by the precomputed unambiguous phase of a reference plane ${\mathrm{\Phi }_r}$. The beat frequency chosen for the proposed method is set to larger than 1, which means we can use one pair of frequencies like 56 and 64 to modulate the object’s information with more phase variation, resulting in a higher accuracy. Before the measurement, a reference plane is placed at the maximum measurement depth, and its unwrapped phase is recovered using the conventional method. During the measurement, the phase of the reference plane is used to assist the unwrapping process of the equivalent phase obtained by heterodyning, which further eliminates the ambiguity of the phase of low and high frequencies. The introduction of the reference plane is aimed at eliminating the requirement of unit frequency in the traditional three-frequency phase unwrapping method, but without increasing the error amplification rate during phase unwrapping, enabling the achievement of the equivalent single-shot processing. As shown in Fig. 3, since the equivalent phase obtained by subtracting the phase of selected frequencies is still wrapped with the frequency of ${f_{eq}} = {f_h} - {f_l} = 8$, an unwrapped phase of the reference plane with the same frequency is employed to assist in eliminating this ambiguity. The plane is placed on the maximum depth of the measurement range, and its unwrapped phase ${\mathrm{\Phi }_r}$ of frequency ${f_{eq}}$ is stored in advance, so the unwrapped equivalent phase can be solved as:

 

Fig. 3. Proposed dual-frequency heterodyne phase unwrapping method assisted by the reference plane. (a) Equivalent phase obtained by subtracting the phase of low and high frequencies. (b) Unwrap the equivalent phase assisted by the reference plane. (c) Unwrap the wrapped phase of high frequency.

Download Full Size | PDF

Furthermore, we can perform phase unwrapping for both high-frequency and low-frequency wrapped phases using the following equations:

Step 4: To reconstruct 3D shapes from the unwrapped phases using the traditional calibration method.

The predictions of M and D of low and high frequency fringes share the same well-trained neural network, which is searched by NAS to implement phase demodulation with low latency and high accuracy. By cyclically projecting dual-frequency fringes, it is possible to obtain a high-precision reconstruction result with each newly captured fringe pattern. Since the small difference in frequency between 56 and 64, the accuracy of the reconstruction results obtained at corresponding frequencies will also be similar. From a temporal perspective, this approach achieves single-shot 3D reconstruction.

The U-shaped architecture illustrated in Step 1 of Fig. 2 is chosen as the backbone network to construct the final network with various cells, as it is the most commonly used architecture in the field of combining FPP and deep learning due to its proven effectiveness in diverse areas, such as fringe analysis and fringe enhancement. Inspirited by NAS-UNet [50] in medical image segmentation, the original UNet's convolutional layers are substituted with cell structures, integrating the down- and up-sampling operations into these cells, as shown in Fig. 4(a). In the contracting path, the down-sampling cells are traversed three times to capture features at different levels. Conversely, the expansive path employs symmetric up-sampling cells to restore spatial information from the contracting path's multi-level features. Notably, the skip connection is considered a learnable operation within the up-sampling cell. Lastly, a 1 × 1 convolutional operation precedes the final up-sampling cell, reducing the filter dimensionality to 2 for phase demodulation.

 

Fig. 4. (a) U-shaped backbone of NAS network. The green arrow “identity” represents passing the feature maps directly to the next cell, as all the down- and up-sampling operations are complete inside the cells. The orange arrow “connection” represents the learnable operations in up-sampling cells corresponding skip connection in UNet. (b) Cell architecture of down- and up-sampling cells, where the red arrows are searched from different primitive operations in these two types of cells.

Download Full Size | PDF

The cell architecture is shown in Fig. 4(b). As introduced in section 2.2, it includes two input nodes and one output node in the cell. The input node receives the output of two previous cells with 1 × 1 Conv to keep dimensions consistent. In the down-sampling cell, Cell_{k- 1} and Cell_{k- 2} represent the two previous cells. In the up-sampling cell, Cell_{k- 1} indicates the previous cell and Cell_{k- 2} means the corresponding cell from the horizontal connection. To construct the input of the first down-sampling cell, the input fringe pattern is passed to a regular 1 × 1 convolution and a 3 × 3 convolution with a stride of 2 and padding of 1. The operations adjacent to the input nodes (marked by the red line) can reduce the spatial resolution and double the dimension of the feature map, or increase the resolution and halve the dimension. The former is called Down operation, and the latter is called Up operation. The normal operations (marked by the green line) are used to extract features while keeping the spatial resolution unchanged. The concatenate operations (marked by the blue line) collect all the feature maps from intermediate nodes and pass them to the next cell.

In order to realize real-time and high-accuracy processing of the searched neural network, our approach focuses on selecting candidate primitive operations that prioritize two critical factors: large respective fields and fewer parameters. By emphasizing a large receptive field, the network can effectively capture and analyze a broader context, enabling robust feature extraction and a more comprehensive understanding of complex visual patterns. Simultaneously, reducing the number of parameters helps optimize computational efficiency, allowing for faster inference and real-time performance, which usually means a better capability to process high-resolution images and faster interface speed. The selected operations are listed in Table 1, in which the up and down operations are searchable in up- and down-sampling cells respectively. The ‘up’ and ‘down’ in operation type mean that it can be derived from the corresponding ‘normal’ operation by setting the stride value of the inner first convolution or transposed convolution as 2.

Table 1. Candidate primitive set of normal, up and down operations. The depth Conv indicates depth-wise separable convolution. The SE, ECA and SGE indicate squeeze-and-excitation block, efficient channel attention block and spatial group-wise enhance block respectively

View Table | View all tables in this article

In addition to the original convolution, the depth-wise separable convolution [51] is selected due to its computational efficiency. The purpose of introducing dilated convolution [52] is to increase the respective field without increasing the number of parameters since high-resolution input is usually required for high-accuracy FPP.

Due to the fact that not all regions in the captured deformed fringes contain valid object information, the attention mechanism [53] is used in candidate operations to selectively focus on informative features and improve the overall performance of the final model, as shown in Fig. 5. The squeeze-and-excitation (SE) block [54] learns a set of weights using squeeze and excitation operations and emphasizes the most useful features by assigning learned weights to the channel dimension. Besides, the efficient channel attention (ECA), which utilizes a 1 × 1 convolution to implement local channel attention with fewer parameters, is added to the candidate operation sets to explore possibilities of achieving a lower latency. The spatial group-wise enhance (SGE) block is selected for a similar reason. The main idea of the SGE is to group feature maps and consider each group as representing a semantic feature. By leveraging the similarity between local and global features, an attention mask is generated to guide the enhanced spatial distribution of semantic features. The attention factor is determined based on the similarity between global and local features within each group. As a result, SGE is highly lightweight.

 

Fig. 5. Schematic diagram of the attention blocks. (a) squeeze-and-excitation (SE) block. (b) Efficient channel attention (ECA) block. (c) Spatial group-wise enhance (SGE) block.

Download Full Size | PDF

In addition, some other commonly used operations, such as identity, max pooling, and average pooling, have also been included in the search space. All the convolution operations have a kernel size of 3 × 3. The kernel sizes of max pooling and average pooling are 2 × 2. A combination of batch normalization [55] and ReLU [56] are placed after convolution-type operations.

We estimate the performance of the searched cells using the following simple criterion:

By exploring different architectural choices, NAS can identify more efficient operations and structures for processing fringe patterns. This results in the model requiring fewer parameters and calculations, leading to faster inference times of phase demodulation and lower resource requirements. Meanwhile, combined with the reference plane-assisted dual-frequency heterodyne phase unwrapping strategy, real-time and high-accuracy 3D reconstruction can be performed in complex measurement scenes.

3. Experiment

3.1 Data acquisition

The experimental setup of the FPP system contained a camera (Baumer, VCXU-31 M) with a resolution of 1280 × 800 pixels and a projector (TI, DLP4500) with a resolution of 912 × 1140 pixels. The objective lens of the camera had a focal length of 12 mm. The exposure time of the camera and the projector were both set to 10 ms. The angle between two optical axes of the camera and the projector was about 16.32°, which is obtained by the triangular stereo calibration process. The field of view of this system was about 300 × 150 mm², and the work distance was about 600 mm. The schematic diagram of the experimental FPP system is shown in Fig. 6(a).

 

Fig. 6. (a) Schematic diagram of experimental FPP system. (b) Collection of the training data and validation data. The 1st column shows the different measured objects including toys, masks, sculptures, etc. The 2nd, and 3rd columns show the ground truth numerator and denominator of the arctangent function of low frequency 56, which are obtained by 12-step phase-shifting fringe patterns. The 4th, and 5th show the corresponding data of high frequency 64.

Download Full Size | PDF

Twelve-step phase-shifting fringe patterns of frequencies 56 and 64 were projected to generate the ground truth of the numerator and denominator in the phase demodulation task and build the dataset for neural network searching, training, and validation. As shown in Fig. 6(b), we collected 100 sets of data in different scenes, including objects with diffuse surfaces such as dolls, masks, sculptures, etc, and divided them into training data and validation data with a split ratio of 9:1. The training set was used for architecture search and fixed neural network optimization. The validation set that has never been seen by the model was used to evaluate the performance of the neural network. To avoid unreliable learning in low-modulation regions, the threshold for average intensity was set to 10. All the fringe patterns were normalized by dividing by 255 (the max intensity of an 8-bit grayscale image) before starting searching and training. A cutoff transform, that randomly selects a square region with a random width of 64-256 in the image and sets the pixels within the region to zero, was applied during the search and training as a data augmentation technique to improve the generalization and robustness of the model, where the random numbers are generated by sampling from uniform distribution.

Similar to the absolute phase unwrapping method using geometric constraints [57], our approach that unwrapping phase through a precomputed unambiguous reference plane has to make a compromise on measurement depth, since the phase variation of the same point on the target object between maximum and minimum depth should be limited within 2π to avoid the incorrect fringe order determination. As the projected fringe pattern of our system occupies the entire FOV, according to trigonometry, the physical size of the period of heterodyne fringe can be calculated by dividing the width of FOV by equivalent frequency $300/({64 - 56} )= 37.5\textrm{mm}$, and the approximate maximum allowed depth range is $37.5/\textrm{tan}16.32^\circ{\approx} 128\textrm{mm}$, which is reasonable for many applications.

3.2 Network searching and training

The network was implemented by Pytorch 1.12.1 [58]. All search and train processes were completed in the hardware environment with Intel Xeon Gold 6226R 2.90 GHz CPU and four 16 GB NVIDIA Tesla P100 GPUs. The latency time was tested on a laptop with Intel i9-13800HX 2.20 GHz CPU and 8GB NVIDIA GeForce RTX 4070 Laptop GPU through TensorRT 8.6.0. The channel dimensions of all operations are 16. We used AdamW [59] optimizer with an initial learning rate of 10⁻³ to minimize the loss function of Eq. (14) for both neural network searching and training. The batch size was set to 4. We searched the architecture of down- and up-sampling cells in 250 epochs and train it in 500 epochs, where the corresponding process will be terminated when there is no improvement on the training set after 25 and 50 epochs respectively. A cosine annealing scheduler with a learning rate from 10⁻³ to 10⁻⁵ was applied to reduce the learning rate according to the epoch.

The total time for network search and training is approximately 9.3 hours. The loss curves are shown in Fig. 7(a). It can be found that the loss value during the search is higher than that during training. This is because that network structure search involves exploring a wide range of possible architectures, which can include ineffective ones that cannot generalize well to the training data. Although the loss function values during the search may be higher, the selected architecture is expected to perform better after a full training process. The searched cell architectures are shown in Fig. 7(b) and (c). It can be seen that dilated convolution and the ECA block are the most commonly used operations in the searched results, which indicates that a lightweight neural network for phase demodulation requires a large receptive field to process high-resolution fringe patterns, and a powerful attention mechanism to effectively locate feature maps of interest.

 

Fig. 7. (a) Loss curves of neural network searching, training and validation. (b)-(c) Searched down-sampling and up-sampling cell architectures of low frequency and high frequency.

Download Full Size | PDF

For comparison purposes, Feng's method [35] and a UNet with a depth of 5 [36] have also been trained and validated using the same dataset. Figure 8 shows the latency time and parameter amount of Feng’s method, UNet and the proposed network. The details of the latency of the three methods are given in Fig. 8(d). The representation of floating-point numbers on GPUs is generally similar to that on CPUs. Both CPUs and GPUs use IEEE 754 standard formats for representing floating-point numbers, which includes FP16 (half-precision floating-point format) and FP32 (single-precision floating-point format). The FP32 is commonly used for most computations during training and inference due to its higher precision. For certain scenarios where memory usage or computational efficiency is a concern, such as on resource-constrained devices or in large-scale distributed training, FP16 may be employed to speed up computations and reduce the memory footprint, since it uses only half the number of bits as FP32 to represent numbers. However, the reduced precision in FP16 can lead to some loss of accuracy and potential numerical instability.

 

Fig. 8. Performance comparison of Feng’s method, UNet and the proposed method. (a) Inference time of three methods in precision FP32. The MAE/RMS/STD of phase error (rad) are given below the methods’ names. (b) Corresponding results of three methods in precision FP16. (c) Model size of three methods. (d) Details of parameters, GPU usage (MiB) and latency (ms) of three methods.

Download Full Size | PDF

As seen in Fig. 8, the mean inference speed of our model can realize 35 fps in FP32 and 58 fps in FP16, while the UNet and Feng’s method are hard to meet the requirement of real-time 3D reconstruction if the exposure time of the camera is considered. Moreover, the proposed method exhibits a remarkably smaller model size, approximately 1/234 compared to UNet, and approximately 1/5 compared to Feng's method. Besides, our method occupies approximately half of the GPU memory compared to the other two methods. This compelling reduction in size renders our model exceptionally well-suited for deployment on edge devices.

The mean absolute error (MAE), root mean square error (RMS) and standard error (STD) of the phase error on the validation set are also shown in Fig. 8. With the reduction in model size and acceleration in inference speed, compared to Feng's method and UNet, the proposed method shows an increase of approximately 0.02 rad in MAE, 0.03 rad in RMS, and 0.03 rad in STD. It can be seen that there is minimal change in the accuracy of the model when utilizing FP16 data representation. The slight reduction in MAE and STD when using FP16 compared to FP32 can be attributed to the regularization effect introduced by the noise inherent in FP16 representation. This noise can sometimes aid in improving the generalization performance of the model, resulting in better accuracy when applied to unseen data. The experimental results in the following section were all tested in FP16.

3.3 Quantitative evaluation of standard components

To verify the effectiveness of the proposed method, we first conducted a quantitative evaluation experiment on two standard spheres. As shown in Fig. 9(a), the diameters of the two standard spheres are 50.7991 mm and 50.7970 mm respectively, and the center distance is 100.2537 mm. The ground truth values for these parameters are obtained by high-precision contact-based measurement using a three-coordinate measuring machine. Figure 9(b)-(c) illustrate the reconstructed results, which are obtained by the proposed NAS phase demodulation method and reference plane-assisted dual-frequency heterodyne phase unwrapping method. The corresponding error distributions and their histograms are shown in Fig. 9(d)-(e) and Fig. 9(f)-(i) respectively. The MAE errors of the two spheres are 0.0816 mm, 0.0722 mm in low frequency fringe and 0.0724 mm, 0.0869 mm in high frequency fringe, while their RMS errors are 0.0821 mm, 0.0731 mm and 0.0809 mm, 0.0919 mm. It's worth noting that as the frequency transitions from low to high, the error of the right sphere increases. This could be attributed to the right sphere being positioned farther from the camera, and experiencing a more severe reduction in modulation compared to the left sphere, leading to decreased measurement accuracy. The results demonstrate that the proposed method can achieve high-accuracy 3D reconstruction in low latency with an absolute error of less than 0.1 mm. Moreover, the accuracy of the obtained results is very similar in both low-frequency and high-frequency cases. This provides robust support for stable measurements in dynamic scenes.

 

Fig. 9. Measurement results of two standard spheres. Symbols d_A and d_B represent the diameter of the left and right spheres respectively, and c represents the center distance. (a) Standard spheres to be measured. (b)-(c) Reconstructed results of the proposed method using low frequency and high frequency respectively. (d)-(e) Corresponding phase error distribution of (b)-(c). (f)-(g) Histogram of the error of the left and right sphere in low frequency. (h)-(i) Corresponding histogram of high frequency.

Download Full Size | PDF

3.4 Quantitative evaluation of complex scenes

The experiments of four scenarios, which include sculptures, masks and toys, were further conducted to show the performance of our method on complex surfaces. These scenes have never been seen in the search and training process. The wrapped phases calculated from the 12-step phase shift method and the corresponding 3D shapes reconstructed were used as the ground truth. The phase error distributions and their histograms of the proposed method and traditional windowed Fourier transform (WFT) method [60] are shown in Fig. 10. And the MAE and RMS errors of both methods in low frequency f_l = 56 and high frequency f_h = 64 are given in Table 2.

 

Fig. 10. Phase error comparison results of the proposed method and traditional method in four scenarios. The phase error distributions and their histogram are shown in odd and even rows, respectively. The wrapped phase error of low frequency and high frequency are calculated in the 2nd and 4th column respectively, and the corresponding results of the traditional WFT method are shown in the 3rd and 5th column.

Download Full Size | PDF

Table 2. MAE and RMS of the phase error of the proposed method and the traditional method in four different scenarios

View Table | View all tables in this article

The proposed method can achieve high-accuracy phase demodulation with the maximum MAE, and RMS values of 0.0607 rad and 0.0802 rad, while the traditional WFT method has a low precision with RMS errors larger than 0.2 rad in all four scenes, especially when dealing with the complex part with depth variation. Due to the scarcity of pixels with low modulation depth in the training samples, it is also challenging for the neural network to learn the mapping relationship between the fringe patterns in these regions and the corresponding ground truth. As a result, the regions with low modulation depth in the output may struggle to obtain accurate phase values. Besides, there is a small difference with RMS errors less than 0.04 rad between the phase results obtained using high- and low-frequency fringe patterns. The phase error obtained by the proposed method from the high-frequency fringe pattern is generally smaller than that of the low-frequency fringe pattern, and the difference between them is approximately in the range of 0.001 to 0.009 rad, which can be accepted in actual measurement, while the WFT method shows similar trends. Since the two frequencies used in our method are relatively close, their accuracy difference is slight and can be acceptable for actual measurement.

The 3D reconstruction results from the predicted phases of the NAS network, as illustrated in Fig. 11 are obtained by the proposed dual frequency heterodyne phase unwrapping method and triangular stereo calibration model. When the phase errors become large, such as the phase of the object’s edges processed by the WFT method, the amplification of errors during the phase unwrapping process will cause phase variation exceeding 2π, resulting in incorrect fringe order determination. Therefore, multi-frequency phase unwrapping method are employed to unwrap the wrapped phase of WFT for better visual comparison.

 

Fig. 11. 3D reconstruction results of the proposed method and traditional WFT method.

Download Full Size | PDF

It can be observed that the proposed dual-frequency method can successfully realize high-frequency phase unwrapping with the assistance of the phase of the reference plane. In comparison with the traditional method, our method demonstrates a significant improvement in phase extraction and provides robust phase unwrapping results in complex scenes.

3.5 Real-time 3D measurement of a dynamic scene

To verify the performance of the proposed method for real-time applications, a measurement system was developed with the same hardware environment with test cases. In the measurement of dynamic scenes, two fringe patterns of frequencies 56 and 64 were projected cyclically and processed using the corresponding NAS network to obtain phase information. The phase unwrapping can then be performed using a dual-frequency heterodyne method assisted by the stored phase of the reference plane. This approach allows for updating a new 3D reconstruction result with each newly captured fringe pattern, enabling real-time processing. A sculpture with complex surface variation placed on a rotary table was measured in the dynamic scene. The point clouds were displayed through OpenGL, as shown in Fig. 12. The real-time measurement result is shown in Visualization 1, which demonstrates the potential of the proposed approach for real-time 3D measurement.

 

Fig. 12. Real-time measurement process and the result of the rotating sculpture with complex surface variation (Visualization 1).

Download Full Size | PDF

For the implementation details of the proposed method, we summarized using a flowchart as shown in Fig. 13. The calibration parameters and the phase of the reference plane were stored on the GPU in advance. The process begins with the camera capturing modulated fringes, which are initially transferred from the CPU to the GPU. Subsequently, the image undergoes normalization and is passed to the neural network for processing. Then, the well-trained NAS neural network is employed to obtain the numerator and denominator of the arctangent function. To reduce the number of pixels to be processed in later stages and speed up the reconstruction, the modulation B is calculated from the arctangent terms according to Eq. (5), and a threshold is applied to generate a mask. Any position with a mask value of 0 is excluded from further processing. After obtaining the wrapped phase, and combining it with the phase of another frequency calculated from the previous frame, the improved dual-frequency unwrapping method is utilized to obtain the unwrapped phase quickly and accurately. Finally, 3D point clouds are computed using calibration parameters.

 

Fig. 13. Flowchart of the proposed algorithm implementation for real-time 3D measurement and the time consumed by each step.

Download Full Size | PDF

All of the aforementioned steps are implemented using C++ and CUDA. We measured the average time for each step after running it 1000 times, as shown in the last row of Fig. 13. It can be observed that the time for the entire 3D reconstruction process is mainly consumed by the inference of the neural network, highlighting the necessity for network optimization. The total time for 3D reconstruction is approximately 14.2 ms (about 70.4 fps), which completely satisfies the real-time requirements. Taking into account other time-consuming operations such as image transfer between CPU and GPU and point cloud visualization, during actual operations, we set the projector's time interval to 17 ms (about 58.8 fps).

4. Conclusions and discussion

In this paper, we utilize the NAS technique to automatically search for lightweight neural networks from carefully selected candidate operation sets. This approach enables us to discover efficient network architectures for real-time high-precision phase demodulation. A pre-measured reference plane's phase is introduced to assist in the dual-frequency phase unwrapping process. By incorporating this additional information, we enhance the accuracy and reliability of the phase unwrapping step, leading to improved overall measurement quality. Our main contributions can be summarized in the following three points:

Compared with prior works that utilize deep learning to extract the phase information (e.g. the method reported in Ref. [41] achieves a reconstruction speed of 15 fps with a resolution of 640 × 480 pixels), a significant step has been taken forward in this paper by addressing the challenges of high resolution and real-time processing, culminating in the proposal of a feasible and lightweight solution. The presented method extends beyond the initial step of obtaining arctangent terms to the broader context of achieving real-time high-precision 3D reconstruction in dynamic scenes. The real-time measurement architecture proposed in the paper can be potentially applied to other FPP methods, accelerating the development of real-time measurement methods based on deep learning in various applications. Compared with the classical three-frequency heterodyne method, the proposed modified dual-frequency phase unwrapping method does have certain limitations concerning measurement depth and fringe frequency, as indicated in Section 3.1. Nevertheless, the proposed requires a reduced number of fringe patterns, making it more suitable for the measurement of dynamic scenes.

However, several aspects need to be further improved in the future investigation. Firstly, compared with neural networks with larger parameter amounts, although the proposed lightweight NAS networks can achieve faster inference speed, a compromise in accuracy has to be made to reduce the model complexity. There are also many lightweight networks available, such as SCNN [61] and BiSeNet [62], but they are usually not specifically designed for FPP measurements in dynamic scenarios, which may result in unsatisfied performance in tasks like fringe analysis. Further research is needed to ensure that lightweight models can achieve the accuracy of large models while enabling real-time inference. Meanwhile, it is also crucial to design the network architecture that achieves a well-balanced trade-off between accuracy and speed, in accordance with the actual requirements and the physical model of fringe analysis.

Secondly, it should be noted that certain factors in dataset construction, such as fringe frequency, intensity noise and object shape, can have a measurable impact on the precision of phase demodulation. When the fringe frequency is excessively high (resulting in reduced modulation) or when intensity noise is significant, phase errors become large, leading to inaccuracies in the ground truth of the training dataset. Conversely, when fringe frequency is too low, from a spectral perspective, the information of the object and the background is heavily mixed, making it challenging for neural networks to extract useful features and generate accurate outputs. When there are significant mismatches between the training dataset and the testing scenarios, the performance of neural network may not be optimal due to the inherent data dependency of supervised learning. Hence, the quantitative selection of appropriate parameters during dataset construction necessitates more in-depth discussion and research to mitigate the adverse effects mentioned above.

Thirdly, while fixed thresholding was utilized for fast mask generation, it may introduce inaccuracies and result in the presence of noisy points in the final results. The development of robust and adaptive thresholding methods is needed to enhance the reliability of the measurements in real-time performance.

Fourthly, while this paper has indeed achieved successful training and testing of the NAS network with datasets of different frequencies, it is necessary to acknowledge that supervised learning inherently carries a degree of data dependency, which can be further addressed by constructing larger simulated datasets [38,63,64] or exploring unsupervised learning approaches [37].

Lastly, there are several model compression algorithms used within a well-trained network to further decrease the parameter amount or accelerate the model, such as pruning, quantization, knowledge distillation and so on, which are much friendlier for the on-chip systems, or personal computers with limited hardware environment.