In this paper, we propose a high-speed 3D shape measurement technique based on the optimized composite fringe patterns and stereo-assisted structured light system. Stereo phase unwrapping, as a new-fashioned method for absolute phase retrieval based on the multi-view geometric constraints, can eliminate the phase ambiguities and obtain a continuous phase map without projecting any additional patterns. However, in order to ensure the stability of phase unwrapping, the period of fringe is generally around 20, which limits the accuracy of 3D measurement. To solve this problem, we develop an optimized method for designing the composite pattern, in which the speckle pattern is embedded into the conventional 4-step phase-shifting fringe patterns without compromising the fringe modulation, and thus the phase measurement accuracy. We also present a simple and effective evaluation criterion for the correlation quality of the designed speckle pattern in order to improve the matching accuracy significantly. When the embedded speckle pattern is demodulated, the periodic ambiguities in the wrapped phase can be eliminated by combining the adaptive window image correlation with geometry constraint. Finally, some mismatched regions are further corrected based on the proposed regional diffusion compensation technique (RDC). These proposed techniques constitute a complete computational framework that allows to effectively recover an accurate, unambiguous, and distortion-free 3D point cloud with only 4 projected patterns. Experimental results verify that our method can achieve high-speed, high-accuracy, robust 3D shape measurement with dense (64-period) fringe patterns at 5000 frames per second.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
12 March 2019: Corrections were made to the author affiliations and funding section.
In recent years, high-speed 3D shape measurement techniques are widely used in various fields such as biomechanics, intelligent monitoring, robot navigation, industrial quality control, and human-computer interaction. Among plenty of state-of-the-art methods, fringe projection profilometry (FPP) [1–4], which is based on the principle of structured light and triangulation, has been proven to be one of the most promising techniques due to its inherent advantages of non-contactness, high accuracy, high efficiency, and low cost. The mainstream FPP techniques can be generally divided into two groups: Fourier transform profilometry (FTP) [5, 6] and Phase-shifting profilometry (PSP) [7–10]. FTP is highly suited for dynamic 3D acquisition and can provide the phase map using a single fringe pattern. But it suffers from the spectrum overlapping problem which limits its measurement quality and precludes the recovery of the fine details of complex surfaces. In contrast, PSP is quite robust to ambient illumination and varying surface properties and can achieve pixel-wise phase measurement results with higher resolution and accuracy, but it generally requires the multiple fringe patterns to reconstruct the 3D shape of the object. When measuring dynamic scenes, the motion will lead to phase distortion artifacts, especially when the object motion during the interframe time gap is non-negligible . With the rapid advances of the high-frame-rate camera, high-speed digital light processing (DLP) projection technique, and high-performance processors, numerous PSP methods have been demonstrated in high-speed 3D shape measurement of dynamic scenes such as fast moving objects, rapidly rotating, or vibrating non-rigid bodies [12–19]. Furthermore, both PSP and FTP adopt the arctangent function ranging between and π for phase calculation, which results in phase ambiguities in the wrapped phase map with phase jumps. Therefore, it is necessary to carry out the phase unwrapping to eliminate the phase ambiguity and convert the wrapped phase maps into the absolute ones.
In general, phase unwrapping algorithms can be classified into three main categories: spatial phase unwrapping [20–23], temporal phase unwrapping (TPU) [24–26], and stereo phase unwrapping [27–30]. By assuming the phase continuity, spatial phase unwrapping generally uses the relationship between the phase information of the spatial neighboring pixels to remove the phase ambiguities in a wrapped phase map . However, the assumption will be easily violated when measuring isolated objects or surfaces with abrupt depth discontinuities. In order to solve the common limitation in spatial phase unwrapping, TPU approaches have to be used to realize pixel-wise phase unwrapping with the aid of additional wrapped phase maps with different fringe periods . Obviously, the increased number of required patterns is undesirable, which limits TPU’s application for high-speed measurements. To address this issue, several composite phase-shifting schemes (e.g., dual-frequency PSP , bi-frequency PSP , and 2+2 PSP ) have been proposed, which can solve the phase ambiguity problem without significantly increasing the number of projected patterns . On a different note, stereo phase unwrapping approaches have been proposed to overcome the limitations of spatial phase unwrapping and TPU. Compared with TPU, stereo phase unwrapping can retrieve the absolute phase using two or more wrapped phase maps obtained from two perspectives based on geometry constraint without projecting any additional patterns [27, 28]. Several methods based on this principle has been presented in the literature and achieved remarkable success [27–30]. However, in order to ensure the stability of phase unwrapping, these methods have to compromise the fringe frequency or the depth of field, leading to limited 3D measurement accuracy or measurement volume .
Geometry constraint has been widely used in stereo vision because it greatly facilitates the establishment of binocular correspondence of the two cameras in different perspectives. Weise et al.  introduced geometric constraint into PSP using only three fringe patterns for a single 3D reconstruction. In addition, some complicated post-process techniques, such as graph cut and loopy belief propagation, are used to get a global absolute phase map but it leads to massive computing overhead [35–37]. It is easy to understand that using more cameras will increase the likelihood that more constraints can be exploited to completely eliminate the phase ambiguities in the multi-view system. Following this idea, Tao et al.  set up a position-optimized quad-camera system to guarantee the reliability of phase unwrapping. The phase differences from different perspectives were applied to eliminate phase ambiguities. Besides, the weighted phase difference scheme was introduced to realize stereo phase unwrapping of high-frequency fringes. However the valid measurement areas, defined as the common regions of the cameras and projector, were significantly reduced. On the other hand, the phase-coding schemes, in which the additional signal is embedded into original fringe patterns, have been proposed for a multi-view system, which provide auxiliary information to assist the absolute phase recovery. As the special signal speckle or triangular wave were chosen, both methods need to perform time-consuming image correlation for each valid pixel which limits the measurement speed [38–40]. In addition, these methods have to reduce the amplitude of the fringe signal, resulting in the reduction of the phase measurement accuracy. To maximize the fringe signal to noise ratio (SNR), Zhang et al.  fully exploited the limited redundancies in three phase-shifting fringe patterns, but the speckle pattern is essentially susceptible to ambient illumination which affects the correlation results, so it still needs complicated post-processing algorithms to produce a more reliable phase unwrapping result. In order to overcome the problems mentioned above, Wang et al.  developed a period-coded phase-shifting (PCPS) algorithm by embedding additional slope signals into standard 4-step phase-shifting patterns. However, the coding patterns become no longer smooth due to the added signals, which are susceptible to lens defocusing and leads to the limited accuracy of 3D measurement. It is obvious that the slope-embedded patterns are not suitable to be applied to the high-speed binary defocusing projection system which has a certain amount of de-focus.
In this work, a high-speed 3D shape measurement technique based on the optimized composite fringe patterns and stereo-assisted structured light system is proposed. Firstly, we review and reproduce the design method of the composite patterns via skillfully embedding speckles without compromising the fringe modulation and the phase measurement accuracy. Then, the optimized speckle pattern is mathematically generated based on the system configuration in the experiment to ensure the period-wise uniqueness for each point instead of the conventional global uniqueness. In addition, we present a simple and effective evaluation criterion for the correlation quality of the designed speckle pattern in order to improve the matching accuracy significantly. Benefited from the optimized design for fringe patterns, the periodic ambiguities can be ruled out effectively based on geometry constraint and the proposed adaptive window image correlation. Though our method can provide superior stereo phase unwrapping performance even under noisy conditions, fringe order errors are still inevitable especially around dark regions and object edges where the fringe quality is low. Furthermore, the mismatched regions will be substantially resolved using the proposed regional diffusion compensation technique (RDC). In general, considering some wrong regions caused by mismatched or shaded effects in multi-view system, left-right consistency check, as the common technique in stereo vision, will be able to eliminate wrong points with fringe order errors. But according to some previous methods [30, 38], the mismatched points can be neither completely removed nor correctly compensated using the check operation, which leads to the final 3D measurement results with a few outliers and holes. Different from the check operation, the proposed compensation technique is a relatively low-cost method that only requires the absolute phase map obtained from main camera and can not only detect but also correct the error points. Therefore, these proposal techniques constitute a complete computational framework that allows to effectively recover an accurate, unambiguous, and distortion-free 3D point cloud with only 4 projected patterns. Experimental results verify that our method can achieve high-speed, high-accuracy, robust 3D shape measurement with dense (64-period) fringe patterns at 5000 frames per second.
2.1. Speckle embedded phase-shifting algorithm
In this subsection, we will discuss in detail how to embed the auxiliary information in the phase-shifting patterns. Firstly, the standard N-step phase-shifting fringe patterns can be described as
Where is the DC intensity, is the fringe signal amplitude, is the phase information, and is the shift offset. To maximize the signal to noise ratio (SNR) and normalize the intensity pattern , and are set as 0.5 so that the value of ranges from 0 to 1. In general, researchers embed the additional information into fringe patterns by modifying or [38–40, 42]. For example, Lohry et al.  insert binary speckle signals into three patterns by adjusting to 0.25 or 0.5. Tao et al.  choose the triangular wave as the coded signal and modify to encode fringe patterns. However, in order to ensure that the value of does not exceed the range , these methods have to reduce the amplitude of , which results in a poor quality of the phase map. To embed the speckle signal without reducing the amplitude of fringe, Zhang et al.  fully exploited the limited redundancies in three phase-shifting fringe patterns as shown in Fig. 1. For Phase-shifting profilometry, the speckle-embedded method applies the theoretical minimum of three images into the absolute phase recovery, which greatly improves the efficiency of 3D measurement. However, the embedded speckle pattern is low quality and essentially susceptible to ambient illumination which affects the correlation results, so it still needs complicated post-processing algorithms to produce a more reliable unwrapping result. To solve this issue, Wang et al.  developed a period-coded phase-shifting (PCPS) algorithm by embedding additional slope signals into standard 4-step phase-shifting patterns. The intensities of the PCPS fringe images in the projector space are:
To guarantee that the retrieved phase is unaffected by the embedded signals, the coding functions should satisfy:
And the value of also should ranges from 0 to 1. So we have:
And the embedded signal is also demodulated from Eq. (2) given by:
According to Eq. (8), we can find the value of along the x axis with the dynamic range of as shown in Fig. 2(a). Notably, the quality of the coding patterns is improved substantially because the relative amplitude of is 1. However, the designed coding signals and also become no longer smooth which is sensitive to lens defocusing as shown in Fig. 2(b). In order to ensure the robust phase unwrapping, the period of fringe is generally restricted to a very small range (), because of the slope signal which is susceptible to noise, which results in the limited accuracy of 3D measurement. Given this, Wang’s method is not suitable to be applied to the high-speed binary defocusing projection system which has a certain amount of de-focus and is much more difficult to generate desired low-frequency fringe patterns than the high-frequency ones. Consequently, the type of the auxiliary signals should be chosen carefully.
In order to overcome the problems mentioned above, we chose the speckles as our auxiliary information instead of the slope patterns. Compared with the original one-dimensional slope signal, the two-dimensional speckle signal is less sensitive to the lens defocusing and suitable for the binary defocusing systems. And the final composite fringe patterns and the embedded speckle pattern generated using the proposed speckle embedded algorithm are shown in Fig. 3.
2.2. Optimized design of the speckle pattern for phase unwrapping
Then, considering the characteristics of speckle, we mathematically developed an optimized design method of the random pattern. By skillfully embedding speckles without sacrificing the phase measurement accuracy, we can realize phase unwrapping with high-frequency fringes. The key challenge of the proposed design method is to achieve global phase unwrapping in combination with speckle patterns and multi-view constraints. By introducing depth constraint into the digital speckle image correlation for phase unwrapping, the only thing we need to do is to eliminate the periodic ambiguity within the pre-defined local range rather than the traditional global range, which means that our method just requires the speckle patterns with period-wise uniqueness instead of the conventional global uniqueness. This idea gives a guideline for the design approach and greatly simplifies the requirements of speckle patterns compared to the traditional speckle-based methods [36, 39, 43, 44]. At the same time, to avoid a universal phenomenon in the multi-view system, such as image distortion caused by perspective transformation (including stretching, skewing and rotation), we are obliged to reduce the baseline between two cameras to make the quality of stereo images better, which can improve the correlation accuracy to realize robust and efficient phase unwrapping. Besides, taking the multi-view system into consideration, some parameters of the speckle pattern should be closely related to the system parameters :
Step 1: A large number of consecutive sub-windows are generated to fill the projection pattern as shown in Fig. 4, and the size is where Nr is its horizontal resolution. For vertical fringes, it is appropriate to set the wavelength of fringes as Nr since the speckle is only used to distinguish periodic candidate points. In order to perform the region match efficiently, it is necessary that the value of Mr should be selected suitably to satisfy , and L is the window radius of image correlation on the camera plane.
Step 2: For any sub-window, we randomly fill the speckle with a size of . Theoretically, each sub-window can contains Ns speckles (). In order to enhance the uniqueness of the speckle, we determine the range () for the number of speckles in each sub-window based on the matching quality metric after an exhaustive empirical search.
The above steps have generated a speckle pattern I as shown in Fig. 5(a). Next, we test the pattern with a simple and effective evaluation standard to judge the quality of the speckle pattern. For any point in Ir, we can also find the candidate target point in It using the formula as follows:Eq. (10) and influences the correlation accuracy. For the speckle pattern with the resolution, M = 5 pixels is acceptable. Since the speckle pattern need to remove a few periodic candidate points to realize phase unwrapping, so let for vertical fringes. When is at the maximum, it indicates that the points Pr and Pt are most likely to be a set of matching points, and k is the correct fringe order of the reference point Pr as shown in Fig. 5(b). In Figs. 5(a)–5(b), the speckle pattern with the speckle of (Ms = 3 pixels) is applied to unwrap the wrapped phase with 64-period fringe patterns, as a result, there are only 317 error points (for the projected image resolution 1024 × 768, the error rate is about ) based on our speckle pattern. In general, the size of the speckle () is a key parameter for the optimized design of the speckle pattern . Likewise, for the designed speckle patterns with different speckle size, the process of phase unwrapping is simulated to find the relationship between the size of the speckle and the quality of the speckle pattern, and the corresponding results of error rate are shown in Fig. 5(c). It can be found that the size of the speckle is suitably set as pixels for the optimized design of the speckle pattern. Although the speckle pattern generated using the optimized design method can provide the great assistance for stereo phase unwrapping, there are still a few mismatched points inevitably (the default value of the error rate is for a reasonable setting) due to the influence of Gaussian noise and the randomness of the speckle. Unaffectedly, it leads to a rational statement that the errors will be further reduced using depth constraint. The test result shows that the proposed design method is effective to improve the robustness of stereo phase unwrapping with high-frequency fringe patterns.
2.3. Computational framework for phase unwrapping using geometry constraint and speckle pattern
In the first two subsections, we have presented that how to design an optimized speckle pattern for phase unwrapping and how to encode the speckles into standard 4-step phase-shifting fringe patterns without compromising the phase measurement accuracy. So the composite fringe images captured by the camera can be described as
After the wrapped phases from different views are obtained according to Eq. (12), since the captured images contain the invalid points (the background) and the valid points (the tested object), the value of for each pixel can be used to suppress the background information by the following formula:
2.3.1. Eliminating the periodic ambiguities based on geometry constraint and the adaptive window image correlation
After the removal operation for invalid points is performed, for the wrapped phases with period fringe patterns, the fringe order of each valid pixel exists N possibilities. Taking each possible order into consideration, the corresponding depth value is calculated using calibration parameters between the main camera and the projector. It is easy to find that some possible depth values are beyond the pre-defined depth range based on depth constraint and the corresponding orders can be excluded from the candidates as shown in Fig. 6. For the remaining ones, the region match based on speckle patterns will be implemented according to Eq. (10) to further remove wrong candidates. To avoid the effect of reflectivity and enhance the quality of the speckle pattern, the speckle pattern are obtained from the composite fringe images using the formula as follows:
In the digital speckle image correlation, how to select the appropriate window radius L is our main concern. It is obvious that the smaller L can speed up the calculation efficiency but just provide a coarse correlation result. On the contrary, the bigger L output reliable corresponding points at the expense of the computational cost. Given this, the smaller L firstly should be applied to obtain a coarse order map in a short time, then a few points with low reliability (such as) can be matched correctly using the larger window radius. The value of L depends on the quality of the speckle patterns from different views caused by the design method of the speckle pattern and the baseline between two cameras. Since the optimized design method is proposed and the baseline is reduced, the adaptive window radius L is set as 5 pixels and 10 pixels adequately in this paper. The operation process of confirming the period order map based on geometry constraint and the adaptive window image correlation is shown in Fig. 7.
Though geometry constraint and the adaptive window image correlation using the optimized speckle patterns can provide superior stereo phase unwrapping performance (obtaining the absolute phase with 64-period) even under noisy conditions, fringe order errors are still inevitable especially around dark regions and object edges where the fringe quality is low. Considering the common cases which are mismatched or shaded effects in stereo vision, left-right consistency check is usually used to eliminate wrong points with fringe order errors. But according to some previous methods [30, 38], error points can be neither completely removed nor correctly compensated using the check operation, which gives rise to 3D measurement results with a few outliers and holes. Therefore, the further compensation step is aimed at identifying and adjusting the residual mistakes automatically.
2.3.2. Regional diffusion compensation (RDC) of fringe order errors
To void this issue mentioned above and acquire 3D reconstruction results with high completeness, a simplified compensation technique based on spatial phase unwrapping is introduced into the traditional stereo phase unwrapping approach. In spatial phase unwrapping, reliability-guided phase unwrapping is one of the most effective methods using its neighboring pixels to obtain the relative fringe order [19, 20]. In this work, we propose the regional diffusion compensation algorithm (RDC) for detecting and correcting those fringe order errors by exploiting additional information in spatial domain. There does exist some cases that false matching points are still reserved after left-right consistency check, but these cases are very rare after RDC. To clearly indicate the compensation procedure of RDC, a flowchart is plotted as shown in Fig. 8.
After image correlation, the elementary absolute phase map can be obtained in which the values of most of valid points are correct and reliable to assist the compensation procedure, despite the fact that there are a small number of wrong points in some areas of high reflectivity, low contrast, occlusion, and edges. Before the implementation of RDC, a binary mask is built, which denotes the valid points to be computed, and the initial value of is set as . For every valid pixel in the absolute phase map , if the absolute phase difference between adjacent pixels is less than the threshold, the corresponding adjacent pixels will be grouped into a class.
In the traditional reliability-guided phase unwrapping method, the input is the wrapped phase map, and the quality map is used to find the optimal phase unwrapping path, which starts from the point with higher quality value to ensure the accuracy rate of phase unwrapping. Different from the spatial phase unwrapping method, the input of RDC is the absolute phase map after geometry constraint and the adaptive window image correlation, where most of the points have been correctly unwrapped, i.e., the points with high-quality value have been unwrapped rightly. Therefore, the quality map no longer plays a significant role for phase unwrapping. And then the number of points in the group (where the absolute phase difference of the adjacent pixels satisfys Eq. (16)) is equivalent to the overall quality of the group, so the phase correction operation is performed firstly from the group with the largest number of points in to its adjacent points. Since our previous algorithm implements a stable stereo phase unwrapping, it can be reasonably derived that groups with more than points are reliable groups, and they cannot be compensated but are used as a reference. The value of depends on the resolution of the cameras, so is acceptable for the 640 × 440 resolution. If its adjacent points belong to different groups, all points of the smaller class will be corrected according to the following formula:
Therefore, the two classes merge into one class, and the region of the group with the largest number of points will be further expanded in , which is similar to the regional growth. In addition, owing to the vertical fringes areused, the absolute phase is limited by the monotonicity constraint in the horizontal direction, so the points in the horizontal direction should take precedence over its in the vertical direction as a reference. After all the neighboring points have been adjusted properly, the processed group will be removed from , and the phase correction operation is performed again from the extant group with the largest number of points in the updated . This step is repeated until the mask is empty, which means that all the valid points have been computed and the compensation procedure is completed. Eventually, the phase errors can be effectively compensated, and the final absolute phase map is obtained. In addition, there is an optional removal operation for isolated points that the processed group with the total number of points less than will be intentionally excluded from the compensation results. In most cases, is set as 200 pixels properly. In general, the left-right consistency check is time-consuming since it needs two absolute phase maps acquired by the image correlation from two views independently. Compared with the check operation, the proposed compensation technique is a relatively low-cost method that only requires the absolute phase map from main camera and can not only detect but also correct the error points as shown in Fig. 9.
In the experimental section, a high-speed FPP system is built to verify the actual performance of the proposed method. This system includes two high-speed complementary metal oxide semiconductor (CMOS) cameras (Vision Research Phantom V611) and a high-speed DLP projection system. The high-speed projection system consists of a DLP development kit (Texas Instruments DLP Discovery 4100), a digital micro-mirror device (DMD) with the XGA resolution (1024 × 768), and a custom-designed optics module . For the optimized speckle patterns in this system, the designed parameters are set to include Nr = 16 pixels (i.e., the wavelength of fringes is 16 pixels), Mr = 32 pixels, Ms = 3 pixels, and the speckle area of each sub-window is at about 40%. Benefited from the binary defocusing fringe projection technique, the optimized composite fringe patterns generated by the binary dithering algorithm are projected into the measured object by the DMD at the speed of20,000 Hz [45, 46]. In this work, two cameras are operated at a reduced image resolution (640 × 440) to match the speed of the projection system with an exposure time of . Since our approach only uses 4 composite structured-light patterns to recover one absolute phase map, our system can achieve a high-speed, dense, and accurate 3D shape measurement with 64-period fringe patterns at 5000 frames per second.
Then, to acquire 3D results with high completeness, it should also be emphasized here that the relative spatial positions between the cameras and the projector are carefully arranged in our multi-view system. In the conventional multi-view system [27, 28], the 3D measurement is realized by using the matching result of the dual-camera as shown in Fig. 10(a). Since the projector is usually placed between two cameras, the measurement accuracy can be improved by increasing the baseline of the dual-camera. But, in our system, the brightness of the binary fringe patterns, which is projected using the high-speed binary defocusing projection technique with limited exposure time, is much less than that of the 8-bit patterns [19, 47]. In order to ensure the measurement quality and void the influence of the ambient light, the measured area (i.e., the defocusing range of the projector) should be closer to the projector to enhance the pattern intensity. In fact, due to the large size of the DLP projector, it results in a reduction in the common area of the dual cameras located on both sides of the projector, which is not suitable for measuring objects with large cross-sections. To solve this problem,we try to narrow the baseline of the dual-camera and place the dual-camera on the same side of the projector, and then obtain the absolute phase map to achieve 3D measurement based on the main camera and the projector. Besides, in order to improve the accuracy of the measurement by increasing the baseline of the main camera and the projector, the auxiliary camera should be brought closer to the projector than the main camera as shown in Fig. 10(b). Meanwhile, as is mentioned in Section 2.2, reducing the baseline between two cameras can enhance the quality of stereo images, which can improve the correlation accuracy to realize robust and efficient phase unwrapping. The high-speed FPP system is finally shown in Fig. 11.
3.1. Precision analysis
To quantitatively analyze the precision of 3D measurement our system can achieve, we measured a standard ceramic plate with the assistance of the proposed method. The tested object and the corresponding one of the four-step phase-shifting patterns from main camera are shown in Figs. 12(a)–12(b). After the wrapped phase and the demodulated speckle pattern are obtained using Eqs. (12) and (15) as shown in Figs. 12(c)–12(d), geometry constraint is firstly executed to preclude some periodic candidate points, because its corresponding depth value is beyond the pre-defined depth range. However, there are still a lot of wrong candidate points for the fringe patterns with 64-period as shown in Fig. 12(e). Then the adaptive window image correlation can efficiently remove a lot of the periodic ambiguities and output the reliable periodic order map in Fig. 12(f). As a common method in stereo vision, left-right consistency check is carried out to eliminate wrong points with fringe order errors, which will results in the periodic order map and the absolute phase map with a few holes as shown in Figs. 12(g) and 12(k). Instead of the check operation, the period orders in the wrong regions can be reasonably compensated using RDC, and then the periodic order map, the absolute phase map, and the final 3D measurement results with higher completeness are presented in Figs. 12(h), 12(l), and 13(a). To obtain the measured errors, we perform the plane fitting using the 3D data in Fig. 13(a), then the fitted plane is set as the ground truth. The differences between the measured data and the ground truth are shown in Fig. 13(b), and we display the quantitative histograms of the differences as shown in Fig. 13(c). It can be easily found that the major measured errors are less than 100 with the rms of 61.73 , respectively. This result verifies that the proposed method can significantly increase the measurement accuracy due to the use of high-frequency fringe patterns and high SNR of the recovered phase map.
3.2. Static scenes measurement
In FFP, in order to acquire the 3D reconstructed results with high-quality, high-frequency fringe patterns are usually projected to restrain phase error and improve the precision of 3D measurement, but at the expense of the robustness of phase unwrapping. After confirming the 3D measurement precision of our system, as 64-period fringe patterns are used in our system, the further experiments need to be carried out to demonstrate the reliability of our method by measuring multiples objects with complex shapes. Some different objects were measured in Figs. 14(a)–14(c) including a girl statue with a ceramic plate, a Voltaire model, a statue of the Goddess, and the statue of Skadi, and the corresponding 3D results are shown in Figs. 14(d)–14(f). And then the corresponding enlarged details are presented to illustrate the reliability of our method which can achieve robust 3D shape measurement for objects with complex surfaces and geometric discontinuities in Figs. 14(g)–14(i).
Besides, in order to quantitatively analyze the robustness of phase unwrapping our method can achieve, the accuracy results of the absolute phase for different measured objects are calculated and shown in Table 1. The number of points is the sum of all valid points in from main camera. To realize the reliability analysis of the accuracy results, the absolute phase map obtained using traditional multi-frequency PSP with the same high-frequency fringe can be served as the reference phase. The left-right consistency check (L-R check) and the regional diffusion compensation algorithm (RDC) are performed as two separate groups. Then the correct matching rate and the error matching rate are easily calculated by making a comparison of the absolute phase for each valid point. The missing matching rate means the ratio of points which is excluded due to poor performance or is invalid in the reference phase. From the comparison results of Cmr and Mmr in Table 1, we caninfer that there exist some mismatched regions in the elementary absolute phase map before L-R check or RDC, and L-R check can remove a number of error points resulting in a relative change between Emr and Mmr. In contrast, the proposed RDC can substantially correct error points which can always provide the better Cmr and the lower Mmr. However, for edge regions and abrupt depth discontinuities of complex objects, the reference phase obtained using traditional multi-frequency PSP is slightly different to . In this case, these phases can be removed in L-R check, but for RGC, it can be adjusted to new phases using Eq. (17) which may be different from the corresponding ones of the reference phase. As a result, RDC has the slightly higher Emr compared with L-R check. Furthermore, since RDC can detect and correct the error points, the Cmr still keep at a high level even for measuring these challenging shapes, which demonstrates the reliability of our method for phase unwrapping.
3.3. Dynamic scenes measurement
Next, our system is applied to record a one-time transient event for high-speed 3D shape measurement: the statue of Voltaire and a free-falling balloon filled with water as shown in Fig. 15(a). Figures 15(b)–(f) show the color-coded 3D reconstruction results at different time points. During the whole dynamic measurement, the Voltaire model is placed alone on one side of the scene. On the other side, the balloon with water is artificially released in the air. Due to the acceleration of freely falling bodies, the balloon arrives quickly and happens to deform by the reaction of the ground. And then, the balloon has undergone severe deformation, losing its original smooth shape. Finally, the balloon gradually returns to its former appearance and bounces back into the air again. The whole 3D measurement results can be referred to in Visualization 1. In the whole measuring procedures, the surfaces of the Voltaire statue and the balloon are correctly reconstructed with high-quality, verifying the reliability of the proposed method to perform the absolute 3D shape measurement with high completeness at high speed.
In addition, our system is applied to imaging high-speed 3D shape measurement of a dynamic object: the rip of paper, and the corresponding color-coded 3-D reconstructions at different time points are shown in Fig. 16. Even if the objects are isolated in motion, moreover, have noticeable changes in surface conditions (such as convex and concave curvatures), no irregularities will be recognizable in the final result ( Visualization 2), which confirms that the proposed method can successfully measure multiple moving objects simultaneously.
In conclusion, we propose a high-speed 3D shape measurement technique based on composite structured-light patterns and a multi-view system. For the 3D measurement precision, by skillfully embedding the speckle pattern into the original phase-shifting fringes without decreasing the fringe amplitude, we realize unambiguous 3D shape measurement with 64-period fringe patterns, achieving a measurement precision of 61.73 . For the reliability for phase unwrapping, the optimized speckle pattern is mathematically generated based on our system configuration to ensure the period-wise uniqueness for each point. A simple and effective test method of the designed speckle pattern is presented to provide the better match results for phase unwrapping. Benefited from the optimized design method, the periodic ambiguities can be effectively eliminated using geometry constraint and the adaptive window image correlation. Furthermore, the mismatched regions will be substantially resolved using the regional diffusion compensation technique (RDC) based on spatial phase unwrapping algorithm. Finally, the absolute phase map with high completeness can be achieved by a whole computational framework above. For the speed of 3D measurement, our method can realize a high-speed, dense, and accurate 3D shape measurement at 5000 frames per second using only 4 composite patterns based on the binary defocusing fringe projection system. This high-performance method greatly improves the measurement speed and accuracy which can provide a powerful technical guide for some high-speed profilometry in engineering applications to a large extent.
There are several aspects that need to be further improved in the proposed method, which we will leave for future consideration. First, there are several parameters in the proposed approach, which should be properly selected for different kinds of objects. Currently, we empirically set these parameters according to the overall property of the test scene. It will be better if these parameters can be automatically determined and adaptively updated. Second, in RDC the smaller group is corrected according to the adjacent larger group. When the measured object contains small isolated areas (pixel number smaller than ), their fringe orders are not easy to be determined by the spatial correlation and are also quite error-prone in the RDC process. How to handle this situation is another interesting direction for further investigation.
National Key R&D Program of China (2017YFF0106403); National Natural Science Foundation of China (61722506, 61705105, 111574152); Final Assembly ‘13th Five-Year Plan’ Advanced Research Project of China (30102070102); Equipment Advanced ResearchFund of China (61404150202); The Key Research and Development Program of Jiangsu Province, China (BE2017162); Outstanding Youth Foundation of Jiangsu Province of China (BK20170034); National Defense Science and Technology Foundation of China (0106173); ‘Six Talent Peaks’ project of Jiangsu Province, China (2015-DZXX-009); ‘333 Engineering’ research project of Jiangsu Province, China (BRA2016407, BRA2015294); Fundamental Research Funds for the Central Universities (30917011204, 30916011322); Open ResearchFund of Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense (3091601410414); China Postdoctoral Science Foundation (2017M621747), and Jiangsu Planned Projects for Postdoctoral Research Funds (1701038A); National Science Centre Poland (2017/25/B/ST7/02049); Polish National Agency for Academic Exchange.
1. S. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Laser Eng. 48(2), 133–140 (2010). [CrossRef]
2. X. Su and Q. Zhang, “Dynamic 3-D shape measurement method: a review,” Opt. Laser Eng. 48(2), 191–204 (2010). [CrossRef]
3. S. Zhang, “High-speed 3-D shape measurement with structured light methods: A review,” Opt. Laser Eng. 106, 119–131 (2018). [CrossRef]
4. S. Feng, L. Zhang, C. Zuo, T. Tao, Q. Chen, and G. Gu, “High dynamic range 3-D measurements with fringe projection profilometry: A review,” Mea Sci Technol 29(12), 122001 (2018). [CrossRef]
5. X. Su and W. Chen, “Fourier transform profilometry: a review,” Opt. Laser Eng. 35(5), 263–284 (2001). [CrossRef]
7. V. Srinivasan, H. C. Liu, and M. Halioua, “Automated phase-measuring profilometry of 3-D diffuse objects,” Appl. Opt. 23(18), 3105–3108 (1984). [CrossRef]
8. C. Zuo, S. Feng, L. Huang, T. Tao, W. Yin, and Q. Chen, “Phase shifting algorithms for fringe projection profilometry: A review,” Opt. Laser Eng. 109, 23–59 (2018). [CrossRef]
9. Z. Cai, X. Liu, H. Jiang, D. He, X. Peng, S. Huang, and Z. Zhang, “Flexible phase error compensation based on Hilbert transform in phase shifting profilometry,” Opt. Express 23(19), 25171–25181 (2015). [CrossRef]
10. Z. Zhang, S. Huang, S. Meng, F. Gao, and X. Jiang, “A simple, flexible and automatic 3D calibration method for a phase calculation-based fringe projection imaging system,” Opt. Express 21(10), 12218–12227 (2013). [CrossRef] [PubMed]
11. S. Feng, C. Zuo, T. Tao, Y. Hu, M. Zhang, Q. Chen, and G. Gu, “Robust dynamic 3-D measurements with motion-compensated phase-shifting profilometry,” Opt. Laser Eng. 103, 127–138 (2018). [CrossRef]
12. Z. Zhang, “Review of single-shot 3D shape measurement by phase calculation-based fringe projection techniques,” Opt. Laser Eng. 50(8), 1097–1106 (2012). [CrossRef]
13. W. Yin, C. Zuo, S. Feng, T. Tao, Y. Hu, L. Huang, J. Ma, and Q. Chen, “High-speed three-dimensional shape measurement using geometry-constraint-based number-theoretical phase unwrapping,” Opt. Laser Eng. 115, 21–31 (2019). [CrossRef]
14. S. Heist, P. Dietrich, M. Landmann, P. Kühmstedt, G. Notni, and A. Tünnermann, “GOBO projection for 3D measurements at highest frame rates: a performance analysis,” Light: Science & Applications. 7(1), 71 (2018). [CrossRef]
16. J. S. Hyun, G.T.C. Chiu, and S. Zhang, “High-speed and high-accuracy 3D surface measurement using a mechanical projector,” Opt. Express 26(2), 1474–1487 (2018). [CrossRef]
17. S. Feng, Y. Zhang, Q. Chen, C. Zuo, R. Li, and G. Shen, “General solution for high dynamic range three-dimensional shape measurement using the fringe projection technique,” Opt. Laser Eng. 59, 56–71 (2014). [CrossRef]
18. Y. Hu, Q. Chen, S. Feng, T. Tao, H. Li, and C. Zuo, “Real-time microscopic 3-D shape measurement based on optimized pulse-width-modulation binary fringe projection,” Mea Sci Technol 28(7), 075010 (2017). [CrossRef]
19. C. Zuo, T. Tao, S. Feng, L. Huang, A. Asundi, and Q. Chen, “Micro Fourier Transform Profilometry (μFTP): 3-D shape measurement at 10,000 frames per second,” Opt. Laser Eng. 102, 70–91 (2018). [CrossRef]
20. X. Su and W. Chen, “Reliability-guided phase unwrapping algorithm: a review,” Opt. Laser Eng. 42(3), 245–261 (2004). [CrossRef]
21. T. J. Flynn, “Two-dimensional phase unwrapping with minimum weighted discontinuity,” J. Opt. Soc. Am. A 14(10), 2692–2701 (1997). [CrossRef]
22. D. C. Ghiglia and L. A. Romero, “Minimum Lp-norm two-dimensional phase unwrapping,” J. Opt. Soc. Am. A 13(10), 1999–2013 (1996). [CrossRef]
23. M. Zhao, L. Huang, Q. Zhang, X. Su, A. Asundi, and K. Qian, “Quality-guided phase unwrapping technique: comparison of quality maps and guiding strategies,” Appl. Opt. 50(33), 6214–6224 (2011). [CrossRef] [PubMed]
24. C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review,” Opt. Lasers Eng. 85, 84–103 (2016). [CrossRef]
25. G. Sansoni, M. Carocci, and R. Rodella, “Three-dimensional vision based on a combination of gray-code and phase-shift light projection: analysis and compensation of the systematic errors,” Appl. Opt. 38(31), 6565–6573 (1999). [CrossRef]
27. K. Zhong, Z. Li, Y. Shi, C. Wang, and Y. Lei, “Fast phase measurement profilometry for arbitrary shape objects without phase unwrapping,” Opt. Laser Eng. 51(11), 1213–1222 (2013). [CrossRef]
28. Z. Li, K. Zhong, Y. Li, X. Zhou, and Y. Shi, “Multiview phase shifting: a full-resolution and high-speed 3-D measurement framework for arbitrary shape dynamic objects,” Opt. Lett. 38(9), 1389–1391 (2013). [CrossRef]
29. Y. An, J. S. Hyun, and S. Zhang, “Pixel-wise absolute phase unwrapping using geometric constraints of structured light system,” Opt. Express 24(16), 18445–18459 (2016). [CrossRef]
30. T. Tao, Q. Chen, S. Feng, Y. Hu, M. Zhang, and C. Zuo, “High-precision real-time 3-D shape measurement based on a quad-camera system,” J. Optics 20(1), 014009 (2017). [CrossRef]
32. C. Zuo, Q. Chen, G. Gu, S. Feng, F. Feng, R. Li, and G. Shen, “High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection,” Opt. Laser Eng. 51(8), 953–960 (2013). [CrossRef]
33. C. Zuo, Q. Chen, G. Gu, S. Feng, and F. Feng, “High-speed three-dimensional profilometry for multiple objects with complex shapes,” Opt. Express 20(17), 19493–19510 (2012). [CrossRef]
34. T. Weise, B. Leibe, and L. Van Gool, “Fast 3d scanning with automatic motion compensation,” 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE pp. 1–8 (2007).
35. R.R. Garcia and A. Zakhor, “Consistent stereo-assisted absolute phase unwrapping methods for structured light systems,” IEEE Journal of Selected Topics in Signal Processing 26(5), 411–424 (2012). [CrossRef]
37. K. Song, S. Hu, X. Wen, and Y. Yan, “Fast 3D shape measurement using fourier transform profilometry without phase unwrapping,” Opt. Laser Eng. 84, 74–81 (2013). [CrossRef]
38. T. Tao, Q. Chen, J. Da, S. Feng, Y. Hu, and C. Zuo, “Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system,” Opt. Express 24(18), 20253–20269 (2016). [CrossRef] [PubMed]
39. W. Lohry and S. Zhang, “High-speed absolute three-dimensional shape measurement using three binary dithered patterns,” Opt. Express 22(22), 26752–26762 (2014). [CrossRef]
40. S. Feng, Q. Chen, and C. Zuo, “Graphics processing unit–assisted real-time three-dimensional measurement using speckle-embedded fringe,” Appl. Opt. 54(22), 6865–6873 (2015). [CrossRef]
41. Y. Wang, K. Liu, Q. Hao, D. L. Lau, and L. G. Hassebrook, “Period coded phase shifting strategy for real-time 3-D structured light illumination,” IEEE Transactions on Image Processing 20(11), 3001–3013 (2011). [CrossRef]
42. X. Liu and J. Kofman, “High-frequency background modulation fringe patterns based on a fringe-wavelength geometry-constraint model for 3-D surface-shape measurement,” Opt. Express 25(14), 16618–16628 (2017). [CrossRef] [PubMed]
43. P. Zhou, J. Zhu, and H. Jing, “Optical 3-D surface reconstruction with color binary speckle pattern encoding,” Opt. Express 26(3), 3452–3465 (2018). [CrossRef]
44. B. Pan, Z. Lu, and H. Xie, “Mean intensity gradient: an effective global parameter for quality assessment of the speckle patterns used in digital image correlation,” Opt. Laser Eng. 48, 469–477 (2010). [CrossRef]
46. W. Lohry and S. Zhang, “Genetic method to optimize binary dithering technique for high-quality fringe generation,” Opt. Lett. 38(4), 540–542 (2013). [CrossRef]
47. Y. Wang and S. Zhang, “Three-dimensional shape measurement with binary dithered patterns,” Appl. Opt. 51(27), 6631–6636 (2012). [CrossRef]