1. Introduction

High-speed three-dimensional (3D) shape measurement is increasingly used due to the ever-growing computation capabilities of personal computers and even mobile devices nowadays. Digital fringe projection (DFP) technique is one of the most popular methods for high-resolution and high-speed 3D shape measurement because of its flexible fringe generation nature [1]; and phase-shifting algorithms are extensively used due to its accuracy, speed, and robustness to noise. However, a phase-shifting algorithm typically only provides phase values ranging from −π to +π [2]. To perform 3D shape measurement, 2π discontinuities have to be removed by employing a phase unwrapping algorithm.

Numerous phase unwrapping algorithms have been developed over the years. In general, conventional phase unwrapping methods can be classified into two categories: spatial phase unwrapping and temporal phase unwrapping. The former methods unwrap the phase of a point by referring to phase values of other points through optimization, under the assumption that surface geometry is smooth. The book edited by Ghiglia and Pritt [3] summarized a number of spatial phase unwrapping methods; Su and Chen [4] reviewed a number of robust quality-guided spatial phase unwrapping algorithms, and Zhao et al. [5] compared different strategies of generating a quality map to guide a phase unwrapping path. Despite the robustness improvements of recently developed spatial phase unwrapping algorithms, they are all fundamentally limited to measure smooth surfaces: the object has to be smooth on at least one path such that the object surface geometry will not introduce more than π phase changes between two successive points. In general, spatial phase unwrapping only provides a relative phase map that depends on the starting point of the unwrapping process. As a result, only relative 3D geometry can be recovered.

Temporal phase unwrapping method, in contrast, can retrieve absolute phase by acquiring additional images at a different time. A typical temporal phase unwrapping algorithm unwraps the phase of a point without using its neighboring phase values. Two and multi-wavelength phase-shifting algorithms [6–8] have been developed for laser interferometry, and hybrid binary coding with phase-shifting methods have been developed for DFP systems [9, 10]. They all work well and are extensively employed. However, the requirement of additional images slows down the entire measurement speed.

The nonconventional absolute phase retrieval methods include adding more camera(s) or projector(s) to capture or project from more than two perspectives such that the geometric constraints of the multi-view system can be used to determine absolute phase for each point through optimization [11–17]; or performing hybrid stereo-vision for coarse measurement and using the wrapped phase constraint for refinement or high-resolution measurement [18]. These methods can work well if the computational framework is properly developed. However, due to the requirement of more than two perspectives, the overall measurement area is reduced: the point has to be seen from all perspectives. Furthermore, the complexity and cost of such a hardware system increase due to the use of additional hardware components.

In short, spatial phase unwrapping methods do not require additional image acquisition or hardware, yet they cannot recover absolute phase; the temporal phase unwrapping methods sacrifice the measurement speed for absolute phase recovery; and multi-view geometry based approaches can recover absolute phase without acquiring additional images, yet they increase system cost and complexity. To address the limitations associated with all existing absolute phase recovery methods, An et al. [19] developed a geometric-constraint based phase unwrapping method that only requires a single camera and a single projector without the need of capturing additional images. This approach sets a distance z_min from the nearest point of object to the camera to generate a minimum phase Φ_min using the geometric constraints of the DFP system. Φ_min is then used to unwrap phase pixel by pixel; and details of this phase unwrapping method will be presented in Sec. 2.2. However, this method is constrained to measure a limited depth range of objects. To maximize sensing depth range, it is desirable to precisely know the closest depth z_min value from the object to camera, making it difficult to be employed for applications where the object moves along the z direction during measurement.

This paper presents a novel method for absolute phase measurement to drastically enhance the method developed by An et al. [19] by relaxing the constraint of determining z_min before measurement. Our proposed method uses a known object (i.e., a ping-pong ball with a 20 mm radius) to provide cues for dynamical z_min determination. In particular, during the measurement, the ping-pong ball is captured with objects to be measured, and the ping-pong ball is positioned to be close to the nearest point of an object to camera. We segment the ping-pong ball and spatially unwrap its phase. Then we employ an optimization algorithm to determine the integer multiple of 2π’s to be added such that the recovered geometry matches its actual shape. The nearest point of the ball provides z_min to generate a minimum phase map Φ_min that can be further used to unwrap phase pixel by pixel for 3D absolute shape reconstruction. If the ping-pong ball is moving with the object, z_min can be determined dynamically for moving object measurement. Experiments demonstrated that only three phase-shifted fringe patterns are required to measure an absolute shape of a dynamic object.

2. Principles

2.1. Three-step phase-shifting algorithm

Phase-shifting algorithms are extensively used in optical metrology because of its accuracy, speed, and robustness to noise. Over years, numerous phase shifting algorithms have been developed with some being more robust than others [2]. For high-speed applications, a three-step phase-shifting algorithm is typically used since it uses the minimum number of patterns to recover the phase pixel by pixel. For a three-step phase-shifting algorithm with equal phase shifts, three fringe images can be described as

Here k(x, y) is an integer number and is often called fringe order. If the fringe order k(x, y) can be uniquely determined based on a pre-defined value, then the unwrapped phase Φ(x, y) is regarded as an absolute phase. As discussed in Sec. 1, fringe orders k(x, y) determined by spatial phase unwrapping methods are typically relative to one point on the phase map, and thus spatial methods cannot give an absolute phase. In contrast, temporal phase unwrapping methods determine absolute fringe orders k(x, y) by referring to additional acquired information (e.g., additional images from the same or different perspectives), and that can lead to an absolute phase.

2.2. Geometric constraint-based phase unwrapping algorithm

An et al. [19] developed the absolute fringe order determination method using inherent geometric constraints of a standard DFP system (i.e., a single camera and a singe projector). For simplicity, let us assume both camera and projector are calibrated under the same coordinate system based on a linear pinhole model. The projections from 3D world coordinates to 2D sensor planes can be described as,

Alternatively, if we define a virtual plane at z = z^w, each camera pixel (u^c, v^c) can have a unique corresponding projector pixel coordinates (u^p, v^p) by solving Eqs. (7)–(8). The corresponding phase can be determined by referring to (u^p, v^p) on the projector sensor plane. Therefore, for a given plane with constant z, an entire phase map of the camera image can be created for that virtual plane. If z = z_min, we call the virtually created phase map the minimum phase map, or Φ_min. Apparently, Φ_min is a function of the distance z_min, fringe period T, and projection matrices P^c and P^p,

Figure (1) illustrates the basic idea of using the minimum phase for phase unwrapping. Assume the red dashed window shows the mapped projector region that the camera captures if an ideal plane is positioned at z = z_min. Since Φ_min is defined on the projector space, it is continuous without requiring any phase unwrapping, as shown in Fig. 1(b). For a plane at a depth z > z_min, the wrapped phase ϕ, captured by the camera, is mapped to the region on the projector marked as the solid blue window. Figure 1(c) shows the cross sections of these phase maps. Clearly, if ϕ < Φ_min, 2π should be added to unwrap the phase ϕ. An et al. [19] showed that the fringe order k(x, y) can be determined by

Here, ceil[] is an operator that gives the nearest upper integer value.

 

Fig. 1 Concept of phase unwrapping using Φ_min [19]. (a) Red dashed window shows camera captured area when the virtual plane is at z = z_min and the solid blue window shows the mapped region the virtual plane is at z > z_min; (b) corresponding Φ_min and Φ on the projector; (c) cross sections of Φ_min and Φ and the wrapped phase maps for the case of only one single 2π discontinuous.

Download Full Size | PDF

2.3. Proposed absolute phase recovery method

The geometric constraint-based phase unwrapping method discussed in Sec. 2.2 has two major advantages [21]: 1) it does not require any additional image acquisition, and thus it is more suitable for high-speed applications; and 2) it is robust to noise because it determines fringe order by referring to an artificially generated ideal absolute phase map Φ_min that is noise free.

However, this phase unwrapping method has the major limitation of confined measurement depth range. The maximum depth range is approximately

To address this problem, we proposed a novel approach to dynamically determine proper z_min value by simultaneously capturing a known-size object (e.g., a ping-pong ball for our case) that is moving along with the object.

Figure 2 explains the pipeline of our proposed absolute phase unwrapping method. Three phase-shifted fringe images are captured by the camera, and they include the object(s) to be measured along with a known size object (e.g., a ping-pong ball). Wrapped phase ϕ(x, y) can be obtained by applying Eq. (5), and the texture image I_t (x, y) can also be extracted by Eq. (4). We then employ an image segmentation algorithm to detect the ping-pong ball from the texture image and separate the ping-pong ball from the rest scene. Since the texture image and the wrapped phase map come from the same phase-shifted fringe patterns, the corresponding wrapped phase ϕ_b (x, y) for the ping-pong ball can also been extracted. Because the ping-pong ball surface is smooth, we apply a spatial phase unwrapping algorithm [22] to unwrap the phase. Here we can only obtain a relative unwrapped phase ${\mathrm{\Phi }}_b^r(x,y)$ for the ping-pong ball. This spatially unwrapped phase could have an offset k₀ × 2π (k₀ is an integer for the entire phase map) from the absolute phase, i.e.,

 

Fig. 2 Our proposed absolute phase unwrapping pipeline. Three phase-shifted fringe patterns are used to obtain wrapped phase map ϕ(x, y) and texture image I_t (x, y). The ping-pong ball is segmented from the texture image and its phase map is extracted from the wrapped phase map ϕ(x, y). Since the spatially unwrapped phase map of the ping-pong ball is shifted by 2k₀π from absolute phase, an optimization algorithm is employed to determine k₀ such that reconstructed 3D geometry matches the actual geometry of the ping-pong ball. The reconstructed ping-pong ball is then used to extract z_min for Φ_min computation. The entire phase map (excluding the sphere) is unwrapped pixel by pixel by referring to Φ_min map. The final unwrapped phase map can be used to reconstruct 3D geometry directly.

Download Full Size | PDF

To extract the constant number k₀ for the ping-pong ball, we developed an optimization approach. From the projected fringe patterns, we know that k₀ is bounded by

Once k₀ is determined, we can recover 3D geometry of the sphere, and determine the minimum z=z_min for Φ_min generation. Typically, if the ping-pong ball is slightly closer to the camera than the object, the minimum z value on the sphere can be used as z_min directly. The computed Φ_min map is then used to unwrap the object surface pixel by pixel by using Eq. (10). Finally, the unwrapped phase is used to recover the entire 3D surface geometry.

It should be noted that in this research, we use a ping-pong ball as an example for this proposed pipeline because its surface geometry is simple and uniform; and it is very inexpensive and very easy to obtain. In practice, one can use any known object to replace the ping-pong ball and the entire pipeline can still be adapted to determine k₀. If the object surface is complex, one may measure the object in advance and determine k₀ by solving the following minimization problem:

3. Experiment

We developed a 3D shape measurement system to test the performance of our proposed method. The hardware system includes a single charge-coupled device (CCD) camera (Model: Pointgray Grasshopper3) and a digital light processing (DLP) projector (Model: LightCrafter 4500). The camera is attached with a 25 mm focal length lens (Model: Fujinon CF25HA-1). The camera resolution was set as 1280 × 960 pixels and the projector’s resolution is 1140 × 912 pixels. The projector projects binary structured patterns at 100 Hz, and the camera captures simultaneously with external trigger signals generated by Arduino. The entire system was calibrated by employing the structured light system calibration method developed by Li et al. [23].

We first verified that our optimization algorithm can generate absolute phase by measuring ping-pong balls. Figure 3 shows a single ping-pong ball measurement results. Figure 3(a) shows one of the three phase-shifted fringe patterns, and Figure 3(b) shows the wrapped phase. Since the sphere surface is smooth, we employed a spatial phase unwrapping algorithm [22] to unwrap the phase, and Fig. 3(c) shows the spatially unwrapped phase map.

 

Fig. 3 Measurement result of a single sphere. (a) One of there phase-shifted fringe patterns; (b) wrapped phase; (c) spatially unwrapped phase.

Download Full Size | PDF

Since the spatially unwrapped phase is a relative phase, the phase map has to be properly shifted by 2π × k₀ to recover an absolute 3D shape. We employed our optimization algorithm on the relative phase by adding different number of 2π’s. Figure 4 shows three 3D frames during the optimization process with different k values. As one can see, when k changes, the size of the reconstructed sphere changes accordingly. Figures 4(a)–4(c) show 3D results when k = 15, 16, and 17, respectively. Figures 4(d)–4(f) show the cross sections of Fig. 4(a)–4(c). Visually, when k = 16, the reconstructed sphere matches with the ideal sphere fairly well.

 

Fig. 4 3D reconstruction examples with different number of 2π offsets for the spatially unwrapped phase map shown in Fig. 3(c). Sphere in red is the ideal sphere with a radius of 20 mm. (a) 3D result when k = 15; (b) 3D result when k = 16; (c) 3D result when k = 17; (d) one cross section of reconstructed sphere and the ideal sphere for (a); (e) one cross section of reconstructed sphere and the ideal sphere for (b); (f) one cross section of reconstructed sphere and the ideal sphere for (c).

Download Full Size | PDF

To automatically determine a proper number of 2π to shift relative phase vlaues, we fitted a sphere for each 3D geometry reconstructured, and find the radius of each fitted sphere. Figure 5 shows the fitted sphere radius with respect to the changes of k. This figure shows that the relationship between estimated radius and k are monotonic; and k = 16 gives the radius that is closest to 20 mm. So we use k₀ = 16 for absolute phase recovery.

 

Fig. 5 Changes of fitted sphere radius with different number k of 2π added to the spatially unwrapped phase map for 3D reconstruction.

Download Full Size | PDF

Once k₀ is obtained, the absolute phase can be obtained, and further 3D shape can be recovered. Figure 6(a) and 6(b) respectively shows unwrapped absolute phase and 3D reconstructed shape. We then compared the result obtained from our proposed method with that obtained from the conventional temporal phase unwrapping method [10]. Figure 6(c) shows the reconstructed absolute phase by the temporal phase unwrapping method; and Fig. 6(d) shows the corresponding 3D shape; they appear identical.

 

Fig. 6 Comparing results between our proposed method and the conventional temporal phase unwrapping method. (a) Unwrapped phase from our method; (b) 3D reconstructed geometry from our method; (c) unwrapped phase from conventional temporal phase unwrapping method; (d) 3D reconstructed geometry from conventional temporal phase unwrapping method.

Download Full Size | PDF

We then plotted the same cross sections of the unwrapped phase, as shown in Fig. 7(a). From Fig. 7(a), we can find that the unwrapped phase maps perfectly overlap with each other. Similarly, we plotted the same cross sections of the reconstructed 3D shapes, and Figure 7(b) shows the result. Once again, they are identical. This experiment demonstrated that our proposed method can successfully measure a known-size ping-pong ball. Our result is consistent with the conventional absolute shape measurement technique, albeit it requires a lot less number of images.

 

Fig. 7 Cross sections of the results from our proposed method and the conventional temporal phase unwrapping method. (a) Unwrapped phase maps; (b) 3D shapes.

Download Full Size | PDF

To further verify that our proposed method can work well for multiple known size objects at different depths z. We simultaneously measured five ping-pong balls. Figure 8 shows the measurement results. Once again, we compared the results between our method and the conventional temporal phase unwrapping method. Figure 8 shows that they are identical to our results. This experiment further demonstrated that our proposed optimization method can indeed recover an absolute phase map of multiple isolated known objects (e.g., ping-pong balls) without capturing additional images.

 

Fig. 8 Measurement results of multiple isolated known size ping-pong balls. (a) One of three phase-shifted fringe patterns; (b) 3D reconstruction from our method; (c) 3D reconstruction from a conventional temporal phase unwrapping method.

Download Full Size | PDF

After proving that our proposed optimization method can be used to recover absolute shape of the ping-pong ball at different depths and different spatial locations, we can use that to assist absolute phase unwrapping for arbitrary objects. We simultaneously measured one ping-pong ball and two sculptures. The ping-pong ball was positioned slightly closer to the camera than the sculptures. Figure 9 shows the result. Figure 9(a) shows one of three phase-shifted fringe patterns. From three phase-shifted fringe patterns, we can obtain the texture image and the wrapped phase map, as shown in Fig. 9(b) and 9(c) respectively. From the texture image, we segment the ping-pong ball and obtain the wrapped phase accordingly, as shown in Fig. 9(d) and Fig. 9(e). Figure 9(f) shows the spatially unwrapped phase of the segmented ball. After employing our proposed optimization algorithm, absolute shape of the ping-pong ball can be obtained, as shown in Fig. 9(g). Finally, we reconstructed the 3D absolute shape and the result is shown in Fig. 9(h).

 

Fig. 9 Measurement of two isolated complex sculptures using a known-size ping-pong ball. (a) One of the three phase-shifted fringe patterns; (b) texture image; (c) wrapped phase map; (d) segmented ping-pong ball; (e) wrapped phase map of the ping-pong ball; (f) spatial unwrapped phase map of the ping-pong ball; (g) unwrapped phase after our proposed optimization algorithm; (h) 3D absolute shape of the ping-pong ball.

Download Full Size | PDF

The reconstructed 3D geometry of the ping-pong ball gives the z_min value to generate the minimum phase map, Φ_min, as discussed in Sec. 2.2. After Φ_min is obtained, the entire phase map can be unwrapped pixel by pixel. Figure 10(a) shows the unwrapped phase map, from which we can recover 3D geometry of the entire scene. The 3D result is shown in Fig. 10(b). As before, we compared our measurement result with that obtained from the temporal phase unwrapping method [10]. Figure 10(c) shows the unwrapped phase and Fig. 10(d) shows the the corresponding 3D reconstruction. Once again, they are identical to the results obtained from our proposed method.

 

Fig. 10 Comparing results between our proposed method and the conventional temporal phase unwrapping method. (a) Unwrapped phase map from our method; (b) 3D reconstruction using phase map shown in (a); (c) unwrapped phase map from conventional temporal phase unwrapping method; (d) 3D reconstruction from the phase map shown in (c).

Download Full Size | PDF

Furthermore, we measured two isolated objects at distinct depth z, as shown in Fig. 11. Since the depth span of these two objects is larger than the maximum sensing range that the Φ_min can handle, two ping-pong balls are required with each being positioned close to one object. Figure 11(a) shows one fringe image of the captured scene. We segment each ping-pong ball and its associated with the object, and separate them into two sub images and process each sub image following the same procedures as we used before. The top half images of Figs. 11(b) and 11(c) show the segmented sub-images; and the bottom half images of Figs. 11(b) and 11(c) show the recover 3D shape for each sub image.

 

Fig. 11 Measurement result of two isolated objects with large depth differences. (a) One of the fringe images; (b) segmented image and 3D reconstruction for the near objects; (c) segmented image and 3D reconstruction for the far object.

Download Full Size | PDF

We combine two separately recovered 3D shapes into one entire 3D object. Figure 12 shows the complete measurement result. Again, we measured the same object using gray coding, and the result is shown in Fig. 12(b). Figure 12(c) shows two cross sections of the measurement results using our proposed method and the gray coding method. Once again, they are identical.

 

Fig. 12 3D result of the entire scene. (a) 3D result by combining result for Figs. 11(b) and 11(c); (b) 3D reconstruction from the gray-coding method; (c) two different cross sections of the 3D results.

Download Full Size | PDF

Lastly, we experimentally verified that our proposed method can be used to measure objects moving along the depth z direction. In this experiment, we measured a ping-pong ball moving along with a hand. We projected two different frequency fringe patterns that allow us to recover absolute phase using the enhanced two frequency method [24] for absolute phase unwrapping. Our method only uses three higher frequency fringe patterns for 3D shape recovery. Figure 13 as well as the associated videos Visualization 1 and Visualization 2 show the measurement results. Figure 13(a) shows one frame of the 3D result obtained from our proposed method using the three high-frequency fringe patterns. Figure 13(b) shows the result of the same frame using the enhanced two-frequency phase-shifting method using 6 fringe patterns. This experiment successfully demonstrated that our proposed method can be used to measure dynamic object moving along depth z direction, enhancing the geometric-constraint based phase unwrapping method developed by An et al. [19]. Since only three fringe patterns are used, this proposed method is very suitable for high-speed measurement applications.

 

Fig. 13 Measurement result of moving object along depth direction (associated with Visualization 1 and Visualization 2). (a) Result from our proposed method; (b) result from enhanced two-frequency phase unwrapping method.

Download Full Size | PDF

It is important to note that, on these two visualizations, one may notice some periodical error on those reconstructed 3D data. The periodical error was a result of motion artifacts due to the “slow” 2D fringe image acquisition speed (i.e., 100 Hz). It takes approximately 33 ms to capture three phase-shifted fringe patterns to reconstruct one 3D frame. For slowly moving hand portion, 33 ms is short enough without introducing any problems. However, for some portion of the video recording, the hand moves faster and 33 ms is too long. As one can clearly see, the motion-introduced artifacts appears similarly on the results reconstructed from our method and the results from the two-frequency phase-shifting method.

4. Summary

This paper has presented a novel method for absolute three-dimensional (3D) shape measurement that uses a known-size object (i.e., a ping-pong ball with a radius of 20 mm) to provide cues for absolute phase unwrapping. The phase of segment ping-pong ball is spatially unwrapped to retrieve relative phase, and the integer multiple of 2π to be added for absolute phase recovery is obtained through an optimization process. The nearest point of the reconstructed 3D ball provides z_min to generate the minimum phase Φ_min for pixel-by-pixel phase unwrapping of the entire wrapped phase map. Experimental results demonstrated the success of our proposed method to measure absolute shape of moving objects using only three phase-shifted fringe patterns.