1. Introduction

In recent years, fringe projection profilometry (FPP) has witnessed tremendous progress and has attracted extensive interest owing to its high speed, accuracy and non-contact measurement capability [1]. The measurement system is usually composed of a projector and a camera, where fringe patterns are projected onto the tested surface and then captured. By analyzing the distortion between the captured and projected patterns, the 3D point cloud of the object is obtained. Based on the assumption that only the rays directly reflected into the camera are received, the performance of the FPP significantly degenerates from inter-reflections [2]. To improve the performance, inter-reflection removal methods have been the focus of a numerous studies during the last decade.

The most straightforward method of removing inter-reflections is to separate the inter-reflection component from the direct component by assuming that the former has a different spectral response [3] or polarization properties [4] from the latter. However, the performance is unstable for surfaces with various colors, and more hardware components, e.g. polarizers are needed, which makes the adjustment of the different components difficult and the system more complicated.

To tackle the above problems, methods robust to color variation and without additional hardware have been proposed. Nayar et al [2] demonstrated that, when the frequency of the fringe pattern is sufficiently high and the surface is almost completely diffuse or a Lambertian surface, the inter-reflection component can be separated from the direct component. Several subsequent methods have been developed based on Nayar’s method, in which the researchers modulated low-frequency patterns with high-frequency signals [5], limited the frequency to a narrow high band [6–9], or constructed other high-frequency patterns e.g. white noise like pattern [10–12]. Although these methods have achieved impressive results in terms of inter-reflection removal, they cannot be applied to inter-reflections induced by specular reflections.

More advanced methods in this regard have been proposed, which are called adaptive projection techniques [13]. The basic idea here is that an inter-reflection region in a captured image is detected, and with a mapping between the camera and projector image planes (established by phases of the captured fringes), the corresponding region in the projector image where the projection intensity is reduced or set zero is determined to remove any inter-reflections. The key step of such methods is the inter-reflection detection in the captured images based on certain measures. By measuring the consistency of different measurement of the same point, Tomoaki et al.[14] successfully detected inter-reflections. Exploiting the epipolar constraint, Zhao et al [15] conducted detection and removal using an epipolar imaging technique. Similarly, Qi et al.[16] also demonstrated that the epipolar constraint can be used to detect invalid-points induced by inter-reflections in more general FPP cases. Despite the success of detection, the measures above are general measures that can be used to detect all kinds of invalid-points, and will therefore result in numerous irrelevant or incorrect regions detected for inter-reflection removal.

To solve the problems above, a micro-frequency shifting (MFS) projection technique is proposed that does not require any additional modification of the measurement system, and generalizes well to all types of inter-reflection (induced by either diffuse or specular reflections).

The remainder of this paper is arranged as follows. Section 2 provides the basic principle of the FPP and the formation of inter-reflections. In Section 3 the proposed MFS is described in detail. Subsequently, the experiment results and some discussion are presented in section 4. Section 5 provides the conclusion.

2. Fundamental principle

2.1 Principle of typical FPP

A typical FPP measurement system consists of a projector and a camera, as shown in Fig. 1(a), where horizontal and vertical fringe patterns are projected onto the tested surface respectively, and captured in succession by the camera. The captured fringe pattern can be expressed as follows: [17,18]

 

Fig. 1. FPP in the presence of inter-reflection: (a) diagram of a FPP system; (b) inter-reflection from concave surfaces, where blue solid line represents direct reflection component and red dashed line is inter-reflection.

Download Full Size | PDF

The wrapped and unwrapped phases are calculated subsequently as follows: [19]

where M is the number of frequency. Finally INT is the round function.

With unwrapped phases φ_H(x,y) and φ_V(x,y) of the horizontal and vertical fringe patterns, the corresponding projector image coordinate of each pixel in the captured patterns is retrieved as follows:

The process of retrieval of the projector image coordinate refers to the establishment of a correspondence, which relates the camera and projector image planes with a one-to-one mapping. Finally, with the corresponding image pairs, the 3D coordinate is obtained.

2.2 Formation of inter-reflection

As shown in Fig. 1(b), in real-world scenes, the camera not only receives directly reflected light, e.i. a direct reflection component (DC), which is considered as the signal, but also “indirectly” reflected light, e.i. an inter-reflection component (IC), which is treated as noises or a kind of error source. Thus, the final received image intensity is the superposition of direct and inter-reflection components, and for a certain point i the intensity is expressed as

There are two types of inter-reflection: that caused by diffuse reflection, which is referred to as a diffuse inter-reflection (DIR) in this study, and that caused by a specular reflection, which is called specular inter-reflection(SIR) herein. According to Nayar’s theory [2] of inter-reflection caused by a diffuse reflection, the intensity of a DIR is the superposition of intensities of all points j inter-reflecting toward point i, as shown in Fig. 2(a), where j≠i and belongs to the surface. The intensity of a DIR is expressed as follows:

 

Fig. 2. Inter-reflection caused by (a) diffuse reflection and (b) specular reflections, where the solid red and blue lines represent DC and IC respectively, indicating the final received ray by the camera is a sum of DC and IC. Please note that since we focus on only inter-reflection removal, other global illuminations such as subsurface scatting are ignored in the figure.

Download Full Size | PDF

The intensity of the SIR has a far more simpler expression owing to the BRDF [20] of a sharply narrow specular-reflection lobe. More specifically, the BRDF decreases exponentially with the angle bias from the specular direction, which leads to a non-zero attenuation factor for the ray from j to i near the specular direction, and is otherwise approximately zero. Accordingly, N in Eq. (6) is reduced to a finite number. Without generality, considering the simplest situation where for each point i, only one point j satisfies the condition above, as shown in Fig. 2(b). As a result, the intensity of IC reduces to a simpler form:

According to Eq. (6), it was proved [2] that the intensity of DIR approximates to a constant when the frequency of the projected patterns is sufficiently high. In addition, under this situation, the constant intensity caused by an inter-reflection induces no phase error, because the retrieved phase does not change when a constant is added according to the phase-shift algorithm, as shown in Eq. (2). That is, if the frequency of projected fringe is sufficiently high, DIR will generate no errors.

In contrast to a DIR, Eq. (7) implies that the intensity of an SIR varies with the phase-shift of the projected patterns, which violates the basic assumption above. The difference between Eqs. (6) and (7) above explains why almost all high-frequency pattern projection methods [6–12] cannot apply to inter-reflections caused by specular reflections. Thus, a further analysis of the SIR is conducted.

2.3 Modulation variation with frequency

In the case of phase-shifting fringe patterns, the captured intensity of a pixel is a cosine wave with respect to phase-shift variable t, according to Eq. (1). Consequently, combining Eqs. (7) and (5), we obtain the following expression of the captured intensity including the IC and DC:

Subsequently, Eq. (8) returns to a simple form as

To investigate the synthetic modulation, we obtain its analytic expression as follows

Recalling Eq. (4), the phase relates directly to a certain corresponding projector image coordinate by φ = 2πfx_p, where f is the spatial frequency of the projected fringes. Combined with this relation, Eq. (11) takes the form below. The meaning of Eq. (12) is illustrated in Fig. 3.

 

Fig. 3. Inter-reflection from point j to i.

Download Full Size | PDF

According to Eq. (12), for a certain point i, the synthetic modulation B’ is a cosine function of f, which provides a novel measure for detecting the SIR. Specifically, by modulating f in a known manner, we can expect to observe a clear variation in the modulation in inter-reflection regions, whereas other regions with only a direct reflection component have no variation. Based on the observation, inter-reflection regions can be detected robustly by determining regions with a modulation variation larger than a certain level or threshold. Insight into the modulation variation establishes the basis of the proposed MFS technique. Note that during the derivation of the equation above, for simplification, it is assumed that only one point, e.i. j, satisfies the law of reflection, but the conclusion generalizes well to multi-point cases.

3. Micro frequency shifting projection

3.1 Primary parameters selection

According to the analysis in Sec. 2.3, by modulating the frequency of fringes f, inter-reflection regions can be determined by detecting regions with high variation. There are various “modulating” forms of the fringe frequency, and the simplest one, e.i. the linear one as shown in Eq. (13), is chosen.

Thus, by substituting f in Eq. (12) with Eq. (13), Eq. (12) has the following form:

During the design of the fringe patterns, there are three primary parameters to be determined, namely, the base frequency f₀, the frequency-shift Δf and the number of frequency T. There are several principles for a parameter selection. First, the frequency should be sufficiently high to guarantee that the phase error caused by a DIR is sufficiently small or near to zero under a diffuse reflection. Therefore, the base frequency f₀ (which corresponds to the smallest frequency when t = 0 according to Eq. (13)) should be sufficiently high. Second, the number of frequency should be as small as possible, because the larger the number, the more patterns that will be projected and the more time cost required. Finally, according to Eq. (14), to measure the synthetic modulation B’, the frequency-shift Δf must satisfy the Nyquist sampling theorem, namely 2πΔxΔf≤π, and to measure the largest variation of B’, the frequency extension in time should be larger than one period, namely 2πΔxΔfT ≥ 2π. Consequently, the frequency-shift should satisfy the inequation as follows:

 

Fig. 4. Examples of MFS patterns.

Download Full Size | PDF

3.2 Framework of the inter-reflection removal

Based on the analysis above, in this section, we provide details on how to remove inter-reflections using the MFS technique, and how to implementthe inter-reflection removal method. As shown in Fig. 5, the proposed inter-reflection removal method shares the same framework as the adaptive projection method [21] used in highlight removal. Specifically, the proposed method consists of two rounds of projection, the first of which aims to detect inter-reflection regions in the captured patterns by projecting MFS patterns, and regional projection patterns are then generated by adjusting the corresponding projection intensity and then projected to the test surface in the second round of projection. Suppose that the inter-reflection region receives the inter-reflection intensity only from non-inter-reflection regions, as shown in the example in Fig. 3, an inter-reflection can be removed by adjusting (reducing) the projection intensity of these non-inter-reflection regions in the second round of projection. A detailed illustration is as follows.

 

Fig. 5. Procedure of inter-reflection removal.

Download Full Size | PDF

With the selected parameters in Sec. 3.1, MFS patterns are generated and then projected to the tested surface. According to phase-shift algorithm, the modulation of captured MFS patterns is calculated as follows:

Subsequently, the variation in modulation with respect to frequency is obtained by taking the standard deviation of B_t(x,y) with respect to t, the expression of which is

With a proper threshold T_b, regions with a high variation in modulation are detected through a simple image thresholding operation, and a map Map(x,y) is obtained as follows:

As mentioned in Sec. 2.1, the unwrapped phases of the horizontal and vertical fringes establish a one-to-one mapping between the image planes of the camera and projector. Consequently, with the mapping and Map(x,y), the regions in the image plane of the projector corresponding to the non-inter-reflection region are determined as follows:

The retrieval of an unwrapped phase refers to Eqs. (2) and (3). It should be note that, since only high frequencies are used for inter-reflection removal, there is no low frequency fringe pattern to be directly applied to Eq. (3) for unwrapping. Therefore we borrowed the idea from [8] to construct a low-frequency fringe by subtracting one high-frequency pattern from another similar high-frequency pattern. Please refer to [8] for more details.

Assuming that the inter-reflection is only caused by a second-reflection or “emission” from non-inter-reflection regions, as shown in Fig. (3) as an example, the projection intensity of these regions is set to zero and thus an “emission” of the inter-reflection will be avoided. Based on this idea, the regional projection (RP) pattern is generated as follows:

In addition, considering that other un-detected regions are free from inter-reflections, the phases of these un-detected regions are accurate and will be combined with the result after inter-reflection removal to form the final result through the following steps.

Based on the assumption above, in the second round of projection, RP patterns are projected onto the tested surface and captured by the camera. Because the projection intensity corresponding to a non-inter-reflection region is zero in the RP pattern, the inter-reflection will be removed. Then by combining the phase of the non-inter-reflection region (free from inter-reflections) in the first round projection and the one of the inter-reflection region (inter-reflection is removed through an RP projection) in the second round projection, a complete phase map without an inter-reflection will be obtained.

4. Experiments and discussion

4.1 Experiments

Experiments were conducted to validate the proposed method. The measurement system was composed of a camera with 4008×2688 pixels and a projector with 1280×800 pixels, as shown in Fig. 6, and calibrated in advance [22]. Three different scenarios were measured during the experiment, as shown in Fig. 7. The first scenario contained a white board with a strong SIR from a nearby mirror, and the second one included a V-shaped board and a sphere with a glossy surface, where the inter-reflection was mainly caused by a specular reflection from a sphere. The last scenario contained multiple objects with diffuse surfaces, where the SIR was relatively weak and the DIR is dominant.

 

Fig. 6. Measurement system.

Download Full Size | PDF

 

Fig. 7. Three different scenarios.

Download Full Size | PDF

In addition, the results were compared with those of traditional phase shifting (TPS, without inter-reflection removal) and micro-phase shifting (MPS) [6] respectively, where the horizontal and vertical fringe frequencies were f_V = 1/16, 1/80 and 1/800 and f_H= 1/16, 1/128 and 1/1280 in TPS, and f₁=1/14.57, f₂=1/16.09, f₃=1/16.24, f₄=1/16.67 and f₅=1/16.60 in MPS (which was proved to be efficient for inter-reflection removal in [6]). The base frequency f₀=1/16, frequency shift = 1/1280 and frequency shift number T = 9 in the proposed MFS. Please note that the phase-shift number was 12 for all methods above, and the same frequency was used for both the horizontal and vertical fringes in MPS and MFS.

 

Fig. 8. results of the first scenario: (a) is inter-reflection on the white board enclosed by the red line; (b) is the modulation of the captured pattern, where points P₁ and P₂ belong to non-inter-reflection and inter-reflection regions respectively; and (c) is modulations with different frequencies of points P₁ and P₂ shown by the dashed red and blue solid lines respectively.

Download Full Size | PDF

The results of the inter-reflection detection are shown in Fig. 9. Figure 9(a) shows a modulation variation map, where the region enclosed in the red line is the detected inter-reflection region achieved by thresholding this map. Compared with the ground-truth position of the inter-reflection region (as shown in the region in Fig. 8(a)), the detection result accurately covers all inter-reflection regions. Furthermore, the captured RP pattern in the second round of projection is shown in Fig. 9(b), where regions with fringes exactly covered the inter-reflection region and other non-inter-reflection regions are with no fringes, which again demonstrates the detection accuracy.

 

Fig. 9. Detection results: (a) is the detected inter-reflection region in the first round of projection, which is enclosed by the red line and (b) is the captured RP pattern in the second round of projection, where the region surrounded by the red line corresponds to the detected region in (a).

Download Full Size | PDF

For comparison, the inter-reflection separation results of MPS are shown in Fig. 10. For further analysis, the intensity distribution of inter-reflection in the cross-section in Fig. 10(a) is shown in Fig. 10(b). As shown in Fig. 10(b), the intensity clearly changed with frequency owing to the SIR, which violates the basic assumption made on the DIR in that the intensity is constant with the phase-shift and frequency. This result explains why the MPS cannot apply to SIR separation or removal.

 

Fig. 10. Separation results of MPS: (a) inter-reflection intensity separated by MPS and (b) inter-reflection intensity distribution of the cross-section indicated by the blue line in (a) with different frequencies.

Download Full Size | PDF

To further investigate the results of the inter-reflection removal, the 3D measurement of the object was compared, as shown in Fig. 11. As Figs. 11(a) and 11(b) indicate, the measured 3D profile of TPS was distorted significantly by an inter-reflection and a similar result of MPS was found owing to the poor performance regarding SIR removal. As expected, MFS showed promising results and the distortion through the inter-reflection was removed, which verifies the effectiveness of the proposed method.

 

Fig. 11. 3D measurement results after inter-reflection removal: (a)–(c) results of TPS, MPS and MFS respectively.

Download Full Size | PDF

For further comparison, the results in the region enclosed by the red dashed line in Fig. 11 are shown in Fig. 12. The distortion by the inter-reflection was obvious in the results of the TPS and MPS, and was eliminated by the MFS, as shown in Figs. 12(a)–12(c). In addition, similar results can be found in the quantitative comparison of the error in Figs. 12(d)–12(e) and Table 1, which demonstrates the superior performance of MFS regarding SIR removal.

 

Fig. 12. Zoomed-in 3D profile and error distribution after inter-reflection removal: the figures in the first row are measured 3D profiles of regions enclosed by the red dashed line in Fig. 11, where (a)–(c) are the results of TPS, MPS and MFS respectively; the figures in the second row are error distributions after plane fitting, where (d)–(f) are the results of TPS, MPS and MFS, respectively.

Download Full Size | PDF

Table 1. Std. of error by different methods

View Table | View all tables in this article

The results of the second scenario are shown in Fig. 13. As we can see in Fig. 13(a), on the surface of the V-shaped board serious SIR was caused by a specular reflection of the sphere’s glossy surface, which resulted in some “fringe-like” distortions in the modulation of the captured pattern. Furthermore, the SIR decreased with the distance from the sphere, which was barely visible in regions far from the sphere, as shown through point P₁ as an example of non-inter-reflection, and was significant in regions near the sphere, as shown through point P₂ as an example of inter-reflection. The modulations of P₁ and P₂ with different frequencies are shown in Fig. 13(b). Similarly, the variation in modulation with the frequency of P₂ was obviously larger than that of P₁, suggesting that the analysis regarding the inter-reflection detection with a variation in the modulation is valid again.

 

Fig. 13. Results of the second scenario: (a) the modulation of the captured pattern, where points P₁ and P₂ belong to non-inter-reflection and inter-reflection regions respectively and (b) modulations with different frequencies of points P₁ and P₂ in dashed red line and blue solid lines, respectively.

Download Full Size | PDF

The results of inter-reflection detection are shown in Fig. 14. Figure 14(a) shows a modulation variation map, where the region enclosed by the red line is the detected inter-reflection region. As expected, regions near the sphere were detected as inter-reflection regions, whereas regions far from the sphere were not. In fact, the inter-reflection covered all regions of the surface of the board, but we only focused on regions with significant inter-reflection, and thus regions with a weak inter-reflection were not detected, as shown in the regions of the board far from the sphere. However, some non-inter-reflection regions were detected as inter-reflection regions, as shown in the regions circled by the blue solid line in Fig. 14(a). There are two main reasons why this occurred. First, some regions near the boundary of the sphere reflected the fringes on the board, where the fringes directly projected onto these regions and the reflected ones were superposed to form an effect similar to an inter-reflection, and thus these regions were detected. However, since the intensity of inter-reflection was far larger than that of the direct projection and the phase error was also large, the proposed method failed. Nevertheless, there are few methods capable of overcoming this type of inter-reflection. Second, some regions near the central region on the sphere surface were influenced by serious highlights, which caused a “blooming” effect of the CCD, and the intensity of the directly projected fringe and the one of blooming effect were superposed, which also formed an effect similar to inter-reflection and was thus detected. This case belongs to the scope of highlight removal, which can be solved through the method in [21].

 

Fig. 14. Detection results: (a) the detected inter-reflection region in the first round of projection, which is enclosed by the red line and (b) the captured RP pattern in the second round of projection, where the region surrounded by red line corresponds to the detected region in (a).

Download Full Size | PDF

The captured RP pattern in the second round of projection is shown in Fig. 14(b), and regions with fringes covered exactly the inter-reflection region as well as the other non-inter-reflection regions with no fringes, which again validates the detection accuracy again.

The results of the inter-reflection removal are compared. As shown in Fig. 15(a), some obvious “fringe-like” distortions are found in the results of TPS. Similar results of MPS are shown in Fig. 15(b), where the effect of the inter-reflection is still significant even after inter-reflection removal. This is because MPS is limited to DIR removal, and is incapable of SIR removal. In contrast, the proposed MFS outperforms both of TPS and MPS, and achieves results with high quality. Quantitative comparisons are shown in Figs. 16 and 17 and Table 2, which further validate the effectiveness of the MFS.

 

Fig. 15. 3D measurement results after inter-reflection removal: (a)–(c) results of TPS, MPS and MFS, respectively.

Download Full Size | PDF

 

Fig. 16. Zoomed 3D profile and error distribution of W₁: (a)–(c) measured 3D profiles and (d)–(f) are error distributions after plane fitting of TPS, MPS and MFS, respectively.

Download Full Size | PDF

 

Fig. 17. Zoomed 3D profile and error distribution of W₂: (a)–(c) measured 3D profiles and (d)–(f) error distributions after plane fitting of TPS, MPS and MFS, respectively.

Download Full Size | PDF

Table 2. Std. of error by different methods

View Table | View all tables in this article

To further investigate the generalization of MFS, a more natural scenario containing multiple objects was applied, the result of which are shown in Fig. 18. Inter-reflection was caused by a V-shaped surface between two neighbor objects. In addition, as most surfaces of the objects were diffuse, the SIR was relatively weak and the DIR was dominant. Nevertheless, the SIR still caused a larger variation in modulation in the inter-reflection regions than in the non-inter-reflection regions. Figure 18(b) shows an example of points in both types of regions, where the modulation variation of P₂ is obviously larger than that of P₁. Again the results support the analysis regarding the variation in modulation.

 

Fig. 18. Results of the third scenario: (a) modulation of the captured pattern, where points P₁ and P₂ belong to non-inter-reflection and inter-reflection regions respectively and (b) modulations with different frequencies of points P₁ and P₂ shown in the dashed red and blue solid lines, respectively.

Download Full Size | PDF

The results of inter-reflection detection are shown in Fig. 19. As shown in Fig. 19(a), there were some regions with an obviously large variation in modulation occurred, which were accurately detected as inter-reflection regions, whereas others were non-inter-reflection regions. The detected results match well with the intuitional observation that the inter-reflection occurred between neighbor surfaces. Furthermore, the captured RP pattern in the second round of projection is shown in Fig. 19(b), and regions with projected fringes covered exactly the detected inter-reflection region, which again demonstrates the high detection accuracy.

 

Fig. 19. Detection results: (a) the detected inter-reflection region in the first round of projection, which is enclosed by the red line and (b) the captured RP pattern in the second round of projection, where the region surrounded by red line corresponds to the detected region in (a).

Download Full Size | PDF

Figures 20 and 21 show the results of inter-reflection removal. Because the SIR was weak, there are no visually significant differences between the results of the applied methods. However, it is still evident that large error was caused by an inter-reflection in the results of the TPS and MPS, which validates the inter-reflection removal performance of MFS. Please note that the error refers to the differences between the results of TPS/MPS and MFS, where the results of MFS were treated as the ground truth (although it is less rigorous to call it the ground truth, it is true that the precision of the results of MFS is higher than that of other two

 

Fig. 20. 3D measurement results after inter-reflection removal: (a)–(c) results of TPS, MPS and MFS respectively.

Download Full Size | PDF

 

Fig. 21. Zoomed-in 3D profile and error distribution after inter-reflection removal: the figures in the first row are measured 3D profiles of regions enclosed by the red dashed line in Fig. 20, where (a)–(c) are the results of TPS, MPS and MFS respectively; the figures in the second row are error distributions, where (d)–(e) are the results of TPS and MPS respectively.

Download Full Size | PDF

methods).

4.2 Discussions

As demonstrated in the experiments above, the effectiveness of the proposed method was validated which showed a promising performance in the removal of diffuse and specular inter-reflections. Nevertheless, to provide more detailed suggestions in real applications, some further discussion and considerations are presented in this section.

An example of the fringe patterns in the simulations is shown in Fig. 22. According to Eqs. (1) and (5), there were two components in the generated fringe patterns (see Fig. 22(c)): As shown in Fig. 22(a), the direct reflection component (DC) has a uniform phase distribution, i.e. 2πfx, and the average intensity A(x,y) and modulation B(x,y) of the fringes were both set to 80; as shown in Fig. 22(b), the inter-reflection component is a linear function of DC but with a different phase distribution, according to Eq. (7), where the attenuation factor α is set to different values (α=0.05, 0.4 and 0.6, from low to high) to simulate the different levels of inter-reflections. The frequency-shifting parameters for both of the two components are set the same as those in the experiment: the base frequency f₀=1/16, frequency-shift Δf = 1/1280, frequency shift number T = 9 and the phase-shift number N = 12. In addition, different levels of noise with a normal distribution are added to the fringe patterns, where the standard deviation σ was set to 0, 10 and 20 successively and the average is set to 0. The simulation results are shown in Fig. 23.

 

Fig. 22. Fringe patterns in the simulation: (a) and (b) are the direct reflection component and inter-reflection component respectively, and (c) is the fringe pattern with inter-reflection, where the inter-reflection region is enclosed by the yellow dotted line.

Download Full Size | PDF

 

Fig. 23. Inter-reflection detection results: the figures in the first to three columns are the results in the presents of different levels of noises respectively, whereas the ones in the first to three rows are the results with different levels of inter-reflections. The regions enclosed by the red solid lines are the detection results.

Download Full Size | PDF

As shown in Fig. 23, it is obvious that both of the level of the inter-reflection α and the standard deviation σ of the noise have a significant influence on the performance of the proposed method. More specifically, the performance degenerated when σ increased and the level of the inter-reflection decreased. As for the result with σ=10 and α=0.05, as shown in the figure in the last row and second column in Fig. 23, some obvious “spurs” by the noise were introduced and the proposed method failed when σ=20 and α=0.05, as shown in the last row and last column in Fig. 23. Nevertheless, this case only occurred when the inter-reflection was sufficiently weak under significant noise. Furthermore, in most cases, despite the presence of noise, the proposed method performed well, as shown in the first and second rows in Fig. 23, which demonstrates the robustness of the proposed method to noise.

First, state-of-the-art methods, e.g. high frequency pattern projection methods[] were developed for the simultaneous removal of all global illuminations including inter-reflections, subsurface scattering and volumetric scattering, whereas the proposed method was focused on only on the removal of inter-reflections and therefore cannot be applied to the removal of other global illuminations. Nevertheless, the proposed method can remove inter-reflections caused by specular and diffuse reflections, whereas most of high frequency pattern projection methods can only remove diffuse inter-reflections.

Second, state-of-the-art methods, e.g. Qi’s method [16], detect inter-reflections without additional projection of fringe patterns whereas an extra sequence of patterns is used in the proposed method, which results in a good performance regarding inter-reflection detection but increased computational cost. For instance, a typical N-step (N = 12) sequence of phase-shifting patterns with N_f=3 frequencies (for phase unwrapping) is projected using Qi’s method, which achieves a satisfying result regarding severe inter-reflection detection. However, with frequency-shifting in the proposed method, around 70 extra patterns (with frequency-shift number T = 9) were applied to achieve high performance and consequently resulted in a relatively high computational cost. Even so, the proposed method has high sensitivity to inter-reflections and can detect much weaker inter-reflections than Qi’s method. A comparison result between the proposed method and Qi’s method is shown in Fig. 24. As Figs. 24(a)-25(c) indicates, the variation caused by inter-reflections in the epipolar distance maps (a measure used to detect inter-reflections in Qi’s method) is barely visible for the naked eye, and only regions with large errors, e.g. are wrongly detected as inter-reflection regions. However, obvious modulation variations were introduced through inter-reflections and an accurate detection result was therefore achieved, as shown in Figs. 24(d)–25(f).

 

Fig. 24. Inter-reflection detection results where the detections regions are enclosed by the red solid lines: (a)–(c) epipolar distance maps using Qi’s method where the intensity represents the possibility that they are inter-reflection regions, and (d)–(f) modulation variations with frequency.

Download Full Size | PDF

In addition, to reduce the computational cost, there are several solutions available, e.g. high-rate projection techniques and methods reducing the frequency-shifting number. Combining the projector defocusing technique [23], which can reach a 10-kHz-rate projection, the time cost on extra pattern projections can be reduced to a negligible level (less than 0.01 seconds for N = 12, T = 9 and N_f=3). On the contrary, for a further reduction of the computational cost, in a future study, we may further investigate the relation between the frequency-shifting number and the inter-reflection detection performance, hoping to find an optimal solution of a relatively small frequency-shifting number without compromising on performance.

Third, for the extreme case of mirror-like surface, e.g. the specular sphere surface in Fig. 14(a), where the inter-reflection component may be dominant whereas the direct reflection component is negligible, the proposed method may treat the former as the useful signal and the latter as noise, which results in an inaccurate detection of the inter-reflection. Furthermore, a highlight on a mirror-like surface will also affect the detection and lead to a misdetection. For the first problem posed by the dominant inter-reflection component, to the best of our knowledge, there are no methods available that achieve satisfying results, and we will leave this to a future study. To address the problem of highlight, we may combine the highlight removal methods [21] with the proposed method, where the highlight is first removed using a highlight removal method and an the inter-reflection is then detected and removed through the proposed method.

5. Conclusion

In this paper, the micro-frequency shifting (MFS) method for inter-reflection removal was presented. Based on an observation that the modulation variation with frequency in the inter-reflection regions is larger than that in other regions, we propose using the MFS technique in which patterns with a specifically designed frequency-shift and a base frequency are projected. By computing the variation in modulation with respect to frequency, the inter-reflection region is accurately detected. Using the detection result, inter-reflections are removed using a regional-projection technique. Since no assumptions are made regarding the reflection property of the tested surface, the proposed method is capable of all kinds of inter-reflection caused by specular and diffuse reflections. Experiment results demonstrated that the proposed MFS shows a more promising performance on both diffuse and specular inter-reflection removal than state-of-the-art methods.