Open Access
Add to BibSonomy
Abstract
For three-dimensional (3D) measurement of object surface and shape by pattern projection systems, we used a hybrid projection system, i.e., a combination of a projection lens and a transmissive freeform to generate an aperiodic sinusoidal fringe pattern. Such a freeform effects a light redistribution, thus leading to an effective and low-loss pattern projection, as it increases the total transmission intensity of the system and has less power dissipation than classical projection systems. In this paper, we present the conception and realization of the measurement setup of a transmissive fringe projection system. We compare the characteristics of the generated intensity distribution with the classical system based on GOBO (GOes Before Optics) projection and show measurement results of different surface shapes, recorded with the new system.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
3D metrology is increasingly applied in many different fields, such as industrial quality control, medicine, or rapid prototyping. Fast data acquisition and processing are gaining importance for 3D measurements in inline processes. Stereo camera systems are particularly suitable for the measurement of highly dynamic processes, as they allow the simultaneous, contactless determination of a large number of surface points. In combination with the active projection of one or more patterns onto the target, they typically offer a higher measurement point density and accuracy than other 3D sensors [1,2]. As a result, pattern projection systems are an excellent choice for optical 3D measurements, especially in combination with emerging deep-learning technologies [3].
A typical pattern projection principle is the fringe projection. The key element of fringe projection systems is the projection unit with a light modulator, which generates the fringe pattern. Depending on the speed [4] and wavelength of light modulation, different types of light modulators can be used. Examples are Ronchi gratings [5], microscopic fringe projection profilometry [6], digital modulators using microdisplay technologies like liquid crystal displays (LCD) [7,8], liquid crystal on silicon displays (LCoS) [9,10] or digital micromirror devices (DMD) [11–13] up to OLED-microdisplay [14]. Another way to create such patterns can be realized by GOBO projectors [15]. A mask is illuminated by a radiation source and then imaged into the measurement plane. Former experiments with only one tailored free-form mirror have also shown the possibility to project fringe patterns in the ultraviolet spectral range [16]. The central element of the system is a purpose-built refractive free-form mirror that generates the fringe pattern into the measurement plane.
A freeform-based fringe pattern is generated by a redistribution of light comparable to classical beam-shaping, which differs from the imaging technique of classical projectors. Refractive elements with a freeform surface can project a pattern in an energy-preserving manner. This beam profile shaping is advantageous for the projection with lower losses in light intensity and fast, continuous 3D measurements. Therefore, our goal was to replace the GOBO wheel by a freeform wheel or a cyclically, laterally moving freeform element. The calculation of the freeform depends on the light source (beam data set) and the desired pattern distribution. However, this method has the disadvantage of reducing the pattern contrast at large working distances as it uses an extended light source (not an ideal point light source), because all real light sources have a given extension. For this reason, a hybrid system has been designed which combines pattern generation by a transmissive freeform element with an imaging system to expand the reproduced image. This significantly reduces the washing-out of the pattern because the intermediate image has only a very small working distance. Furthermore, freeform and imaging optics can be adapted more flexibly and independently to different projection needs. For the surface measurement the object is observed by one (classical approach) or two (stereo vision based fringe projection) cameras. The coordinates of surface points can be calculated by means of the geometrical arrangement of the measurement system.
2. Measurement principle
Two principles for spatial light modulation with high speed and high accuracy dominate the market for 3D reconstruction: digital light modulation and GOBO projection. Both principles have the problem that they achieve only 50 % of optical power. This is due to the working principle of modulating light to the desired intensity by reflecting (LCoS, DMD) or blocking (GOBO) light from the optical path. In order to achieve higher efficiency than conventional active dense pattern projection systems, the light from the radiation source must be spatially redistributed. Thus it can use the entire optical energy. For this purpose we use a transmissive freeform element. To generate 3D surface models, we combine a freeform projection system with a camera system in stereo vision arrangement [17].
2.1 Freeform projection system
The surface shape of a freeform element is calculated by means of the input beam (the beam profile of the light source) and the target light distribution (the pattern to be generated). With extended light sources, however, this method reduces the contrast at long working distances. Deconvolution approaches, applied at the simulation level [18,19], could improve the contrast. Figure 1 shows the calculation process of a continuous refractive surface. The freeform surface $z(x,y)$ should be calculated in such a way that the input distribution (intensity $I_S(x,y)$ and direction $s_1$) of the light source is transformed by refraction into the desired output distribution $I_T(x,y)$. The solution for the determination of the freeform surface is derived from the Jacobian equation and the ray-tracing equations for refractive surfaces [18]. Figure 2 shows the blurring effect of three different working distances for an extended light source. By increasing the working distance, a larger measuring field can be achieved. However, because of a larger blurring the image quality is reduced.
Fig. 1. Calculation of a single freeform surface by the given sizes of the input distribution (intensity $I_S(x,y)$ and direction $s_1$) and the desired output distribution $I_T(x,y)$. The input vector field is redirected to the target points $u(x,y)$ at the freeform surface $z(x,y)$ according to the law of refraction/reflection and produces the desired output distribution [18].
Fig. 2. (top) Principle design of the freeform system and (bottom) blurring effect by three different working distances due to usage of an extended light source.
By combining the freeform component with an imaging system, the sharp intermediate plane is imaged into a larger area (see Fig. 3). This hybrid approach [19] solves the problem of blur at long working distances. Additionally to that, it is more flexible and allows easy adaptation of the freeform component and the imaging system to different projection requirements.
Fig. 3. Layout of the hybrid system for freeform beam shaping projection. (left) the (extended) light source, (middle) the transmissive freeform, (right) the imaging system and the object space projection result.
2.2 Camera system in stereo vision arrangement
To achieve high accuracy and correct dense point clouds for measurement purposes (independent of the object surface characteristics), it is useful to project a fringe pattern. 3D measurements based on fringe patterns are performed with the well-known stereo vision-based arrangement and using the triangulation principle: the intersection of two beams with given starting points and the directions of the coordinates of an unknown point. For optical 3D measurement, triangulation can be performed between a projection unit and one camera (classical approach) or between two cameras (active stereo vision). The latter approach is shown in Fig. 4, the camera centers $\textit {C}_1$ and $\textit {C}_2$ forming the triangulation baseline. An unknown object point P is determined by rays which have their origin in the point P and pass through the camera centers [20]. In order to precisely find these corresponding points in the two cameras, a sequence of fringe patterns, i.e., temporally and spatially varying intensity distributions are projected onto the object surface. With the developed projector aperiodic sine patterns are generated. The intensity profile can be described by the following formula [20]:
with spatially varying offset $a_i(x)$, amplitude $b_I(x)$, period length ${2\pi }/{c_i(x)}$, and phase shift $d_i(x)$. A sequence of aperiodic sinusoidal patterns is projected onto the measuring plane. In both cameras the same unique gray value stack at each pixel is found by using a normalized cross correlation function (Eq. (2))Fig. 4. Triangulation principle in two dimensions with the camera centers $\textit {C}_1$ and $\textit {C}_2$ forming the triangulation baseline to determined an unknown object point P [20].
3. Experimental setup of the transmissive freeform fringe projector
The active pattern projection system uses a specific aperiodic sinusoidal fringe pattern [21]. To realize a compact system, the freeform element is not designed as a rotating system but as a laterally shifted component. The requirements for continuous 3D measurements with such a component have been analyzed in a master thesis [22]. Using this mechanism, only a small freeform object has to be fabricated. The freeform is currently available only as a prototype. It has a size of 34 mm by 34 mm (see Fig. 5) and was calculated and manufactured in the ultra-precision machining department of Fraunhofer IOF.
3.1 Investigation of freeform pattern projection
Only a part of the freeform with a size of 20.5 mm by 20.5 mm is illuminated, while the lateral displacement of $\pm$ 5 mm of the freeform ensures the requirements for continuous 3D measurements for a 2D rate of up to 250 Hz. The mechanical setup consists of a cooler, a light source, a motion mechanism for the laterally shifting of the freeform, and a projection lens (Fig. 6).
Fig. 6. (left) Side view of the mechanical design of the freeform-based projector and (right) realization of a compact prototype.
The detailed motion mechanism is shown in Fig. 7. A double camshaft attached to the motor is rotated with an angular velocity of $\omega _m$, thus generating a force $F_m$. This leads to a lateral displacement $v_m$ of the freeform ($-$ mm < $v_m$ < 5 mm). The shape of the double camshaft determines the maximum displacement of the freeform, while the rotation speed defines the rate of pattern changing. The backward movement of the freeform is driven by the force $F_s$ of two springs.
An Osram SFH 4715AS-LED with a wavelength of 850 nm and at a typical radiation intensity of $900\,\frac {\text {mW}}{\text {sr}}$ [23] was used as light source. To generate a pattern with low blur, the incident wavefront on the freeform must be as plane as possible. The collimator, an aspherical lens specially designed for this purpose, produces the desired intensity distribution over the entire field. As this lens requires a very strong curvature, the manufacturing costs are very high. An available stock lens (Thorlabs ALU5040U [24]) with a diameter of 50 mm and a focal length of 40 mm is used for this prototype as collimator lens additionally to the freeform. Figure 8 shows the different intensity of light distribution of the lenses in front and behind the freeform. Using the stock lens reduces the total incident intensity. The resulting intermediate image, 3.4 mm behind the freeform (Fig. 8(b)), has a modulation loss of about 12.5 %. This lower modulation is accepted to avoid the 20 times higher costs of the designed lens. To project the intermediate image into the measurement field, a Linos Inspec.x M 1.4 / 50 NIR lens is used as the projection lens.
Fig. 7. Front view of the motion mechanism for the lateral displacement of $v_m$ by rotating the double camshaft with an angular velocity of $\omega _m$.
Fig. 8. Intensity distribution of designed lens (blue) and stock lens (orange) at (a) entering intensity $I_S(x,z=z_0)$ onto the freeform, (b) generated pattern intensity $I_T(x,z=z_T)$ in the intermediate plane, 3.4 mm behind the freeform (see Fig. 1)).
Figure 9 shows the comparison between simulated (red) and experimental (blue) evaluation of the resulting pattern intensity distribution with the hybrid freeform pattern projection system. The lower graph presents the gray value of each pixel in the measurement field. The measured pattern distribution of the experiment corresponds with the previously simulated distribution. Due to a limited selection of rays and tolerance in manufacturing, differences in intensity occurred.
Fig. 9. (top) Exemplary section of 2D images from simulation and experiment. (bottom) profile line of pattern contrast generated by the hybrid transmissive freeform setup between simulated (red) and experimental (blue) evaluation.
In order to evaluate the advantages of the transmissive freeform optics, the resulting optical power of the freeform is compared with the optical power generated by a GOBO slide (blocking metal mask) using similar numbers of line pairs under measurement conditions. To determine the optical power of both projection approaches, three images of the measurement field are captured with a camera. The first image is without any pattern, the second is with the GOBO slide, and the third image is with the freeform. Afterwards the sum of the gray values of each image is calculated and compared with each other. This results in an optical power of only 57.4 % of the total optical power for the GOBO slide, due the absorbed structure. In contrast the transmissive freeform optics reach a value of 99.87 % of the total optical power. By using such a freeform optics, the overall performance can be significantly increased. For 3D reconstruction the modulation (the difference between dark and light fringes) is crucial, so that the effectiveness of the freeform pattern projection decreases. Figure 10 presents the modulation of adjacent maxima and minima for a total of 14 line pairs. It shows that the use of the transparent freeform element for pattern generation results in an effectively 19 % higher modulation than of the modulation of the blocking metal mask. However, unlike the GOBO metal mask, which absorbs light almost completely in dark areas, the freeform cannot redistribute light perfectly from dark to light areas. This negative effect is intensified by the use of the extended light source.
Fig. 10. Differences between adjacent intensity maxima and minima generated by the hybrid transmissive freeform setup (approximately 36.4 intensity, blue) with the conventional (metal mask) GOBO setup (approximately 29.4 intensity, orange) at 5 ms image exposure time.
3.2 3D results
In order to three-dimensionally measure the surface shape of objects, we combined our freeform pattern projection system with two Blackfly cameras (BFS-U3-20S4M-C) in stereo vision arrangement. The cameras are placed at a distance of 130 mm from each other (see baseline indicated in Fig. 11) and used at 250 frames per second. The working distance of 500 mm results in a measurement field diameter of 250 mm.
Fig. 11. Realized prototype of transmissive fringe projection system between two cameras in stereo arrangement.
In order to characterize the measurement quality of our freeform projection system, we captured the measurement scene shown in Fig. 12 with 10 different patterns at a frame rate of 250 Hz. The 3D result of a frustum of a pyramid and two calibrated spheres is shown on the right in Fig. 12. The distance between the center points of the fitted spheres ("ball 1" and "ball 2") is 221 µm smaller than the specified distance with a value of about 199.3 mm, corresponding to a deviation of 0.11 %. The 3D standard deviation of the points on the flat calibration body to a fitted plane ("plane 1") is approximately 102 µm.
Fig. 12. (left) Measurement scene of the freeform-based 3D scanner prototype, (right) 3D surface measurement examples of a frustum of a pyramid and two spheres, with color-coded 3D deviations.
Figure 13 shows an exemplary dynamic measurement of facial movements. By using 10 different patterns to reconstruct one 3D point cloud, we achieved a 3D rate of 25 Hz, corresponding to a temporal resolution of 40 ms. Within the total measurement time of 240 ms, the test person opens his mouth and his eyes. We were able to carry out this fast dynamic measurement using our compact, low-loss pattern projection system without increasing the optical power of the light source.
Fig. 13. (top) Camera images of the freeform projected aperiodic sinusoidal fringes, which are recorded with a resolution of 808 $\times$ 620 px at a frame rate of 250 Hz and (bottom) reconstructed point clouds captured with 10 different patterns.
4. Conclusion
This article presents a way to increase the speed of continuous 3D measurements by improving the efficiency of pattern projection for high-precision 3D metrology. Our approach is based on applying a transmissive freeform, prototypically implemented in a hardware demonstrator to prove its real advantages. The transmissive freeform-based pattern generation can significantly increase the optical power by 27.3% compared to classical projection systems with absorbing structures. This is especially important for dense, high-precision 3D reconstruction applications with limited performance of commercially available light sources for compact sensor designs, e.g., in the long infrared wavelength spectrum for measuring transparent objects. The freeform helps to avoid pattern mixing and to compensate the inevitable reduction in exposure time if more than one sensor is used to simultaneously capture different areas of the same object. A hybrid approach, as a combination of the freeform component with an imaging system, allows the view of larger measurement fields. Additionally to that, it is more flexible and allows easy adaptation of the freeform component and the imaging system to different projection requirements. Compared with classical projectors the experiment shows an improvement in pattern modulation by 19 %. Further improvements can be reached by applying an adjusted aspherical lens or refined freeform design methods, e.g., deconvolution [18,19]. For practical use, the manufacturing process must be further developed into a sustainable precision glass molding process. By manufacturing two freeform surfaces on one lens (front and back), the projection lens can be replaced, allowing to create a much more compact system.
Funding
Thüringer Aufbaubank (2016 FGR 0044).
Acknowledgments
This project was supported by the Free State of Thuringia from the European Social Fond (ESF) under project number 2016 FGR 0044 (research group "DIADEM").
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. F. Chen, G. M. Brown, and M. Song, “Overview of 3-D shape measurement using optical methods,” Opt. Eng. 39(1), 10–22 (2000). [CrossRef]
2. S. Zhang and P. S. Huang, “High-resolution, real-time three-dimensional shape measurement,” Opt. Eng. 45(12), 1–8 (2006). [CrossRef]
3. C. Zuo, J. Qian, S. Feng, W. Yin, Y. Li, P. Fan, J. Han, K. Qian, and Q. Chen, “Deep learning in optical metrology: a review,” Light: Sci. Appl. 11(1), 39 (2022). [CrossRef]
4. S. Zhang, “High-speed 3D imaging with digital fringe projection echniques,” in Tribute to James C. Wyant: The Extraordinaire in Optical Metrology and Optics Education, vol. 11813V. N. Mahajan and D. Kim, eds., International Society for Optics and Photonics (SPIE, 2021), pp. 226–233.
5. K. Leonhardt, U. Droste, and H. J. Tiziani, “Microshape and rough-surface analysis by fringe projection,” Appl. Opt. 33(31), 7477–7488 (1994). [CrossRef]
6. Y. Hu, Q. Chen, S. Feng, and C. Zuo, “Microscopic fringe projection profilometry: A review,” Opt. Lasers Eng. 135, 106192 (2020). [CrossRef]
7. H. Nguyen, N. Dunne, H. Li, Y. Wang, and Z. Wang, “Real-time 3D shape measurement using 3LCD projection and deep machine learning,” Appl. Opt. 58(26), 7100–7109 (2019). [CrossRef]
8. T. Bothe, W. Osten, A. Gesierich, and W. Jüptner, “Compact 3D camera,” in Interferometry XI: Applications, vol. 4778W. Osten, ed., International Society for Optics and Photonics (SPIE, 2002), pp. 48–59.
9. P. Kühmstedt, G. Notni, J. Hintersehr, and J. Gerber, “>CAD-CAM-System for dental purpose - an industrial application,” Proc. of Fringe’01, (Ed. W. Jüptner) pp. 667–672 (2001).
10. G. Lazarev, S. Bonifer, P. Engel, D. Höhne, and G. Notni, “High-resolution LCOS microdisplay with sub-kHz frame rate for high performance, high precision 3D sensor,” in Digital Optical Technologies 2017, vol. 10335B. C. Kress and P. Schelkens, eds., International Society for Optics and Photonics (SPIE, 2017), pp. 292–297.
11. G. Notni, W. Schreiber, M. Heinze, and G. H. Notni, “Flexible autocalibrating full-body 3D measurement system using digital light projection,” in Optical Measurement Systems for Industrial Inspection, vol. 3824M. Kujawinska and W. Osten, eds., International Society for Optics and Photonics (SPIE, 1999), pp. 79–88.
12. G. Frankowski, M. Chen, and T. Huth, “Real-time 3D shape measurement with digital stripe projection by Texas Instruments Micro Mirror Devices DMD,” in Three-Dimensional Image Capture and Applications III, vol. 3958B. D. Corner and J. H. Nurre, eds., International Society for Optics and Photonics (SPIE, 2000), pp. 90–105.
13. P. S. Huang, C. Zhang, and F.-P. Chiang, “Digital fringe projection technique for high-speed 3D shape measurement,” in Process Control and Inspection for Industry, vol. 4222S. Zhang and W. Gao, eds., International Society for Optics and Photonics (SPIE, 2000), pp. 54–60.
14. G. Notni, S. Riehemann, P. Kuehmstedt, L. Heidler, and N. Wolf, “OLED microdisplays: a new key element for fringe projection setups,” Interferometry XII: Applications, vol. 5532W. Osten and E. Novak, eds., International Society for Optics and Photonics (SPIE, 2004), pp. 170–177.
15. S. Heist, P. Lutzke, I. Schmidt, P. Dietrich, P. Kühmstedt, A. Tünnermann, and G. Notni, “High-speed three-dimensional shape measurement using GOBO projection,” Opt. Lasers Eng. 87, 90–96 (2016). [CrossRef]
16. S. Zwick, S. Heist, Y. Franzl, R. Steinkopf, P. Kühmstedt, and G. Notni, “Wide-band phase-shifting fringe projector on the basis of a tailored free-form mirror,” Opt. Eng. 52(2), 023001 (2013). [CrossRef]
17. C. Munkelt, H. Speck, C. Bösel, C. Junger, S. Töpfer, and G. Notni, “Continuous low-latency 3D measurements using efficient freeform GOBO pattern projection and close-to-sensor image rectification,” Dimensional Optical Metrology and Inspection for Practical Applications IX, vol. 11397K. G. Harding and S. Zhang, eds., International Society for Optics and Photonics (SPIE, 2020), pp. 31–38.
18. C. Bösel and H. Gross, “Single freeform surface design for prescribed input wavefront and target irradiance,” J. Opt. Soc. Am. A 34(9), 1490–1499 (2017). [CrossRef]
19. C. Bösel and H. Gross, “Compact freeform illumination system design for pattern generation with extended light sources,” Appl. Opt. 58(10), 2713–2724 (2019). [CrossRef]
20. S. Heist, A. Mann, P. Kühmstedt, P. Schreiber, and G. Notni, “Array projection of aperiodic sinusoidal fringes for high-speed three-dimensional shape measurement,” Opt. Eng. 53(11), 112208 (2014). [CrossRef]
21. S. Heist, P. Kühmstedt, A. Tünnermann, and G. Notni, “Theoretical considerations on aperiodic sinusoidal fringes in comparison to phase-shifted sinusoidal fringes for high-speed three-dimensional shape measurement,” Appl. Opt. 54(35), 10541–10551 (2015). [CrossRef]
22. S. Töpfer, “Aufbau und Optimierung eines mechanischen Musterprojektionsprinzips für die 3-dimensionale Messung basierend auf linearer Mustertranslation,” master’s thesis, Fraunhofer Institute for Applied Optics and Precision Engineering IOF, Albert-Einstein-Strasse 7, 07745 Jena, Germany (2019). Unpublished master’s thesis.
23. Osram, “Datasheet SFH4715AS,” https://www.osram.com/ecat/OSLON%C2%AE%20Black%20SFH%204715AS/com/en/class_pim_web_catalog_103489/global/prd_pim_device_2219819, (02. July 2019, accessed 12.07.2022).
24. Thorlabs, “Datasheet aspherical lens ALU5040U,” https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=3835, (accessed 12.07.2022).
