Depth from defocus measurement method based on liquid crystal lens

Mao Ye; Xiaoxi Chen; Qichang Li; Jun Zeng; Shuda Yu

doi:10.1364/OE.26.028413

1. Introduction

With the development of computer vision and image sensor technology, more and more methods have been developed for depth measurement which is helpful for three-dimensional (3D) reconstruction of scene. As an active depth measurement, time-of-flight (TOF) [1–3] measures the phase difference between emitted and reflected infrared waves to estimate the depth from each sensor pixel to objects. It has been applied to many fields such as robot navigation [4,5], gesture recognition, etc. Another active method is structured light technique [6–8]. A modulated light is projected on the object, based on the decode analysis of the reflected image, the scene depth can be obtained accurately. The technique is mainly used in games, 3D scanning, and in particular, in industrial automation [9,10]. Both methods are sensitive to environment light, and are not suitable for outdoor applications. The passive depth measurement, on the other hand, use the natural light images acquired with imaging systems, and use the image characteristics, such as disparity, ambiguity, etc., to derive the scene depth. Binocular stereo vision acquires depth via position difference in images [11–13]. In binocular imaging method, however, there includes camera calibration procedure and the image match algorithm are complicated, so that long execution time is necessary to get dense depth information. Depth from focus (DFF) method reconstructs a depth map from a sequence of differently focused images [14]. Depth from defocus (DFD) method, on the other hand, obtains depth from two images basing on blur radius [15,16].

Traditional DFD method derives depth from different defocused images taken by the imaging system with different lens position, which resulting in change in image magnification that has to be compensated in calculation. Furthermore, the change of lens position needs accurate mechanical moving equipment, and the long term friction loss leads to a short lifecycle of the system. The problem of magnification change can be avoided using a telecentric optical system [17]. However, a telecentric optics is of large size and heavy weight.

Recently, researches in depth measuring using lenses with electrically controllable focal lengths have been reported [18–20]. It becomes unnecessary for any mechanical lens movements in these approaches. It shows that depth can be measured via DFF using an LC lens [18]. However, one has to take a stack of images for the calculation, which is a time-consuming process. Generally, DFD approaches needs only two images for the calculation and required much short processing time. Recently, a depth measuring method via DFD using a liquid lens has been reported [19,20]. The system projects a Ronchi fringes on the object. The contrast of the fringes on the object varied with the change of the focal length of the liquid lens. By comparing the contrast curve with the calibration result, a high precision depth map is obtained. For there are a light source, an LCD, a beam splitter, etc., the system is rather complicated.

In this paper, we propose a DFD measuring method using an LC lens [21] to recover the scene depth. Only two images are acquired and analyzed, and the system is simple and compact. For there are no lens movements involved in, the system is of long lifecycle and lower power consumption. The principle is discussed, and the proposal is successfully demonstrated experimentally.

2. DFD optic model

2.1 LC lens

The structure of the LC lens in this work is shown in Fig. 1 [21,23]. Indium Tin Oxide (ITO) transparent electrode are coated on the inside of glass substrate 1 and 2, respectively. The upper patterned electrode is composed of two parts: one includes a hole of 2.00 mm diameter applied with square-wave voltage V₁, and the other is a circle of 1.98 mm diameter with square-wave voltage V₂. The resistive film is made from ZnO, and its surface resistivity is in the order of 10⁸ Ω/□. The LC (MLC 6080 from Merck) layer is of thickness of 30 μm. The electric field resulted from V₁ and V₂, rotate the LC director an angle of $θ$ , and the effective refractive index experienced by the e-wave is [22]:

Fig. 1 Structure of LC lens.

Download Full Size | PDF

n_{e} (θ) = n_{e} n_{o} / \sqrt{n_{e}^{2} \sin^{2} θ + n_{o}^{2} \cos^{2} θ} .

The properties of the LC lens are measured using a Mach-Zehnder interferometer with a laser beam of 532 nm wavelength. The aberrations and the optical powers of the LC lens are obtained by analyzing the interference fringes. The results are shown in Table 1. The optical power can be tuned from negative to positive lens states, and the rms aberration is kept below 0.07 wave. The frequency of both voltages is 800 Hz. In positive lens state, V₁ is kept to be a constant of 3.5 V_rms and V₂ tunes the optical power from 0.0 to 4.4 m⁻¹, and in negative lens state, V₂ is kept a constant of 2.2 V_rms and V₁ tunes the power from 0.0 to −3.9 m⁻¹. The power range of the LC lens is $[P_{max}^{-}, P_{max}^{+}]$ = [-3.9 m⁻¹, 4.4 m⁻¹], and the rms aberration remains below 0.07 wave, which is usually considered to be a tolerable value for an optical lens [22]. $P_{max}^{-}$ and $P_{max}^{+}$ are, respectively, the maximum powers of the LC lens working in negative and positive lens states.

Table 1. Optical power and rms aberration.

View Table | View all tables in this article

The LC lens is placed in front of a lens module (the main lens), and its aperture is used as the aperture stop of the compound lens. The magnification of the imaging system for one object point is then a constant even the optical power of the LC lens is changed.

2.2 DFD optic model

Suppose the focal length of the main lens is f_g, the optical power is P_g = 1/f_g. In this study, the aperture of the main lens is 2.0 mm and f_g = 8.0 mm. The power of the compound lens is then P = P_g + P_LC-dP_gP_LC. So, the power range of the compound lens is $[P_{m i n}, P_{m a x}]$ = $[P_{g} + P_{max}^{-} - d P_{g} P_{max}^{-},$ $P_{g} + P_{max}^{+} - d P_{g} P_{max}^{+}$ ].

We define ∇P as:

P_{g} + P_{max}^{+} - d P_{g} P_{max}^{+} .

Let the image on the image sensor of one object O of object distance u is I when the optical power of the system is P, we introduce a parameter ∂∈(−1,1) to describe P, that is:

P = P_{m a x} - (1 - \partial) \nabla P = P_{m i n} + (1 + \partial) \nabla P .

P = (P_max + P_min)/2 when ∂ = 0, P = P_max when ∂ = + 1, and P = P_min when ∂ = −1. The images I, I₁, and I₂ of the object O correspond, respectively, the optical power of P, P_max, and P_min, as shown in Fig. 2, and:

Fig. 2 DFD optic model of imaging system with LC lens.

Download Full Size | PDF

\begin{array}{l} \frac{1}{u} + \frac{1}{v_{0}} = P \\ \frac{1}{u} + \frac{1}{v_{1}} = P_{max} \\ \frac{1}{u} + \frac{1}{v_{2}} = P_{min} \end{array}

Watanabe, et al [16] proposed an effective method to compute the depth from the two images using the telecentric optics imaging system taken at two sensor positions. Here, adopt a similar optic model.

Let’s the radius of the aperture of the system is a. In the experiment, firstly the object of distance u₀ is adjusted to be exactly focused when the LC lens is in non-lens state, that is, P_LC = 0. Then the image of distance v₀ = f_gu₀/(u₀-f_g). When the object O is imaged on plane v₁, from Eqs. (2)-(4), the radius of the defocused spot on the sensor plane is:

R_{1} = a v_{0} (1 - \partial) \nabla P = \frac{a f_{g} u_{0}}{u_{0} - f_{g}} (1 - \partial) \nabla P .

Similarly, when O is imaged on the plane v₂, the radius of the defocused spots on the sensor plane is:

R_{2} = a v_{0} (1 + \partial) \nabla P = \frac{a f_{g} u_{0}}{u_{0} - f_{g}} (1 + \partial) \nabla P .

For the blur function, we adopt the pillbox ambiguity function

h (x, y) = \frac{1}{π R^{2}} \prod^{​} \frac{1}{2 R} \sqrt{x^{2} + y^{2}}

The ∏(r) is a rectangular window function. If r < 0.5, then ∏(r) = 1, else ∏(r) = 0.

The Fourier transform of the pillbox is

H (p, q) = \frac{1}{π R \sqrt{p^{2} + q^{2}}} j_{1} (2 π R \sqrt{p^{2} + q^{2}}),

where (p, q) represent spatial frequency.

The defocus functions of the object O when the system in the maximum power and minimum power states is as following equation, respectively,

h_{1} (x, y) = h_{1} (x, y; (1 - \partial) \nabla P, a f_{g}) = \frac{1}{π {[a f_{g} (1 - \partial) \nabla P]}^{2}} \prod^{​} (\frac{1}{2 a f_{g (1 - \partial) \nabla P}} \sqrt{x^{2} + y^{2}})

h_{2} (x, y) = h_{2} (x, y; (1 + \partial) \nabla P, a f_{g}) = \frac{1}{π {[a f_{g} (1 + \partial) \nabla P]}^{2}} \prod^{​} (\frac{1}{2 a f_{g (1 + \partial) \nabla P}} \sqrt{x^{2} + y^{2}} .

According to Watanabe, et al, the normalized ratio:

\frac{M}{P} = \frac{M (p, q)}{P (p, q)} = \frac{H_{2} (u, v, (1 + \partial) \nabla P, a f_{g}) - H_{1} (u, v, (1 - \partial) \nabla P, a f_{g})}{H_{2} (u, v, (1 + \partial) \nabla P, a f_{g}) + H_{1} (u, v, (1 - \partial) \nabla P, a f_{g})},

Where H₁(p, q, (1-∂)∇P, af_g) and H₂(p, q, (1 + ∂)∇P, af_g) are the Fourier transform of Eq. (9) and (10) respectively. It can be seen that M/P is a monotonic function of α for −1≤∂≤1, provided the radial frequency

f_{r} = \sqrt{p^{2} + q^{2}}

is not too large. We will prove how larger radial frequency is feasible. When the focal power of imaging system changed from P_min to P_max, as a rule of thumb, this frequency range equals the width of the main lobe of the defocus function H when it is maximally defocused. Obviously, the ambiguity of I₁ is larger than I₂. If the first zero crossing [14] of H₂ is

f_{2_{0}}

, we have

1.22 λ = 1.22 / f_{2_{0}} = 2 R_{(1 + \partial)} = 2 (1 + \partial) \nabla P a f_{g} u_{0} / (u_{0} - f_{g})

. Therefore, space frequency of the image I₂ is:

f_{r} \leq f_{2_{0}} = \frac{1.22 (u_{0} - f_{g})}{2 a f_{g} u_{0} (1 + 1) \nabla P} .

We can find out when the space frequency f_r≤0.61(u₀-f_g)/(2af_gu₀∇P),the normalized image ratio M/P ratio is monotonic [24].

Efficient DFD computation methods have been proposed in previous works [25–27]. In this paper, we just select an approach based on rational filters [14] which is better for real time computation. Rational filters are broadband linear filters. We can implement them with small convolution kernels, low computational cost and high spatial resolution. In our system, we also adopt the simple model: $\frac{M}{P} = \frac{M (u, v, \partial)}{P (u, v, \partial)} = \frac{G_{P 1} (u, v)}{G_{M 1} (u, v)} β + \frac{G_{P 2} (u, v)}{G_{M 1} (u, v)} β^{3}$ . The theory of the rational filters generation, implementation has been clarified and detailed by Watanabe, et al [16].

In fact, the final design issue pertains to maximum frequency f_rmax. Since the discrete Fourier transform of a kernel of size ks has the minimum discrete frequency period of 1/ks. The maximum frequency shall be above 1/ks. According to Nyquist theorem, we can specify the condition as: f_rmax≥2/ks where f_rmax = 0.73(u₀-f_g)/(2af_gu₀∇P). This condition can be interpreted as follows: The max blur circle diameter 2af_gu₀∇P/(u₀-f_g) must be smaller than 73% of the kernel size ks.

For our imaging system with LC lens, if the radius of the aperture is 2.0 mm, the focal length f_g = 8 mm, the focal power of the LC lens is P∈[-2, 2] m⁻¹, one can easily get the maximum radius of defocused spot is r_max = 32 μm. Provided the pixel size is 2.2 μm, r_max = 14.544 pixels. From this computation we can clear acknowledge how large the rational filter we need and how to setup the experiment.

3. Experimental results

The experimental setup is composed of a camera, a main lens, and an LC lens; the LC lens is stick to the main lens of 8.0 mm focal length. The distance d between the LC lens and the main lens is negligible. The camera is MD50-T from MINGMEI Company with resolution of 2592*1944 (1280*960 is actually used in experiment) and pixel size of 2.2 μm. Firstly, tune the LC lens working in the non-lens status, then adjust the object distance with u₀ correspond to v₀. The image distance keeps unchanged with the adjustment of the optical power.

There are seven objects including an oil painting in the scene placed with different distances. The camera module is first adjusted to focus object at distance u₀ = 250 mm, and two images are taken with the optical powers of 4.4 and −3.9 m⁻¹ of the LC lens, as shown in Figs. 3(a) and 3(b), respectively. The depth map, as shown in Fig. 4, is derived from the two images assuming the depth value be 128 when the object distance is u₀ = 250 mm. The depth values are summarized in Table 2. Figure 5 shows the linearly relationship between the depth value and the reciprocal of the object distance.

Fig. 3 Images with maximum positive (a) and negative (b) optical power.

Download Full Size | PDF

Fig. 4 Depth map.

Download Full Size | PDF

Table 2. Object distance and depth value.

View Table | View all tables in this article

Fig. 5 Relationship between the depth value and the reciprocal of the object distance.

Download Full Size | PDF

4. Conclusions

A DFD measurement method using LC lens with constant-magnification structure is proposed. We introduce the focal power variation and parameterized defocus function relative to focal plane. Defocus function is also redefined so that the normalized image ratio is monotonic under the designed space frequency. The depth map can be recovered from near focused image and far focused image. In the experiment, the depth values of objects ranging from 150 to 2250 mm are obtained.

Funding

Sichuan Science and Technology Program (2017JY0022); Fundamental Research Funds for the Central Universities (ZYGX2016J076).

References

1. M. Hansard, S. Lee, O. Choi, and R. Horaud, Time of Flight Cameras: Principles, Methods, and Applications (Springer Publishing Company, 2012).

2. S. Foix, G. Alenya, and C. Torras, “Lock-in time-of-flight (ToF) cameras: A survey,” in Proceeding of IEEE Conference on Sensors Journal (IEEE, 2011), pp. 1917–1926.

3. B. Kang, S. Kim, S. Lee, K. Lee, J. Kim, and C. Kim, “Harmonic distortion free distance estimation in ToF camera,” Proc. SPIE 7864, 786403 (2011). [CrossRef]

4. S. May, B. Werner, H. Surmann, and K. Pervolz, “3D time-of-flight cameras for mobile robotics,” in Proceeding of IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2007), pp. 790–795.

5. T. Schamm, M. Strand, T. Gumpp, R. Kohlhaas, J. Zollner, and R. Dillmann, “ Vision and ToF-based driving assistance for a personal transporter,” in Proceeding of IEEE Conference on Advanced Robotics (IEEE, 2009), pp. 1–6.

6. P. J. Besl, Active Optical Range Imaging Sensors (Adcances in Machine Vision, 1989).

7. C. H. Lin, R. A. Powell, L. Jiang, H. Xiao, S. J. Chen, and H. L. Tsai, “Real-time depth measurement for micro-holes drilled by lasers,” Meas. Sci. Technol. 21(2), 025307 (2010). [CrossRef]

8. E. Horn and N. Kiryati, “Toward optimal structured light patterns,” in Proceeding of IEEE International Conference on Recent Advances in 3-D Digital Imaging and Modeling (IEEE, 1997), pp. 28–35.

9. C. H. Wu, Y. N. Sun, and C. C. Chang, “Three-dimensional modeling from endoscopic video using geometric constraints via feature positioning,” in Proceeding of IEEE Transactions on Bio-medical Engineering (IEEE, 2007), pp. 1199–1211.

10. J. F. Li, Y. K. Guo, J. H. Zhu, X. Lin, Y. Xin, K. Duan, and Q. Tang, “Large depth-of-view portable three-dimensional laser scanner and its segmental calibration for robot vision,” Opt. Lasers Eng. 45(11), 1077–1087 (2007). [CrossRef]

11. N. Short, “3-D point cloud generation from rigid and flexible stereo vision systems,” Masters Degree Thesis of Virginia Polytechnic Institute and State University, (2009).

12. S. Mattoccia, “Stereo Vision Algorithms and Applications,” Lecture of Department of Computer Science (DISI) University of Bologna, (2011).

13. M. Gerrits and P. Bekaert, “Local stereo matching with segmentation-based outlier rejection,” in Proceeding of IEEE Canadian Conference on Computer and Robot Vision (IEEE, 2006), pp. 66–72. [CrossRef]

14. P. Grossmann, “Depth from focus,” Pattern Recognit. Lett. 5(1), 63–69 (1987). [CrossRef]

15. A. S. Malik and T. S. Choi, “A novel algorithm for estimation of depth map using image focus for 3D shape recovery in the presence of noise,” Pattern Recognit. 41(7), 2200–2225 (2008). [CrossRef]

16. M. Watanabe and S. K. Nayar, “Rational filters for passive depth from defocus,” Int. J. Comput. Vis. 27(3), 203–225 (1998). [CrossRef]

17. M. Watanabe and S. K. Nayar, “Telecentric optics for constant-magnification imaging,” Technical Report CUCS-026–95, Dept. of Computer Science, Columbia University, New York, NY, USA, (1995).

18. M. Kawamura and S. Ishikuro, “Feature extraction from multiply focal images by using a liquid crystal lens,” Mol. Cryst. Liq. Cryst. (Phila. Pa.) 613(1), 51–58 (2015). [CrossRef]

19. S. Pasinetti, I. Bodini, M. Lancini, F. Docchio, and G. Sansoni, “A depth from defocus measurement system using a liquid lens objective for extended depth range,” in Proceeding of IEEE Transactions on Instrumentation and Measurement (IEEE, 2017), pp. 441–450. [CrossRef]

20. S. Pasinetti, I. Bodini, M. Lancini, F. Docchio, and G. Sansoni, “Automatic selection of focal lengths in a depth from defocus measurement system based on liquid lenses,” Opt. Lasers Eng. 96, 68–74 (2017). [CrossRef]

21. M. Ye, B. Wang, and S. Sato, “Liquid-crystal lens with a focal length that is variable in a wide range,” Appl. Opt. 43(35), 6407–6412 (2004). [CrossRef] [PubMed]

22. M. Born and E. Wolf, Principle of Optics (Cambridge University Press, 1999).

23. M. Ye, B. Wang, M. Uchida, S. Yanase, S. Takahashi, M. Yamaguchi, and S. Sato, “Low-voltage-driving liquid crystal lens,” Jpn. J. Appl. Phys. 49(10), 100204 (2010). [CrossRef]

24. A. N. Joseph Raj and R. C. Staunton, “Rational filters design for depth from defocus,” Pattern Recognit. 45(1), 198–207 (2012). [CrossRef]

25. A. P. Pentland, “A new sense for depth of field,” in Proceeding of IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE, 1987), pp. 523–531. [CrossRef]

26. M. Gokstorp, “Computing depth from out-of-focus blur using a local frequency representation,” in Proceeding of 12th International Conference on Pattern Recognition (IEEE, 1994), pp. 153–158. [CrossRef]

27. Y. Xiong and S. A. Shafer, “Depth from focusing and defocusing,” in Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 1993), pp. 68–73. [CrossRef]

V₁ (V_rms)	V₂ (V_rms)	P (m⁻¹)	rms aberr.	V₁ (V_rms)	V₂ (V_rms)	P (m⁻¹)	rms aberr.
3.5	1.2	4.4	0.05	1.1	2.2	−3.9	0.07
3.5	1.3	4.3	0.05	1.2	2.2	−3.6	0.06
3.5	1.5	3.8	0.07	1.3	2.2	−3.2	0.05
3.5	1.7	3.2	0.06	1.4	2.2	−2.9	0.04
3.5	1.9	2.5	0.05	1.5	2.2	−2.5	0.04
3.5	2.1	2.0	0.04	1.6	2.2	−2.2	0.03
3.5	2.3	1.4	0.03	1.7	2.2	−1.9	0.03
3.5	2.5	1.0	0.03	1.8	2.2	−1.6	0.03
3.5	2.7	0.5	0.03	1.9	2.2	−1.3	0.03
3.5	2.9	0.2	0.03	2	2.2	−1.0	0.03
3.5	3.1	−0.1	0.03	2.1	2.2	−0.8	0.03
3.5	3.3	−0.3	0.03	2.2	2.2	−0.5	0.03

object	u (mm)	depth value
1	125	48
2	180	87.6
3	250	109.2
4	410	159.8
5	480	174.2
6	570	181.7
7	2250	207.8

V₁ (V_rms)	V₂ (V_rms)	P (m⁻¹)	rms aberr.	V₁ (V_rms)	V₂ (V_rms)	P (m⁻¹)	rms aberr.
3.5	1.2	4.4	0.05	1.1	2.2	−3.9	0.07
3.5	1.3	4.3	0.05	1.2	2.2	−3.6	0.06
3.5	1.5	3.8	0.07	1.3	2.2	−3.2	0.05
3.5	1.7	3.2	0.06	1.4	2.2	−2.9	0.04
3.5	1.9	2.5	0.05	1.5	2.2	−2.5	0.04
3.5	2.1	2.0	0.04	1.6	2.2	−2.2	0.03
3.5	2.3	1.4	0.03	1.7	2.2	−1.9	0.03
3.5	2.5	1.0	0.03	1.8	2.2	−1.6	0.03
3.5	2.7	0.5	0.03	1.9	2.2	−1.3	0.03
3.5	2.9	0.2	0.03	2	2.2	−1.0	0.03
3.5	3.1	−0.1	0.03	2.1	2.2	−0.8	0.03
3.5	3.3	−0.3	0.03	2.2	2.2	−0.5	0.03

object	u (mm)	depth value
1	125	48
2	180	87.6
3	250	109.2
4	410	159.8
5	480	174.2
6	570	181.7
7	2250	207.8

Depth from defocus measurement method based on liquid crystal lens

Abstract

1. Introduction

2. DFD optic model

2.1 LC lens

2.2 DFD optic model

3. Experimental results

4. Conclusions

Funding

References

Cited By

Figures (5)

Tables (2)

Equations (12)

Optics Express