1. INTRODUCTION

Structured light systems for three-dimensional (3D) imaging have succeeded in diverse fields, including engineering [1], medicine [2], forensics [3], and heritage preservation [4]. Structured light systems employ a projector to illuminate the object using a known pattern [5], then a camera captures the resultant images, and the 3D shape of the object is retrieved by triangulation [6]. In this process, the camera and projector parameters need to be accurately known [7].

In a simplified manner, a camera or a projector can be considered a black box device, as shown in Fig. 1(a). The device’s input is a point ${\boldsymbol p}$ of the 3D space, and the output is the corresponding point $\boldsymbol \mu$ of the image or slide plane. The primary aim of calibration is to specify a mathematical model and its parameters to reproduce accurately the physical device of interest [8]. The general calibration scheme consists of adjusting the model to minimize the output error, also known as reprojection error, as shown in Fig. 1(b). The calibration is influenced by three main aspects [9]: the imaging model, the accuracy of the input and output points, and the optimization method.

 

Fig. 1. (a) Camera or projector considered as a black box receiving points of the 3D space and returning points of the image of the slide plane. (b) Device calibration by estimating the model parameters that minimize the output (reprojection) error. (c) Calibration using a 3D target. (d) Calibration using a 2D target at different poses.

Download Full Size | PDF

Imaging model. This model is aimed to describe the imaging process with high fidelity while remaining as simple as possible [10]. The pinhole model is widely used because it reproduces most typical low-distortion devices acceptably [11]. In addition, the pinhole linearity permits the design of simple parameter estimation algorithms [12], whose outcomes are also useful as an initial guess to fit more advanced models. Some advanced imaging models are distortion lens compensation [13], distorted pinhole [14], fish-eye [15], vision ray [16], division [17], and pixel-wise rational [18].

Point correspondences. Matched input and output points are known as point correspondences and must be precisely determined for accurate calibration [19]. Single-shot calibration is possible using a 3D target [20] since spatially distributed points become available, as shown in Fig. 1(c). However, 3D targets are bulky and expensive; moreover, the determination of point correspondences is challenging due to self-occlusion [21]. A more practical approach consists of using a 2D target placed at several arbitrary positions [22], as shown in Fig. 1(d). Typical 2D targets are checkerboards, circle arrays, circular points, and square-radial patterns [23–25].

Optimization method. The most widely used optimization method for calibration is the least squares [26]. This method minimizes the sum of the squared output error, allowing the development of simple algorithms and stable computations. Sophisticated methods such as particle swarm optimization [27], genetic algorithms [28], and machine learning [29] are more robust to point location uncertainty and avoid local sub-optimality at the price of high computational complexity.

Physically, cameras capture light rays while projectors emit light rays. However, despite their reverse operation, cameras and projectors can be calibrated with the same methodology. Nonetheless, the specific implementation requires some considerations that can be confusing. For instance, it is erroneously understood that the calibration of a projector is possible only with an attached auxiliary camera; even more, the calibration of a camera–projector pair is quite different from the independent calibration of a camera or a projector.

In this paper, we present a simple calibration methodology by considering cameras and projectors as generic devices. The calibration process is illustrated experimentally by calibrating a camera and a projector individually. Then, we show that camera–projector calibration is equivalent to the individual calibration of the devices but performed simultaneously and with the advantage of automatically capturing point correspondences for projection calibration. The results provide useful insights into how a structured light system is calibrated. The MATLAB code and data employed in this study are available in (Code 1, Ref. [30]). This paper is organized as follows. First, the theoretical calibration principles are reviewed in Section 2. Next, Section 3 illustrates experimentally the proposed methodology by camera, projector, and camera–projector calibrations. Finally, the conclusion is presented in Section 4.

2. THEORY

For illustration purposes, the classical pinhole model, a checkerboard target, and least squares optimization criterion are employed. However, the general calibration process remains valid even considering alternative models, other calibration targets, and more sophisticated optimization methods.

A. Pinhole Model

In this model, a point ${\boldsymbol p} = [x,y,z{]^T}$ of the 3D space and its correspondence $\boldsymbol \mu$ on the image or slide plane are related as

B. Homography Estimation

Without loss of generality, we can assume that the flat calibration target is lying on plane $xy$ while the device moves freely. The input points can be represented by 2D vectors of the form ${\boldsymbol \rho} = [x,y{]^T}$. In this manner, an input ${\boldsymbol \rho}$ and its associated output $\boldsymbol \mu$ comprise a point correspondence, denoted by

Then, given the set of $m$ point correspondences

Although $\bar {\boldsymbol g}$ has nine entries, there are only eight free unknowns because of the uniqueness up to scale of $G$. Since a point correspondence provides two constraints, the number of correspondences required to estimate $\bar {\boldsymbol g}$ is

Equation (8) can be solved for $\bar {\boldsymbol g}$ using the singular value decomposition (SVD) method [34]. Specifically, if $A = {U_A}{S_A}V_A^T$ is the SVD of $A$, then the least squares solution for $\bar {\boldsymbol g}$ is the last column of ${V_A}$.

It is worth mentioning that the point correspondences must provide as much spatial information as possible for effective calibration. If the calibration target is excessively small, points ${{\boldsymbol \mu} _k}$ tend to cluster around a single point, making the homography estimation sensitive to noise. For this reason, a calibration target covering most of the image area while lying on the focus plane is recommended. Also, arranging the projector to illuminate approximately the same area of the calibration target is desirable. When the minima of four point correspondences are available, three of them must not be collinear.

C. Intrinsic Parameter Estimation

Consider that the device is placed in different positions and orientations, and the corresponding homography is estimated in each case. The resultant homographies will have the form

Note that the first two columns of ${G_k}$ are of the form

The third column of the homographies is omitted to remove the unknown ${{\boldsymbol t}_k}$. Also, since vectors ${\bar {\boldsymbol r}_{1,k}}$ and ${\bar {\boldsymbol r}_{2,k}}$ are rows of a rotation matrix, they can be removed by their orthonormality property, $\bar {\boldsymbol r}_{1,k}^T{\bar {\boldsymbol r}_{2,k}} = 0$ and $\bar {\boldsymbol r}_{1,k}^T{\bar {\boldsymbol r}_{1,k}} = \bar {\boldsymbol r}_{2,k}^T{\bar {\boldsymbol r}_{2,k}}$, leading to

The constraints given by Eq. (13) from the $n$ available homographies can be stacked to construct the system of linear equations

Equation (16) can be solved for ${\boldsymbol w}$ using the SVD method [34]. Then, matrix $W$ is obtained from the solution vector ${\boldsymbol w}$. Next, the Cholesky factorization of $W$ is performed, obtaining ${K^{- 1}}$, which is inverted to obtain the required intrinsic parameter matrix $K$. Theoretically, matrix $W$ in Eq. (14) is symmetric positive definite, and its Cholesky factorization exists. However, $W$ is obtained from the solution of the homogeneous matrix Eq. (16), where ${-}{\boldsymbol w}$ is also a solution, in which case, the Cholesky factorization will fail. In that case, the factorization must be retried using ${-}{\boldsymbol W}$.

Note from Eq. (12) that only the orientation is relevant for estimating $K$ (the position is on the unused third column of ${G_k}$). This peculiarity is helpful because imaging lenses introduce blur to scene points far from the focus plane. Accordingly, rotating the calibration target from the focus plane with positive and negative angles around the $x$, $y$, and $z$ axes is recommended, as shown in Fig. 2(a). For devices with large depths of field, the position can be varied to capture the calibration target into the whole workspace volume.

 

Fig. 2. (a) Illustrative poses ${P_1}, \cdots ,{P_5}$ of the calibration target around the focus plane for intrinsic parameter estimation. Precise control of position and orientation is not required. (b) True and apparent device poses obtained from the homographies $G$ and ${-}G$, respectively. Note that the captured/emitted light rays are at the back of the device in the apparent pose.

Download Full Size | PDF

 

Fig. 3. (a) Point correspondences obtained by moving freely the reference plane, the device, or both. (b) Calibration by the homography-based parameter estimation approach. (c) Checkerboard on the reference plane for camera calibration. (d) Checkerboard on the slide plane for projector calibration.

Download Full Size | PDF

D. Extrinsic Parameter Estimation

Once matrix $K$ is known, the extrinsic parameters are estimated from the intrinsic-free homographies

Note that the columns of ${\tilde R_k}$ are not unit vectors because of scale factors ${\lambda _k}$ and $\lambda _k^2$. Also, the columns of ${\tilde R_k}$ might not be orthogonal because of noise and errors in the homography estimation process. Let ${\tilde R_k} = {U_R}{S_R}V_R^T$ be the SVD of ${\tilde R_k}$. Since all the singular values of a rotation matrix are unity, the orthogonal matrix closest to ${\tilde R_k}$ in the least squares sense is

When rotation matrix ${R_k}$ is known, scalar ${\lambda _k}$ is estimated using the least-squares method as

Even though the homography is invariant to the multiplication by a scalar, the extrinsic parameters will be different if the multiplying scalar is negative. When this occurs, the camera seems to be under the reference plane and capturing light rays from the back, as shown in Fig. 2(b). This case is easily detected by checking whether the $z$ component of ${{\boldsymbol t}_k}$ is negative and correcting by using ${-}G$, which is equal to changing the sign of the first two rows of the rotation matrix and the $z$ component of  ${{\boldsymbol t}_k}$.

E. Checkerboard Location and Detection

Figures 3(a) and 3(b) depict the general scheme of the calibration process, and the MATLAB implementation is provided in (Code 1, Ref. [30]). The calibration process does not distinguish whether the device is a camera or a projector. However, both the location of the checkerboard and the registration of point correspondences differ for camera and projector calibrations, as shown in Figs. 3(c) and 3(d).

1. Camera Calibration

For camera calibration, the checkerboard is placed on the reference plane and captured by the camera, as shown in Fig. 3(c). In this manner, points ${{\boldsymbol \rho} _i}$ on the reference plane are known from the checkerboard design, while points ${{\boldsymbol \mu} _i}$ are extracted from the image by automatic checkerboard detection algorithms [35]. Therefore, the required point correspondences can be obtained accurately without user intervention.

2. Projector Calibration

For projector calibration, the checkerboard is displayed on the slide plane to illuminate the reference plane, as shown in Fig. 3(d). In this case, points ${{\boldsymbol \mu} _i}$ on the slide plane are known from the checkerboard design, while points ${{\boldsymbol \rho} _i}$ on the reference plane need to be registered.

The direct approach for registering points of the reference plane consists of manual coordinate measurement. Alternatively, an auxiliary (uncalibrated) camera can be employed as follows. A checkerboard is placed on the reference plane to obtain the homography ${G_a}$ that relates the reference and image planes, as given by Eq. (4) [7]. Then, for any given point $\boldsymbol \mu$ of the image plane, corresponding point ${\boldsymbol \rho}$ on the reference plane is computed by inverse homography $G_a^{- 1}$ as

Thus, when the projector illuminates the reference plane, the camera captures the resultant scene, and the image is processed for automatic checkerboard detection. Then, the obtained checkerboard points ${{\boldsymbol \mu} _i}$ are “backprojected” by Eq. (26), obtaining the required points ${{\boldsymbol \rho} _i}$ for projector calibration.

3. Camera–Projector Calibration

Camera–projector calibration requires detecting two superposed checkerboards, one located on the reference plane and another displayed on the slide plane. These two checkerboards can be detected simultaneously using different colors [36]. Suppose the checkerboards are yellow and cyan for the reference and slide planes, respectively. The captured image will provide the two checkerboards separately in red and blue color channels. In this manner, the point correspondences required to calibrate the camera and the projector are available by automatic image processing procedures [35].

F. Supplementary Remarks

Estimation refinement. The presented calibration methodology follows an estimation sequence (first ${G_k}$, next $K$, then ${R_k}$, and finally ${{\boldsymbol t}_k}$), where errors can be accumulated. A further refinement to fit all the parameters together is recommended. For example, the parameters obtained by the presented methodology can be used as the initial guess for bundle adjustment [37].

Checkerboard or device movement. In Section 2, the reference checkerboard was assumed to be on the $xy$ plane of the global frame. Since global frames are usually considered absolute and static, it is generally accepted that the checkerboard is fixed while the device is moving. However, the calibration exploits the relative pose between the reference checkerboard and the device. Thus, the checkerboard, the device, or both can move freely. Remember, however, that the device’s pose will be given with respect to the reference checkerboard because it has attached the global frame.

Axis-angle representation of rotations. A 3D rotation matrix has nine elements but only three free parameters. That redundancy is difficult to handle by further estimation refinement. Axis-angle representation is more convenient for 3D rotations because of its inherent numerical stability and compact form (a vector with three elements). The axis-angle representation of a 3D rotation is given by the vector

Pixel coordinates normalization. The accepted unit of length in computer vision is the pixel. However, since the resolution of modern cameras and projectors overcomes thousands of pixels, the pixel unit is not always convenient because of the induced numerical instability. For this reason, pixel coordinate normalization is recommended [38]. Let $\{{{\boldsymbol u}_1}, \cdots ,{{\boldsymbol u}_n}\}$ be a set of $n$ points on the plane. The standard isotropic normalization translates the centroid to the origin and applies a scaling such that the average distance is the root square of two. The normalized points are computed as

For the image/slide plane, a simpler and intuitive normalization can be applied such that the center is in the middle of the image/slide, the range along the larger side is $[- 1,1]$, and the aspect ratio is preserved. Assuming the size of the image/slide is $M \times N$ pixels, $M \lt N$, the centroid and scaling are

3. EXPERIMENTAL CALIBRATION EXAMPLES

Here, the presented calibration methodology is illustrated experimentally. A color camera (iDS UI-3880CP-C-HQ, $3088 \times 2076$ pixels, and 6 mm focal length) and a portable projector (Kodak Luma 150, $1280 \times 720$ pixels) were utilized.

A. Camera Calibration

A hard clipboard covered with a black and white $7 \times 10$ checkerboard (square size of $15 \times 15\,\,{\rm{mm}}$) was used as the reference plane, as shown in Fig. 4(a). Then, the clipboard was moved at $n = 10$ different poses while the camera captured the resultant images. Figure 4(b) shows the first five captured images. Next, the $m = 54$ checkerboard inner corners were detected, obtaining points ${{\boldsymbol \mu} _{i,k}}$. Afterward, points ${{\boldsymbol \rho} _i}$ on the reference plane were replicated $n$ times to match feature points ${{\boldsymbol \mu} _{i,k}}$ and execute the calibration method. The estimated parameters are given in Table 1 (first data row).

 

Fig. 4. (a) Camera calibration setup with the checkerboard on the reference plane and (b) five captured images with checkerboard points detected automatically. (c) Projector calibration setup with the checkerboard on the slide plane and (d) five millimetric gridded papers with checkerboard points registered manually. (e) Camera–projector calibration setup with a cyan checkerboard on the slide plane to illuminate a yellow checkerboard on the reference plane and (f)–(h) five captured color images and the red and blue color channels, respectively.

Download Full Size | PDF

Table 1. Estimated Parameters^a

View Table

B. Projector Calibration

A black and white $7 \times 10$ checkerboard (rectangle size of $102 \times 128$ pixels) was displayed by the projector to illuminate a hard clipboard covered with millimetric gridded paper, as shown in Fig. 4(c). The $m = 54$ inner corners of the projected checkerboard were marked manually on the gridded paper for each of the $n = 10$ free poses of the clipboard, obtaining points ${{\boldsymbol \rho} _{i,k}}$. Figure 4(d) shows five sets of the points registered on the clipboard. Next, points ${{\boldsymbol \mu} _i}$ of the displayed checkerboard were replicated $n$ times to match ${{\boldsymbol \rho} _{i,k}}$ and execute the calibration method. The estimated parameters are given in Table 1 (second data row).

C. Camera–Projector Calibration

The color of the checkerboards was set to yellow (on the reference plane) and cyan (on the slide plane), as shown in Fig. 4(e). Then, the clipboard was moved at $n = 10$ different poses while the camera was capturing the resultant color images, as shown in Fig. 4(f). The two checkerboards were recovered from the red and blue channels of the captured images, as shown in Figs. 4(g) and 4(h). The $m = 54$ points of the $n$ checkerboards on the red channel were detected, obtaining points ${\boldsymbol \mu}_{i,k}^{\rm{red}}$. Similarly, points ${\boldsymbol \mu}_{i,k}^{\rm{blue}}$ were obtained from the checkerboards on the blue channel. Next, points ${{\boldsymbol \rho} _i}$ of the checkerboard on the reference plane were replicated $n$ times to match points ${\boldsymbol \mu}_{i,k}^{\rm{blue}}$ and perform camera calibration. The estimated parameters are given in Table 1 (third data row). The homographies $G_k^{\rm{blue}}$ obtained during camera calibration were employed to “backproject” points ${\boldsymbol \mu}_{i,k}^{\rm{red}}$ to the reference plane as

Then, points ${{\boldsymbol \mu} _i}$ of the checkerboard on the projector slide plane were replicated $n$ times to match ${\boldsymbol \rho} _{i,k}^{\rm{red}}$ and perform the projector calibration. The estimated parameters are given in Table 1 (fourth data row).

 

Fig. 5. Reprojection error from the experimental calibration of the (a) camera and (b) projector.

Download Full Size | PDF

D. Reprojection Error and Accuracy

The estimated parameters were employed to reproject the calibration checkerboard points and then compute the difference to the detected correspondences. The resultant reprojection error is shown in Fig. 5. Table 1 shows that calibration is comparable regardless of whether the devices were calibrated independently or simultaneously. Also, Fig. 5(a) shows that the camera parameters have similar accuracy in independent and simultaneous calibration. In contrast, obtaining the projector parameters by camera–projector calibration is much more accurate than independent projector calibration; see Fig. 5(b). This occurs because the dispersion is influenced by the resolution of the gridded paper used to register correspondences. Improving the projection calibration accuracy by manual point registration with a sub-millimetric resolution is impractical and more susceptible to human errors. For this reason, projector calibration is usually performed using an auxiliary camera to obtain point correspondences with more accuracy and without human intervention. Furthermore, if the camera and projector are elements of a structured light projection system, the camera is an auxiliary for the projector, but at the same time, it is being calibrated by applying camera–projector calibration.

4. CONCLUSIONS

Calibrations of cameras, projectors, and camera–projector pairs follow the same methodology. First, point correspondences between the reference and image/slide planes are provided. Then, the device parameters are estimated using an appropriate imaging model and optimization method. However, the experimental implementation requires some considerations about where the checkerboard is located and how the correspondences are detected.

The checkerboard is on the reference plane for camera calibration, and the correspondences can be obtained automatically and accurately by image processing algorithms. For projector calibration, the checkerboard is on the slide plane, and the correspondences on the reference plane are registered manually or by an auxiliary camera. For camera–projector calibration, the camera being calibrated is also an auxiliary for the projector, the two checkerboards (on the reference and slide planes) are detected simultaneously using different colors, and all point correspondences are obtained accurately by automatic image processing algorithms.

The camera, projector, and camera–projector calibrations were illustrated experimentally. The results highlighted the advantages of camera–projector calibration regarding the automatic detection of point correspondences and higher accuracy for projector parameter estimation. Accordingly, despite the independent calibration ability, the device parameters of a structured light system are estimated using camera–projector calibration.