1. Introduction

With the rapid development of visual technology, smartphones usually have deployed multiple CMOS camera modules [1]. The combination of these camera sensors and 3D face reconstruction techniques has spawned many popular applications, such as security authentication [2] (e.g. face reconstruction and face anti-spoofing [3]), user-generated content [4] (e.g. face filtering in augmented reality), interactive entertainment [5] (e.g. Animoji), etc. On the other hand, the operating environment of smartphones is complex and diverse. So, how to use the smartphone to obtain a stable and accurate 3D reconstruction of human face [6,7] is a very challenging but urgent task.

Generally speaking, 3D face reconstruction can be categorized into three groups: 1) Software-based. This kind of method usually utilizes 3D processing software [8] (e.g. 3DS MAX, Maya, Unity3D) to establish a face model. One of the biggest problems is that it requires professional experience, the number of the candidate face model is limited, and the reconstructed face is lack of sense of reality. 2) Hardware-based. This kind of method usually adopts specialized projector or laser (e.g. iPhoneX and HUAWEI Mate20 Pro) [9] to obtain the face shape in a non-contact way. Although it can be accurate to millimeter level, it involves high hardware cost and increases the operation difficulty. 3) Image-based. This kind of method usually uses one or more pictures to build a 3D face shape based on some computer vision models (e.g. Lambert, Phong, Cook-Torrance, etc.) [10]. It has the advantages of simple operation and low cost, but there are still few high-precision reconstruction results based on smartphones, which is heavily affected by the environment. Therefore, it is meaningful to investigate how to accurately reconstruct the 3D face reconstruction using the mobile camera in natural scenarios at low cost.

To reconstruct an accurate 3D face surface, some efforts [11–17] have been made in the past few years. Among them, parametric face modeling and photometric stereo are two main representative methods. The earliest parametric-based face is 3D morphable model (3DMM) [11], which firstly obtains a 3D face dataset through high-precision scanning, and then decouples the 3D face data into shape and texture features via principal component analysis (PCA). The parametric-based methods make it feasible to reconstruct a 3D face from a small number or even a single image. However, the limitation of these methods is that they can only recover the low-frequency shape information, and are difficult to recover the high-frequency detail information of the 3D face surface. In addition, the reconstruction results are also constrained by the capacity of the related face databases. Compared with parametric face modeling, photometric stereo [13,18] is able to reconstruct the 3D face surface with high-frequency details. The advantage of the photometric stereo-based methods is that they can restore the normal vector for each pixel, which makes it feasible to restore the high-frequency details of the 3D surface. However, the selection and calibration of the light source have a non-negligible influence on the reconstruction results. For example, Vogiatzis [12] et al. introduced a multi-view geometry and multi-spectral photometric stereo for 3D face reconstruction. A coarse face model was first reconstructed via surface-from-motion (SFM), and it was then used for the light source calibration. After obtaining the light source parameters, the corresponding face normal map can be estimated. In the near-point light source, Cao et al. [13] adopted 3DMM to fit the proxy face model, and estimated the light source direction for each surface patch. After obtaining the light source parameters, they further utilized the photometric stereo method to generate the surface normal, and iteratively obtained a high-precision face normal map. Wang et al. [14] proposed a two-stage coarse-to-fine face surface normal restoration network. In the first stage, the proxy estimation network obtained the 3DMM parameters from the input face images, and subsequently rendered the corresponding coarse normal map. In the second stage, the normal estimation network has been employed to obtain the enhanced normal map. However, the above methods need to be performed under ideal laboratory conditions, based on complicated light source and camera configurations, so it is difficult to be applied to real scenes in daily life.

Most recently, some face reconstruction methods have been developed based on camera sensors from smartphone or laptop. Agrawal et al. [15] used a smartphone to take a video around the face, and reconstructed the corresponding 3D face shape via multi-view geometry. However, it needs a long shooting time ($15-20$ seconds) and consumes a high computing resource, which makes it difficult to be used in practical security authentication-based applications. In addition, the reconstructed faces are overly smoothed due to the filtering of video frames. Farrukh et al. [16] designed a liveness detection system via smartphone camera based on uncalibrated 3D reconstruction. Due to the unknown lighting assumption, they have to leverage an additional face template and singular value decomposition (SVD) to estimate the surface normal. As a result, this method usually generated oversmoothed surface texture and distorted face shape. Moreover, the computation of SVD is a non-negligible payload for a power-constrained smartphone.

In this article, we propose a Smartphone sCreen Illumination-based Face reconstructIon (SCIFI) scheme based on the front camera and screen lighting. Specifically, to improve the reconstructed details of face surface, we propose to adopt the calibrated planar lighting to solve the Lambertian-based reconstruction. Further, we employ the face landmarks to align multiple face photographs taken by handheld devices, which can effectively relieve the position mismatch effect caused by jitter in face reconstruction. In addition, we design different normal outlier elimination methods according to the characteristics of human face, which can avoid the face deformation caused by singular normals. Finally, qualitative and quantitative experiments show that SCIFI can effectively restore the 3D surface shape as well as the texture of human face. Compared with recent representative methods, SCIFI has the following main contributions.

The organization of this article is arranged as follows. In Section 2., we briefly introduced the basic foundation of the Lambertian model for 3D surface reconstruction. In Section 3., we provide the details of the proposed face reconstruction scheme. In Section 4., we provide the comparison results, and performance analysis of SCIFI. Finally, the overall conclusion is drawn in Section 5.

2. Background

2.1 Lambertian model

Given an object to be reconstructed, the calibrated photometric stereo requires a fixed viewpoint (i.e. camera position) and more than three calibrated parallel light sources. For Lambertian surfaces, pixel intensity $i(x,y)$ at any position $(x,y)$ on the object surface can be represented by

Under multiple light sources, Eq. (2) can also be converted into a matrix form as

3. Proposed SCIFI framework

In this section, we give a detailed description for the proposed SCIFI scheme as illustrated in Figure 1. SCIFI has been specially designed for both dark and bright lighting conditions. When a lighting pattern is selected and the related face photographs are captured, face alignment will be conducted to address the position mismatch issues caused by hand jitter. Subsequently, a calibrated planar lighting is employed to estimate the face surface normal. To remove outlier normals, different filtering methods have also been designed for different facial regions. Finally, a discrete geometry-based surface from normal method is employed to generate 3D face shape.

 

Fig. 1. Pipeline of the proposed Smartphone sCreen Illumination-based Face reconstructIon (SCIFI) framework.

Download Full Size | PDF

3.1 Screen lighting patterns

According to the isometric model of the planar light source, we need to obtain the relative distance $D$ between each face surface patch and the planar light source to perform the surface reconstruction. However, this is a dilemma that since once we know the relative distance $D$ for each surface patch, we have already reconstructed the related surface. Considering that the distance between face regions and the distance between the face and the planar light source have a large gap, we assume that all face surface patches are on the same plane. We take the nose tip as a reference, and assume that the distance between the reference and the screen of smartphone is a constant for simplicity, and the face is parallel to the phone screen. In other words, the distance between each screen region and the face surface is considered as a constant, and it is set as $20~cm$ in the experiments.

Figure 2 shows the illustration of the proposed screen lighting patterns with three different colors. When a subject holds the smartphone, different screen regions are sequentially lighted. At the same time, the front camera is used as the photograph acquisition device. The screen of a smartphone can be divided into $4$-zones or $9$-zones. According to the color of the lighting source, it can be further divided into white ($W$), red ($R$), green ($G$), and blue ($B$).

 

Fig. 2. Illustration of the proposed screen lighting patterns. (a) The relative position relationship between the human face and the screen of smartphone. (b) Two lighting patterns (top: $4$-zones; bottom: $9$-zones). From left to right in the zig-zag way, we mark them as $W4$, $R4$, $G4$, $B4$, $W9$, $R9$, $G9$, and $B9$, respectively.

Download Full Size | PDF

3.2 Face alignment

In the 3D face reconstruction based on the smartphone camera, hand jitter is an inevitable problem in capturing photographs. Due to the hand jitter, the face positions of the participated subject in different photographs may be different. To solve this mismatch problem, we propose to employ face alignment [19] to relocate the face position. In the experiments, we generate 106 landmarks to estimate the similarity matrix to relocate the face position.

In the experiments, we take the first face photograph as the reference, and align the other face photographs. The relationship between the landmarks $(x_i^{\prime },y_i^{\prime })(i=1,2,\ldots,106)$ of the reference photograph and the landmarks $(x_i,y_i )(i=1,2,\ldots,106)$ of the other photographs can be modeled as

For computational efficiency, the mappings for all landmarks are organized as a matrix multiplication, and Eq. (4) is reorganized as

For convenience, Eq. (5) can also be represented as

Finally, the solution of $\mathbf {F}^{\prime }$ can be converted to solve the following optimization

In order to deal with the impact of some seriously deviated landmarks, we can also adopt iteratively reweighted least squares (IRLS) to solve Eq. (7).

3.3 Feature extraction

According to the illumination condition, the use case of SCIFI can be colloquially divided into dark environment and bright environment. In a dark environment, we conduct the experiments under the white light, such as the $W4$ and $W9$ modes as shown in Figure 2. In a bright environment, it is difficult to create shadows on the face due to the similarity of white light and natural light. For this, we propose a “shading-from-color-lighting" method to use the color light as the lighting sources.

In the experiments, we first take a face photograph under the natural light as the reference $\mathbf {I}_{ref}$, where the screen is not lighted at this time. Then, we obtain the face photograph $\mathbf {I}_k$ by sequentially lighting the colored lights in different screen regions. Finally, we obtain the difference image by subtracting the photograph obtained under the color lighting source from the reference one, which is used as the feature image $\mathbf {I}_{type}^k$ for the corresponding color type.

For example, when the green light is used, the feature component $\mathbf {I}_{G}^k$ is extracted. In fact, we can also use the color lighting source in a dark environment, and directly extract the corresponding color component as the feature image. In other words, $\mathbf {I}_{ref}=\mathbf {0}$. It is worth noting that we need to flip the captured photograph horizontally before obtaining the feature image due to the use of the front camera.

3.4 Calibrated planar lighting

Under a planar lighting condition $\vec {\boldsymbol {l}}$, using the parallel light assumption to solve Eq. (1) will bring errors to the estimation of $\vec {\boldsymbol {n}}$. Therefore, it is necessary to incorporate different types of light source assumptions, such as near-point light sources [18,20], circular plane light sources [21], and rectangular planar light sources [22]. In [22], Clark et al. has made an assumption that the illumination effect of a rectangular planar light source could be equivalent to a point light source at infinity. As an illustration in Figure 3, we set the position of the face surface patch to the origin of the world coordinate system, and assume that the rectangular planar light source is perpendicular to the $z$-axis. The emissivity of the resulting micro surface patch can be modeled by $\mathcal {R}_{s}(\cdot )$, and defined as

 

Fig. 3. Illustration of the relative position between a planar lighting source and a micro face surface patch.

Download Full Size | PDF

Consequently, Eq. (10) can be rewritten into the vector form as,

3.5 Normal recovery

After obtaining the feature component $\mathbf {I}_{tpye}^k$, we further recover the normal vector of the face surface. First, we look for a convex hull based on the aligned 106 landmark points to locate the interesting region. Subsequently, we carry out the pixel-by-pixel light modeling and normal estimation. Assume that the height and width of the smartphone screen are $H_{sp}$ and $W_{sp}$, respectively.

In the screen lighting pattern of $4$-zones, the position matrix with respect to the position of the nose tip can be denoted by $\mathbf {P}_{4}$, and expressed as

Moreover, we represent the height and width of the bounding box of the interesting face region as $H_{bb}$ and $W_{bb}$, respectively. The height of the average human face, $H_{face}$, is assumed to be $15~cm$. Then, the width of a real face, $W_{face}$, can be estimated as

Let the low-left corner of the bounding box be the origin of the coordinate system. Let the nose tip be the center of the bounding box, and the position is denoted as $(c_x, c_y)$. Let the pixel position in the $i$-th row and $j$-th column of the face image be $(p_x, p_y)$. Then, we deduce the offset between the face image and the real face with respect to the nose tip by Eq. (14).

To calculate the relative position of the face surface patch and lighting source, we only need to subtract the corresponding offset from the $x$ and $y$ coordinates in the position matrix. For instance, the position matrix of $\mathbf {P}_{4}^{i, j}$ corresponding to the $i$-th row and $j$-th column is

When obtaining the surface patch position $\mathbf {P}_{4}^{i, j}$, we can further obtain the planar lighting direction. According to Eq. (10), we first compute three internal parameters for the $k$-th planar light under the 4-zones pattern by

Then, we can compute the planar light direction by

By the similar way, the planar light direction can be obtained for the 9-zones lighting pattern. Consequently, the direction of each parallel light source can be obtained, and the normal vector for the face surface patch can be computed by via Eq. (11).

3.6 Outlier elimination

Under an ideal condition, if the reconstructed object is rigid and stationary relative to the camera, the surface conforms to the Lambertian reflection. However, illumination, hand jitter, and micro expressions together make the reconstruction condition complex. As a result, outlier normals usually appear in areas, such as eye, nose, eyebrows, and face contour.

In the eye region, the surface of the eyeball is of specular reflection. Further, hand jitter and the alignment error will jointly cause slightly overlapping artifacts. In the nose region, the grease on the nose tip tends to produce specular reflection. In the eyebrow area, the slight movement of the face may lead to the mixing of black non-reflective hair and normal skin. In the face contour, the inaccuracy of landmark prediction makes the face contour expansion error, resulting in the edge area containing complex background. Therefore, it is necessary to remove local outliers. Figure 4(b) and 4(c) show the results before and after the outlier elimination, respectively.

 

Fig. 4. Illustration of the outlier normal elimination.

Download Full Size | PDF

The eyes and nose belong to the low-frequency components in the face surface, so we adopt the mean filter to remove the high-frequency noise and retain the low-frequency shape. For the eyes, we construct bounding box with landmarks as the filtering area, and then perform mean filtering with a kernel size of $(w_{eye}, w_{eye})$. For the nose tip, we reduce the bounding box to half of the landmarks as the filtering area, and perform mean filtering with a kernel size of $(w_{nose}, w_{nose})$. For the eyes and nose regions, $\mathbf {N}$ is filtered by

The eyebrows have high-frequency details, so we design a special operation as follows. For the left eyebrow ( Figure 4(d)), we construct the area with the upper vertex of the nose contour and the upper left vertex of the face. As shown in Figure 4(e), we remove the eyes, eyebrows and edges in this area, and denote the average normal of this area as the reference $\boldsymbol {n}_{ref}^{x,y}$. Considering that the angle between normal vectors in this area should be similar, we propose to use a threshold $t_{ebs}$ to remove abnormal normals. For any normals in the eyebrow area, if the angle between it and $\boldsymbol {n}_{ref}^{x,y}$ is bigger than the threshold $t_{ebs}$, it is regarded as an outlier. Then, the related normal is replaced by the average normal in this area. The eyebrow processing can be formulated as

For the face contour, we obtain the facial boundary via the landmarks in Eq. (7), and denote the boundary width by $w_{edge}$ (Figure 4(h) and (k)). The main goal is to remove outliers and smooth the face contour region. For the left boundary area, we construct the reference area at the left vertex of the nose contour, and remove the eyes and edge. Accordingly, this reference area is shown in Figure 4(i), and $t_{edge}$ is used as the threshold to filter abnormal normals by Eq. (19). For the right edge area, we take the similar approach as shown in Figure 4(j). Finally, we further perform a smooth filtering Eq. (18) on the edge area with a kernel size of $(w_{edge}, w_{edge})$.

3.7 Face surface reconstruction

After recovering the normal map, we can integrate it to obtain the final reconstructed face surface, which is also called surface-from-normal (SfN) [23]. Specifically, [24] developed an efficient SfN method based on discrete geometric processing (DGP) to reconstruct the 3D surface. In DGP, each pixel in a normal map is expressed as a quadrilateral facet. Let the quadrilateral facet related to a pixel $({x,y})$ be represented by $\mathcal {F}_{x,y}$ with the normal $\vec {\mathbf {n}}_{x,y}$. $\mathcal {F}_{x,y}$ has four corner vertexes, like $\mathbf {v}_{x,y}$, $\mathbf {v}_{x,y+1}$, $\mathbf {v}_{x+1,y+1}$, and $\mathbf {v}_{x+1,y}$ ordered from the upper-left to the bottom-left. Let $\mathbf {d}_{x,y}$ be the depth component. In addition, let $\mathcal {Z} (\cdot )$ represent the height (or depth) field function, and $\mathcal {Z}(\mathbf {v}_{r,c})=\mathbf {d}_{x,y}$.

Given $\vec {\mathbf {n}}_{x,y}$, a shape-up strategy is used to predict the depth value of $\mathbf {v}_{x,y}$. A straightforward way to reconstruct a face surface ${\mathcal {F}}$ is to move every vertex $\mathbf {v}_{x,y}$ to a proper position, where the normal of ${\mathcal {F}_{x,y}}$ should be the same as $\vec {\mathbf {n}}_{x,y}$. Thus, the SfN can be represented by the following optimization problem.

Numerical DGP solver: We solve the problem of (20) by a local-global DGP strategy. In other words, the 3D face shape is reconstructed by iteratively applying the local projection and global deformation operations. To be more specific, each facet ${\mathcal {F}_{x,y}}$ is first lifted up by locally projecting the related depth value onto a feasible region one-by-one, and then globally blend all facets by minimizing the shape variation that satisfies the normal constraint.

1) Local projection: In the local projection, the vertexes of ${\mathcal {F}_{x,y}}$ are projected onto a plane with the normal $\vec {\mathbf {n}}_{x,y}$, and the plane is supposed passing though the center $\vec {\mathbf {c}}_{x,y}$. Then, the local projection of each vertex in ${\mathcal {F}_{x,y}}$ can be obtained by

2) Global deformation: In the global deformation, the face surface is deformed that all vertexes are expected to generate the same surface shape as the projected one in Eq. (21). Thus, the global deformation process, $\mathcal {G} \left ( {\mathcal {F}_{x,y}} \right )$, can be modeled by minimizing the following least square problem,

After initializing $\mathbf {d}_{x,y}$, it is employed to predict the local position by Eq. (21) in the local projection. Then, the resulting $\mathcal {L}(\mathbf {v}_{x,y})$ is used to update the vertex position by minimizing Eq. (22) in the global deformation. Such a local-global operation is iteratively performed, which can eventually form the mesh surface $\mathcal {F}$ whose orientations are expected to be the same as the input normals.

4. Experimental results and analysis

In the experiments, we choose Xiaomi Mi 8SE as the test smartphone. The focus length of the front camera is $3.86~mm$. The height $H_{sp}$ and width $W_{sp}$ are $13.5~cm$ and $6.5~cm$, respectively. In the outlier elimination, $w_{eye}=5$, $w_{nose}=11$, $w_{edge}=11$, $t_{ebs}=30$, and $t_{edge}=90$. To satisfy the assumption conditions as mentioned above, we draw a face contour on the front screen. When capturing an image, once a subject fits the reference face contour, we consider the test face meets the previous assumptions for simplicity.

4.1 Visual analysis of the reconstruction results

4.1.1 Face reconstruction in a dark environment

Figure 5 shows the reconstruction results based on two screen lighting patterns (i.e. 4-zones and 9-zones) in a dark environment. We have the following observations based on the practical reconstructed faces: 1) 9-zones provides better visual results than 4-zones under the same color lighting. The main reason is that Eq. (3) can obtain a better estimation of $\mathbf {N}$ when the number of lights increases. 2) White light ($W4$ and $W9$ in Figs. 5(a) and (e)) generates a better surface than the R-G-B color lights. The main reason can be that the three primary lighting colors can be absorbed or affected by the facial skin color, which will further increase the outlier normals. 3) Red light in 4-zones generates the worst reconstructed surface. It can be seen that in Figure 5(b), the red light $R4$ is heavily interacted with the skin color. The central axis area of the face in red lighting will have large noise, which will cause serious distortion on the reconstructed face shape. However, the reconstruction performance can be greatly improved (see $R9$ in Figure 5(f)) when the input number of face images increases.

 

Fig. 5. Reconstruction results of SCIFI in a practical dark environment. Two screen lighting patterns (4-zones and 9-zones) are evaluated under four different color lights, including white ($W4$ and $W9$), red ($R4$ and $R9$), green ($G4$ and $G9$), and blue ($B4$ and $B9$). From left to right: face photographs, estimated normals, surface in view 1, and surface in view 2.

Download Full Size | PDF

4.1.2 Face reconstruction in a bright environment

Figure 6 shows the reconstruction results in a natural light environment. From left to right, we provide the captured images under different light sources, the restored normal map, and the reconstruction results in two viewpoints. In a natural light environment, the blue and green lights can achieve satisfying reconstruction results than the red light. Further, the reconstructed result of the red light in a bright environment is slightly better than in a dark environment. It also confirms that the red color is similar to our skin color, and the environment lighting can alleviate the effect of the red light.

 

Fig. 6. Reconstruction results of SCIFI in a practical natural lighting condition. Two screen lighting patterns (4-zones and 9-zones) are evaluated under three different color lights, including red ($R4$ and $R9$), green ($G4$ and $G9$), and blue ($B4$ and $B9$). From left to right: face photographs, estimated normals, surface in view 1, and surface in view 2.

Download Full Size | PDF

4.1.3 Facial expression reconstruction

Figure 7 shows the reconstruction results of facial expression details under the 4-zones pattern in a dark environment. For comparison, we magnify some key face regions, like mouth, nose, and brow. It shows that SCIFI can successfully restore the micro textures of different facial expressions. This can be very usefully for face recognition, especially in liveness detection.

 

Fig. 7. Reconstruction results of SCIFI on facial expressions. Micro expression details can be successfully reconstructed in a dark environment. Please zoom in the electronic version for details.

Download Full Size | PDF

4.1.4 Reconstruction performance comparisons

Figure 8 shows the comparison results of five different subjects. Without loss of generality, we provide the reconstruction results under the white light in the 4-zones pattern. For each subject, we compare three representative reconstruction methods, including SCIFI, Farrukh2020 [16], and Chen2019 [25]. Compared with the other two latest methods, SCIFI provides better reconstruction results. Further, the deep learned model in Chen2019 [25] has been trained with image data in a laboratory environment, and the reconstruction results are poor under a really changeling condition. This also confirms that domain adaptation is one of the main unsolved problems in deep learning.

 

Fig. 8. Comparison results for five different subjects under the $W4$ pattern. From left to right: face photographs, SCIFI, Farrukh2020 [16], and Chen2019 [25].

Download Full Size | PDF

4.2 Quantitative comparison and analysis

4.2.1 Shape angle change analysis

To evaluate the shape correctness of the reconstruction results, we extract $20$ models from the high-precision face dataset Facescape [17], and extract the average normal map as the baseline (denoted by “Baseline"). Figure 9(a) shows a typical example. We conduct the evaluation by comparing our reconstructed normal map with Baseline.

 

Fig. 9. Illustrations of sampling lines and the related surface angle change curves. Please zoom in the electronic version for better details.

Download Full Size | PDF

As shown in the yellow line in Figure 9(a), we take a cross-section in the vertical direction of the 3D face, and then we sample it. Specifically, the profile of this cross-section is represented as $y(x)$. The derivative function can be obtained by the difference of neighboring normals, which represents the slope at each point. Finally, by converting the slope into an angle, we obtain the angle function $\theta (x)$, and then investigate the 3D face property based on $\theta (x)$.

Further, we perform the sampling segment with the help of the acquired face landmarks. The position of the key point (in blue line) on the cross-section is shown in Figure 9(b), and the corresponding angle change is shown in Figure 9(c). Compared with the Baseline, the angle change curve of Farrukh2020 [16] is larger, indicating that the recovered face shape is severely curved in the vertical direction. For Chen2019 [25], the angle change curve is almost a straight line, indicating that the recovered face shape has almost no angle change in the vertical direction. Compared with Farrukh2020 and Chen2019, SCIFI archives the highest similarity to the curve of Baseline. Similarly, we also conduct experiments in the horizontal direction as shown in Figure 9(d). The experimental results show the similar trend, which validates the efficiency of SCIFI.

4.2.2 Quantitative comparison

To quantitatively analyze the quality of face reconstruction, we compute the average angle deviation between each sampling point and the corresponding position of the baseline. The average angle deviation is denoted by $\hat {\theta }$, and defined as

Table 1 provides the comparison results of the average angle deviation and the running time in dark environment. The average angle deviation is computed by Eq. (23), where the five subjects in Figure 8 are represented by #1–#5 from the top to bottom. Compared with Farrukh2020 and Chen2019, SCIFI achieves the smallest angular deviation value, which shows the reconstruction fidelity of the biological face structure.

Table 1. Comparison of the average angle deviation and the running time in dark environment.

View Table

5. Conclusion and future work

3D face reconstruction on smartphones is a very useful technique, which can be widely used in face recognition, liveness detection, face animation, etc. In this paper, we present a robust 3D face reconstruction system for smartphones, which is only relied on a single front camera with screen illumination. To restore micro facial textures, we have explored the calibrated planar lighting for Lambertian-based reconstruction. In addition, we have employed the face landmark remove the position mismatch between multiple photographs caused by hand jitter. Moreover, we have designed different methods to eliminate singular normals based on the characteristics of human face. Experimental results show that the proposed method can effectively restore more satisfying surface shape as well as micro facial texture in comparison with the latest uncalibrated and data-driven schemes. We believe that the proposed method can provide new inspirations for smartphone-based security authentication in the future.

In the future, some further explorations can be developed to improve the performance of the proposed method. First, the current algorithm is simple and efficient for a single-view 3D face surface, but its performance on a multiple-view case is unknown and needed to be investigated. Second, the proposed method is dependent on the screen illumination, and a complicated lighting condition can be an obstacle. Learning-based methods can be a useful tool to address this problem. Third, the estimated normal precision by the proposed method is relative lower compared with a hardware-based method, and it is worth further investigating normal enhancement method to improve the reconstruction quality.