Superresolution of 3-D computational integral imaging based on moving least square method

Hyein Kim; Sukho Lee; Taekyung Ryu; Jungho Yoon

doi:10.1364/OE.22.028606

1. Introduction

Integral imaging is a method which collects a multiple of small two-dimensional(2-D) images, which are called elemental images, of the same object from different perspectives using an array of microlenses [1,2]. The collected elemental images are stored digitally using a CCD camera. The three-dimensional(3-D) image is reconstructed when these images are displayed through the same array of microlenses. This kind of reconstruction is called the optical integral imaging reconstruction (OIIR). The OIIR technique has several problems, such as the low resolution of the displayed 3-D images due to insufficient overlapping of elemental images, and the degraded image quality caused by the physical limitations of optical devices, e.g., diffraction and aberration. To overcome these drawbacks, the computational integral imaging reconstruction (CIIR) has been introduced, in which 3-D images can be computationally reconstructed via a digital simulation of the geometrical optics [3–8]. One major advantage of the CIIR process is that a 3-D image can be reconstructed at an output plane of desired distance. This advantage results in several applications such as depth extraction or occlusion removal of a certain object [7, 8].

However, both in the OIIR and the CIIR, the resolution of the reconstructed 3-D image is still low compared to other 3-D display techniques. This is due to the fact that in integral imaging the number of the collected images is larger, while the resolution of the images are lower. Therefore, integral imaging produces a 3-D image having a large viewing angle but low resolution. Studies have been undergone both in the OIIR and the CIIR fields to overcome the low resolution problem. Studies in the OIIR field like [9–13] try to increase the resolution by mechanically actuating the lens array, while studies in [14–17] try to increase the resolution by using an additional lens system. However, such approaches require for extra settings in the acquisition stage, and introduce some problems like the motion blur artifact. Meanwhile, studies in the CIIR field try to tackle the low resolution problem by projecting and superposing pixels of the elemental images with different spot sizes [5], applying a rapid moving of the lenslet array to increase the sampling rate of the elemental images [6], or employing the intermediate-view reconstruction (IVR) technique [18, 19].

Superresolution reconstruction [20] was first introduced to the field of integral imaging in [21]. Here, a sequence of integral images is captured using a moving lenslet array technique, which are processed digitally with superresolution reconstruction algorithms. Following this work, an optical system called TOMBO (thin observation module by bound optics) has been developed which collects several low resolution images in parallel with multiple narrow-aperture lenses [22, 23]. In [24], a similar framework has been proposed for infrared camera systems. The optimization and theoretical limits of applying superresolution to lenslet array systems have been anaylized in [25]. In [26], the superresolution framework has been used to obtain the sub-pixel disparity between the elemental images, while in [27], it has been applied on the reconstructed images.

However, until now, the superresolution framework has not been applied to the CIIR method. In the CIIR case, the resolutions of the elemental images have to be increased directly by the superresolution technique, since the resolution of the reconstructed 3-D image depends directly on them. However, there exists difficulties in applying superresolution directly on the elemental images. First, it is difficult to obtain an accurate registration due to the very low resolution of the elemental images. Second, the interpolation has to be done on a non-equally spaced data set. The quality of the reconstructed 3-D image depends on how well the superresolution deals with these difficulties.

In this paper, we propose a superresolution method which sets its target on the CIIR method and works directly on the low resolution elemental images. The proposed method is based on the properties of the moving least square method [28]. Due to the property of the moving least square method, the proposed method can overcome the above mentioned problems to some extent. To deal with the abrupt changes in the values at the edge region, we propose the edge directed moving least square method for interpolation which takes the direction of the edge into account. Furthermore, a framework is proposed which shows how the data have to be collected for the superresolution problem in the CIIR application. It is shown experimentally that the proposed method shows good reconstruction results.

2. Related works

To understand the proposed method, the following related works have to be understood.

2.1. Computational integral imaging reconstruction

Computational integral imaging reconstruction (CIIR) is a technique which computationally reconstructs the 3-D image at an output plane by superposing all the inversely mapped elemental images, based on the pinhole array model. One of the major merits of the CIIR technique is that the 3-D image can be reconstructed at an output plane of desired distance. Figure 1 illustrates this fact. The object which is obtained by the lenslet system can be digitally reconstructed at a plane with arbitrary depth by the reconstruction algorithm. This has several applications such as depth extraction or occlusion removal of a certain object. For example, in [7], a depth extraction method is proposed by observing the fact that in the 3-D image reconstructed at distance z, only the 3-D object with depth z becomes sharp while objects with different depths become blurred. Figure 1 shows the 3-D images reconstructed at different distances. Using an algorithm which discriminates between sharp and blurred objects, the depth of the sharp object can be estimated. In a similar way, object occlusion removal can be achieved by eliminating the object from the elemental images which appears blurry in the reconstructed 3-D image.

Fig. 1 Computational integral imaging reconstruction at output planes with different distances.

Download Full Size | PDF

However, the greatest problem in the CIIR method is the low resolution of the reconstructed 3-D image. This is due to the low resolution of the digitally recorded elemental images. A primitive solution to this problem is to increase the resolution of the sensor in the acquisition system by sensor manufacturing techniques. However, there exists limitation on the pixel size reduction. Furthermore, the light available also decreases with the pixel size, which generates shot noise that degrades the image quality severely. It is known that the current image sensor technology has almost reached this level. Therefore, resolution enhancing methods based on signal processing techniques such as superresolution techniques have to be applied to tackle this problem.

2.2. Superresolution

Superresolution techniques suit well for integral imaging since multiple images of the same scene with translational shifts can be obtained. Especially, due to the irregularity in the positions of the data points, interpolation based superresolution techniques suit better to the integral imaging problem than other superresolution techniques. Most of the interpolation based superresolution techniques require three fundamental steps. First, registration is performed so that the data samples from the low resolution images find their positions in the high resolution grid. Then, a non-uniform interpolation is performed to obtain the high resolution(HR) image from the non-uniformly spaced data samples. Last, image restoration is applied to the HR image to remove blurring and noise.

Until now, superresolution methods have not been applied directly on the LR elemental images. The superresolution framework has been used in [26] to obtain the sub-pixel disparity between the elemental images, and has been applied on the reconstructed images in [27], but not on the LR elemental images directly. The proposed method applies superresolution directly on the LR elemental images, and overcomes the problem of the sensitiveness to the registration error by taking the properties of the moving least square method into account, which are explained in the following section.

2.3. Properties of the moving least square method

The moving least square (MLS) method is a powerful approximation method which takes the data and their position to define the structure of the approximation model so that the local characteristics of the data can be taken into account in the approximation [28]. The MLS approach has several properties which are desirable to overcome the difficulties in superresolution. First, the order of approximation can be set to an arbitrary order, and therefore, it is possible to achieve an interpolation of higher order than linear. This helps to achieve a registration of higher accuracy than linear.

Second, the MLS based interpolation is less sensitive to the error in the registration. Figure 2 illustrates this fact using 1-D data profiles. The blue dots in Fig. 2 show the true 1-D data profile, while the red dots show the data deviated from their true position, which simulates the error in the registration. It can be seen that the difference in the constructed MLS functions (shown in the blue and the red lines in Fig. 2) is not large, even though the positions of the data points deviate much from the original positions. This means that even if we have some error in the registration, we can obtain an acceptable interpolation result. Figure 3 shows this fact using images. Figure 3(a) shows the data points from the LR elemental images embedded into the HR grid, where all the data points are embedded at the correct positions, i.e., the registration is correct, while Fig. 3(c) shows the data points embedded at incorrect positions. However, as can be observed from Fig. 3(b) and 3(d), the interpolated results using the MLS based interpolation are similar. Third, the MLS approximation function can be calculated from a non-equally spaced data point set. Figure 4 shows an example where the MLS approximated elemental image (Fig. 4(c)) is obtained from the non-equally spaced data point set shown in Fig. 4(b). In the next section, we propose a superresolution method which is based on the moving least square method and thus contains the desirable properties described in this section.

Fig. 2 Showing the property of the moving least square method in 1-D.

Download Full Size | PDF

Fig. 3 Showing the property of the moving least square method in 2-D.

Download Full Size | PDF

Fig. 4 MLS approximation using low resolution elemental image data (a) low resolution elemental image (b) embedding of low resolution images into high resolution grid (c) MLS approximation using (b) as data.

Download Full Size | PDF

3. Proposed method

In this section, we propose an edge directed moving least square based superresolution (ED-MLS based superresolution) method to increase the resolution of the reconstructed 3-D image using the data acquired by the conventional CIIR system. The proposed scheme is based on the properties of the moving least square method explained in the previous section. However, in superresolution, there exists an abrupt change in the value of the data, especially at the edge regions. Therefore, the MLS reconstruction has to consider such abrupt changes. Therefore, we propose an edge directive MLS reconstruction, which takes the direction of the edges into account in the reconstruction process.

In Sec. 3.1, we first propose a framework for the ED-MLS based superresolution, and show how the data for the ED-MLS approximation are provided. Then, in Sec. 3.2, we explain the proposed ED-MLS approximation model.

3.1. Overall system diagram

The proposed ED-MLS based superresolution constructs the high resolution elemental image (HR-EI) from low resolution elemental images (LR-EIs) according to the following steps:

Registration step: perform registration on all the LR-EIs which are neighboring to the LR-EI under consideration.
Embed all the LR-EI data into the HR-grid according to the subpixel shift estimated by the registration step.
ED-MLS based interpolation: perform ED-MLS based interpolation to obtain the HR-EI by the following steps:
1. Divide the domain of the HR-EI into subdomains. Here, the subdomain is the region in which a single overall ED-MLS approximation function is constructed (denoted by 𝒩 in Sec. 3.2).
2. For each subdomain, compute an overall ED-MLS approximation function.
3. Concatenate all the overall ED-MLS approximation functions of the subdomains to obtain the HR-EI.
After all the HR-EIs are obtained by the above mentioned steps, the CIIR is performed on the HR-EI array to obtain the final high resolution 3-D image.

Figure 5 shows the overall diagram of the proposed method. The pickup stage is the same as in the conventional CIIR method, i.e., the 3-D object is recorded by a digital CCD camera through a lenslet array to produce the elemental images. After the pickup stage, registration is performed between neighboring elemental images, where registration refers to the method that computes the relative subpixel shifts of the LR-EI images with respect to the reference LR-EI image. To perform the registration, first a similarity measure has to be defined, which computes the similarity of two LR-EIs. Then, for each neighboring LR-EI, the image similarity with the reference LR-EI is computed for several different subpixel shifts, and the subpixel shift that results in the largest similarity is chosen as the subpixel shift of that LR-EI with respect to the reference LR-EI. After the subpixel shift is computed, every pixel in the LR-EI is embedded in the HR grid at the subpixel shifted position (Fig. 6). Finally, the ED-MLS based interpolation is performed to obtain the HR-EI array as will be explained in section 3.2, and finally the reconstructed high resolution 3-D image is obtained by the CIIR process.

Fig. 5 Overall diagram of the proposed system

Download Full Size | PDF

Fig. 6 Illustrating the embedding of LR-EI data into the high resolution grid.

Download Full Size | PDF

3.2. Proposed ED-MLS based superresolution

In this section, we explain the ED-MLS based interpolation step to obtain an HR-EI from several LR-EIs. We describe the neighboring LR-EIs as functions f_i (i = 1, 2,...,S), where S denotes the number of neighboring LR-EIs involved in the ED-MLS based interpolation. The proposed ED-MLS based interpolation constructs an HR-EI from these non-equally spaced discrete data functions f_i by solving the following quadratic minimization problem for each point x_j ∈ 𝒩, where 𝒩 denotes the domain for which we want to construct an overall ED-MLS approximation function:

\underset{p_{x_{j}} (\cdot) \in Π_{m}}{argmin} {\sum_{i = 1}^{S} \sum_{n = 1}^{N_{i}} {‖ p_{x_{j}} (x_{n}) - f_{i} (x_{n}) ‖}_{2}^{2} θ_{i} (x_{j}, x_{n})}, for all x_{j} (j = 1, 2, \dots N_{i}) .

Here, Π_m denotes a polynomial space of dimension m, and N_i is the number of pixels in the elemental image f_i lying inside 𝒩, i.e., x_n (n = 1,..., N_i) are the data points lying inside 𝒩. Furthermore, p_{x_j} (·) denotes the polynomial approximation function calculated at point x_j, and p_{x_j} (x_n) is the function value of p_{x_j} (·) at point x_n.

Compared to the conventional MLS problem, the minimization problem in (1) has to be solved using all the data in the neighboring LR-EI, which fact can be observed by the additional summation over the registered LR-EIs in (1). Furthermore, the weight function θ has to be designed such that the characteristics of the data in the image, e.g., the abrupt change in the brightness value across the edge region, are taken into account.

The θ function is a weight function which decides how much the data point f_i(x_n) contributes to the construction of the polynomial approximation function p_{x_j} (·). If θ(x_j, x_n) is large, f_i(x_n) contributes much to the construction of p_{x_j} (·). In the proposed ED-MLS based superresolution, we choose θ as :

θ (x_{j}, x_{n}) = e^{- {(x_{j} - x_{n})}^{T} M (x_{j} - x_{n}) / σ}

where σ > 0 is a scale parameter and M is a 2×2 matrix chosen so that (x_j − x_n)^T M(x_j − x_n) = 1 is an ellipsoid capable of taking edge information into account. To be more precise, (x_j − x_n)^T M(x_j − x_n) = 1 is an ellipsoid which eigenvector corresponding to the smaller eigenvalue (λ₂) points in the same direction as the gradient vector (v) of the edge (Fig. 7). Therefore, the function θ(·) is elongated along the orthogonal direction of the gradient vector (i.e., along the edge direction). Hence, the values of θ(·) are increasing mildly along the edge direction but rapidly along the gradient directions, which results in involving data along the edge rather than across the edge. This is based on the premise that the data points near to x_j and along the edge direction should have a large effect in the construction. The minimizer p_{x_j} (·) differs from pixel to pixel, i.e., for each x_j. Since we construct p_{x_j} (·) in the polynomial space Π_m of dimension m, we can write p_{x_j} (·) as a linear combination of the polynomial basis

{p_{ℓ}}_{l = 1}^{m}

:

p_{x_{j}} (\cdot) : = \sum_{ℓ = 1}^{m} c_{ℓ} p_{ℓ} .

For example, the polynomial basis in the polynomial space Π₆ (dimension 6) becomes

{p_{ℓ}}_{ℓ = 1}^{6} = {1, x, y, x^{2}, x y, y^{2}}

. Therefore, we see that Eq. (1) is a least squares problem that can be solved for the coefficients {c_ℓ : ℓ = 1,...,m} of the polynomial approximation function p_{x_j} (·). The coefficients {c_ℓ : ℓ = 1,...,m} satisfying Eq. (1) are found by calculating the derivative of the function in Eq. (1) with respect to {c_ℓ : ℓ = 1,...,m} and setting them equal to zero:

\frac{\partial (\sum_{i = 1}^{S} \sum_{n = 1}^{N_{i}} {‖ p_{x_{j}} (x_{n}) - f_{i} (x_{n}) ‖}_{2}^{2} θ_{i} (x_{j}, x_{n}))}{\partial c_{ℓ}} = 0,

for ℓ = 1,...,m. The above quadratic minimization problem has to be solved for every point x_j ∈ 𝒩, i.e., p_{x_j} (·) has to be obtained for every x_j ∈ 𝒩. Therefore, the above problem has to be solved as many times as the number of points x_j ∈ 𝒩. After all the polynomial approximation functions p_{x_j} (·), ∀_{x_j∈𝒩} are obtained, the final overall ED-MLS approximation function Lf can be obtained by the following definition:

L f (x_{j}) = p_{x_{j}} (x_{j}), \forall_{x_{j} \in 𝒩} .

The meaning of the definition is that the function value Lf (x_j) for the point x_j becomes Lf (x_j) = p_{x_j} (x_j). Figure 8 shows the relation between Lf (x_j) and p_{x_j} (x_j) for the 1-D case. It can be seen that the collection of all the p_{x_j} (x_j) points becomes the final approximation function Lf according to Eq. (5).

Fig. 7 Concept of the ED-MLS interpolation.

Download Full Size | PDF

Fig. 8 Illustrating the relationship between the ED-MLS approximation function and the local polynomial approximation functions.

Download Full Size | PDF

We explain now in details, how to solve the problem by a linear system. To this aim, we express the solution that satisfies Eq. (1) as a row vector:

c = {c_{ℓ} : ℓ = 1, \dots, m} .

Likewise, we express also the data points of the i’th LR-EI into vector form as

f_{i} = {f (x_{n}) : x_{n} \in 𝒩} .

Furthermore, in order to express Eq. (4) in a matrix form, we denote

E_{i} : = (p_{x_{j}} (x_{n}) : x_{n} \in 𝒩, j = 1, \dots, N_{i}) for i = 1, 2, \dots, S .

Recalling that the given weight function θ is non-zero for any (x₁, x₂) ∈ ℝ² × ℝ², let D_θ,i indicate the N_i × N_i diagonal matrix corresponding to the matrix E_i whose (k, k)-component is defined by

D_{θ, i} (k, k) : = θ_{i} (x_{k}, x), k = 1, \dots, N_{i} .

Then, using these notation, a simple calculation reveals that the condition Eq. (4) is equivalent to the following linear system:

0 = \sum_{i = 1}^{S} ({cE}_{i}^{T} - f_{i}) D_{θ, i} E_{i} .

This leads to the identity

c = (\sum_{i = 1}^{S} f_{i} D_{θ, i} E_{i}) \cdot {(\sum_{i = 1}^{S} E_{i}^{T} D_{θ, i} E_{i})}^{- 1} .

Finally, with this coefficient vector c, we can obtain all the p_{x_j} (·) for every point x_j by Eq. (3), and using p_{x_j} (·), we can construct the overall approximation function Lf by Eq. (5).

The above mentioned procedure has to be done for every subdomain 𝒩 in the HR-EI. After the ED-MLS approximation functions Lf (·) are obtained for every subdomain 𝒩, they become concatenated to obtain the final HR-EI. Again, the HR-EI has to be obtained for every elemental image in the elemental image array. After all the HR-EIs are obtained, the CIIR is performed on the HR-EI array to obtain the final high resolution 3-D image.

3.3. Computational cost

The computational cost of the registration step is negligible, since the same relative subpixel shifts of the neighboring LR-EIs can be applied to all the LR-EIs, and therefore, the subpixel shifts of the neighboring LR-EIs have to be calculated only once for a single LR-EI.

The major computational cost is due to the computation of the coefficients of the ED-MLS approximation function, shown in Eq. (11), which has to be performed for every LR-EIs. The first term at the righthand-side in Eq. (11), i.e., the term $\sum_{i = 1}^{S} f_{i} D_{θ, i} E_{i}$ , is S times the multiplication of one vector and two matrices of size N_i × N_i, where S denotes the number of neighboring LR-EIs convolved in the computation. However, since D_θ,i is a diagonal matrix, only the diagonal components of D_θ,i are multiplied to the rows of E_i, and therefore, about $S (N_{i}^{2} + N_{i})$ floating point operations are required. Similarly, the computation of $\sum_{i = 1}^{S} E_{i}^{T} D_{θ, i} E_{i}$ is in fact the multiplication of two matrices, multiplied by the diagonal values of D_θ,i, and therefore, requires about $S (N_{i}^{3} + N_{i})$ floating point operations. The inverse of $\sum_{i = 1}^{S} E_{i}^{T} D_{θ, i} E_{i}$ is computed by the LU decomposition, and therefore requires $- \frac{5}{6} N_{i} + \frac{3}{2} N_{i}^{2} + \frac{4}{3} N_{i}^{3}$ floating point operations [30]. Finally, the multiplication of $\sum_{i = 1}^{S} f_{i} D_{θ, i} E_{i}$ and ${(\sum_{i = 1}^{S} E_{i}^{T} D_{θ, i} E_{i})}^{- 1}$ is a multiplication of a vector and a matrix and needs $N_{i}^{2}$ floating point operations. Letting S = 8, i.e., using eight neighboring LR-EIs, the total yields in $\frac{28}{3} N_{i}^{3} + \frac{19}{2} N_{i}^{2} + \frac{91}{6} N_{i} \approx \frac{28}{3} N_{i}^{3}$ floating point operations. This kind of computation has to be done for every LR-EI, and therefore, if the number of LR-EIs is M, the computation cost is approximately $\frac{28}{3} M N_{i}^{3}$ floating point operations. Again, the linear combination of the basis functions computed by Eq. (3) is negligible compared to this computation cost. The computation cost is large, and requires 10–40 seconds depending on the size of the image on a 3.2GHz CPU PC with 16GB memory running under Window7. However, since the computation can be done in parallel for each LR-EI, the computation time can be significantly reduced if parallel computing methods such as CUDA are used or if the code is implemented in hardware by FPGA.

4. Experimental results

For the experiments with CIIR reconstruction, we used three different types of computational lenslet arrays. The lenslet array we used in the experiment on the ‘Ewha’ image has 10 × 10 lenslets and is located at z = 0 mm. The resolution of the recorded elemental images is given by 900 × 900 pixels, since the resolution of each lenslet is 90 × 90 pixels. The focal length and the lateral size of each lenslet are 3 mm and 1.08 mm, respectively, which means that the simulated CCD for the ‘Ewha’ image has pixel pitch of 3mm/90 = 33μm and 1.08mm/90 = 18μm. For the color image ‘Cupcake’, we used a lenslet array having 15 × 15 lenslets, where each lenslet has a resolution of 60 × 60 pixels. Therefore, the resolution of the recorded elemental images becomes 900 × 900 pixels. Finally, for the color image ‘Cup’, we used a lenslet array having 30 × 30 lenslets, where each lenslet has a resolution of 60 × 60 pixels, making the resolution of the recorded elemental images to become 1800 × 1800 pixels. The focal length and the lateral size of each lenslet are the same as in the experiment on the ‘Ewha’ image. For the experiments on color images, we converted the color images into the YCbCr format and then performed the algorithms on each channel. To verify the performance of the proposed method, we increased the size of the recorded elemental images by four times with the proposed method, the bilinear interpolation method, and the bicubic interpolation method, and then obtained the 3-D reconstructed images using the resized elemental images. Therefore, the resolution of each elemental image in the elemental image array are increased by four times, making the final resolution of the ‘Ewha’ and the ‘Cupcake’ elemental image arrays be 1800 × 1800 pixels, and the ‘Cup’ elemental image array be 3600 × 3600 pixels. The number of the lenslets is not changed.

Figure 9 compares the results on the ‘Ewha’ image where the object is located at a distance of z = 8 mm. As can be seen from Fig. 9(a) and 9(b), the bilinear and the bicubic methods cannot recover some information such as the window frames, which the proposed method is able to do. This is due to the fact that, with the proposed method, the neighboring HR-EIs contain different information obtained from the nine neighboring LR-EIs. These information are combined together into the high resolution 3-D image by the CIIR.

Fig. 9 Results on the ‘Ewha’ image. (a) bilinear (b) bicubic (c) proposed. First row : HR-EI, second row: reconstructed high resolution 3-D image.

Download Full Size | PDF

Figure 10 and Fig. 11 compare the 3-D images reconstructed from the elemental images of the ‘Cupcake’ and the ‘Cup’ images, where the objects are located at z = 11 mm for both images. It can be seen from the enlarged regions that the proposed method enhances the details better than other interpolation methods. Especially, the letters in Fig. 10(c) and the notes in Fig. 11(c) are perceivable with the proposed method.

Fig. 10 Results on the ‘Cupcake’ image. (a) bilinear (b) bicubic (c) proposed. First row : HR-EI, second row: reconstructed high resolution 3-D image.

Download Full Size | PDF

Fig. 11 Results on the ‘Cup’ image. (a) bilinear (b) bicubic (c) proposed. First row : HR-EI, second row: reconstructed high resolution 3-D image, third row : enlarged image of partial region.

Download Full Size | PDF

Table 4 compares the PSNR (peak signal-to-noise ratio) and the SSIM (structural similarity index measure) values of the different methods. In the table, we added the results for three additional cases, i.e., the experimental results on the ‘Ewha’ image with distance z = 4 mm, and the ‘Cupcake’ and ‘Cup’ images of distance z = 7 mm. As can be seen, the proposed method has the highest PSNR and SSIM values in all cases.

Table 1. Comparison of the PSNR (peak signal-to-noise ratio) and the SSIM (structural similarity index measure) values between the bilinear, bicubic, and the proposed method

View Table

We also tried to compare the 3-D images reconstructed by a real optical system (Figure 12(a)). We set the experimental condition for the real optical system as follows: we used a lens array with 65 × 42 elemental lenses where each elemental lens has a diameter of 7.47 mm, and a focal length of 29.88 mm. The number of elemental images are 30 × 30, where each single elemental image is of size 60 × 60 pixels. We used as the display panel the 22 IBM T221 model which has a pixel pitch of 0.124 mm, and a total of 3440 × 2800 pixels. The foreground object has a depth of z = 4 mm and the background image has depth z = 6 mm. The original size of the elemental image array is 900 × 900 pixels, where each elemental image has a resolution of 30 × 30 pixels. Using the proposed method and the bilinear interpolation method, we increased the resolution of each elemental image to 60 × 60 pixels making the size of the resulted elemental image array be 1800 × 1800 pixels.

Fig. 12 Reconstruction of multiple objects with different depths by (a) optical system (b) original photo shot (c) foreground object (z = 4 mm) (d) background object (z = 6 mm) (e) elemental image array. Third row : reconstructed with resolution enhancement by the proposed method - (f) left view (g) middle view (h) right view. Fourth row : reconstructed without resolution enhancement - (i) left view (j) middle view (k) right view. The red arrow indicates the parallax between (i) and (k). Fifth row : reconstructed result focusing on the background (l) without resolution enhancement (m) with resolution enhancement.

Download Full Size | PDF

Figure 12(b) shows an original photo shot of the 3-D image reconstructed by the optical system. The third row and the fourth row in Fig. 12 compares the reconstruction results of the elemental images obtained by the proposed method and the bilinear interpolation, respectively. We set the system up to focus on the object with z = 4 mm, so the background image shows some artifacts which is due to the fact that the background plane becomes out of focus and the projected images of neighboring elemental images interfere with each other. It can be seen that the proposed method is capable of obtaining a 3-D image of higher resolution than the bilinear interpolation method. Moreover, it can be observed that a slight parallax can be obtained. However, a large parallax cannot be obtained, since a high-resolution 3-D image in front of the display has a small depth range. Normally, the resolution and the parallax have a trade-off relationship. Therefore, the drawback of obtaining a high resolution 3-D image is the small parallax. This can be overcome if a large lenslet system is used, and if the elemental image array becomes larger, but this will increase the cost of the optical system and requires a larger computation. Figure 12(l) and 12(m) show the reconstruction results by focusing on the background.

As a final attempt, to obtain viewpoint images with larger parallax, we applied the proposed method on a more complex scene which contains multiple objects with different depths. With our optical system having the above-mentioned setting conditions, it is difficult to obtain reconstructed 3-D images with large parallax and high resolution. Therefore, we used the multiple-pixel sub-image transform proposed in [29]. This method transforms elemental images into sub-images by collecting multiple-pixels at the same position. In this experiment, we collected the multi-pixels from the elemental image array enlarged with the proposed scheme. The original elemental image array has the resolution of 1800 × 1800 pixels, and consists of 30 × 30 lenslets, and therefore, each elemental image has a resolution of 60 × 60 pixels. The resolution of the elemental image array is increased to 3600 × 3600 pixels by the proposed method. Figure 13(a) and 13(b) show the increased elemental image arrays of the bilinear interpolation and the proposed method, respectively. Then, the multiple-pixel sub-image transform extracts 15 × 15 pixels from each elemental image to constitute a sub-image, which makes the sub-image to have a size of 450 × 450 pixels.

Fig. 13 Viewpoint images obtained by the multi-pixel sub-image transform of multiple objects with different depths. (a) bilinear interpolated elemental image array. (b) superresolution enhanced elemental image array. (c) viewpoint sub-image array obtained from (b). (d)(e) enlarged images of partial regions (red box and blue box) in (c). (f) enlarged image of a partial region of the sub-image array obtained from the elemental image array in (a). (g) enlarged image of a partial region (red box) in (e).

Download Full Size | PDF

Figure 13(c) shows the sub-image array obtained by the multiple-pixel sub-image transform applied on Fig. 13(b). As can be seen in Fig. 13(c), the transformed sub-image array contains many different perspective images of 3-D objects. We used four objects with depths of z = 7 mm, z = 8 mm, z = 9 mm, and z = 10 mm. Figure 13(d) and 13(e) show the enlarged images of the left-top and the right-bottom sub-images in Fig. 13(c), respectively. It can be seen that the objects show different shifts due to their different depths. Figure 13(g) shows the enlarged region extracted from Fig. 13(e). Figure 13(f) shows the enlarged region of the sub-image obtained by applying the multi-pixel sub-image transform on the bilinear interpolated elemental image array. Again, it is verified that the resolutions of the objects enhanced by the proposed method are increased.

5. Conclusion

In this paper, we proposed an ED-MLS based superresolution framework for CIIR. The proposed method suits well with the CIIR method due to the approximation property of the moving least square method. It has been shown experimentally that the proposed method increases the resolution and the visuality of the reconstructed 3-D image in terms of both quantitative measure and visual perception quality.

Acknowledgments

This work was supported by the IT R&D program of MKE/KEIT. [ 10041682, Development of high-definition 3D image processing technologies using advanced integral imaging with improved depth range], the Priority Research Centers Program 2009-0093827 and the Basic Science Research Program ( 2013R1A1A4A01007868) through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology.

References and links

1. J. Hong, Y. Kim, H.-J. Choi, J. Hahn, J.-H. Park, H. Kim, S.-W. Min, N. Chen, and B. Lee, “Three-dimensional display technologies of recent interest: principles, status, and issues [Invited],” Appl. Opt. 50(34), H87–H115 (2011). [CrossRef] [PubMed]

2. A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94, 591–607 (2006). [CrossRef]

3. H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. 26(3), 157–159 (2001). [CrossRef]

4. S. Hong, J.-S. Jang, and B. Javidi, “Three-dimensional volumetric object reconstruction using computational integral imaging,” Opt. Express 12(3), 483–491 (2004). [CrossRef] [PubMed]

5. D.-H. Shin, E.-S. Kim, and B. Lee, “Computational reconstruction technique of three-dimensional object in integral imaging using a lenslet array,” Jpn. J. Appl. Phys. 44(11), 8016–8018 (2005). [CrossRef]

6. S.-H. Hong and B. Javidi, “Improved resolution 3D object reconstruction using computational integral imaging with time multiplexing,” Opt. Express 12(19), 4579–4588 (2004). [CrossRef] [PubMed]

7. M. Zhang, Y. Piao, and E.-S. Kim, “Occlusion-removed scheme using depth-reversed method in computational integral imaging,” Appl. Opt. 49(14), 2571–2580 (2010). [CrossRef]

8. J.-Y. Jang, D. Shin, and E.-S. Kim, “Optical three-dimensional refocusing from elemental images based on a sifting property of the periodic δ-function array in integral-imaging,” Opt. Express 22(2), 1533–1550 (2014). [CrossRef] [PubMed]

9. L. Erdmann and K. J. Gabriel, “High-resolution digital integral photography by use of a scanning microlens array,” Appl. Opt. 40(31), 5592–5599 (2001). [CrossRef]

10. J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging with nonstationary micro-optics,” Opt. Lett. 27, 324–326 (2002). [CrossRef]

11. H. Navarro, J.C. Barreiro, G. Saavedra, M. Martinez-Corral, and B. Javidi, “High-resolution far-field integral-imaging camera by double snapshot,” Opt. Express 20(2), 890–895 (2012). [CrossRef] [PubMed]

12. M. Martinez-Corral, B. Javidi, R. Martinez-Cuenca, and G. Saavedra, “Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays,” Appl. Opt. 43(31), 5806–5813 (2004). [CrossRef] [PubMed]

13. H. Yoo, “Axially moving a lenslet array for high-resolution 3D images in computational integral imaging,” Opt. Express 21(7), 8873–8878 (2013). [CrossRef] [PubMed]

14. Y.-T. Lim, J.-H. Park, K.-C. Kwon, and N. Kim, “Resolution-enhanced integral imaging microscopy that uses lens array shifting,” Opt. Express 17(21), 19253–19263 (2009). [CrossRef]

15. J.-H. Park, J. Kim, Y. Kim, and B. Lee, “Resolution-enhanced three-dimension/two-dimension convertible display based on integral imaging,” Opt. Express 13(6), 1875–1884 (2005). [CrossRef] [PubMed]

16. C.-W. Chen, M. Cho, Y.-P. Huang, and B. Javidi, “Improved Viewing Zones for Projection Type Integral Imaging 3D Display Using Adaptive Liquid Crystal Prism Array,” J. Display Technol. 10(3), 198–203 (2014). [CrossRef]

17. M.A. Alam, G. Baasantseren, M.-U. Erdenebat, N. Kim, and J.-H. Park, “Resolution enhancement of integral imaging three-dimensional display using directional elemental image projection,” J. Soc. Inf. Display 20(4), 221–227 (2012). [CrossRef]

18. D.-C. Hwang, J.-S. Park, S.-C. Kim, D.-H. Shin, and E.-S. Kim, “Magnification of 3-D reconstructed images in integral imaging using intermediate-view reconstruction technique,” Appl. Opt. 45(19), 4631–4637 (2006). [CrossRef] [PubMed]

19. J.-S. Park, D.-C. Hwang, D.-H. Shin, and E.-S. Kim, “Enhanced-resolution computational integral imaging reconstruction using an intermediate-view reconstruction technique,” Opt. Eng. 45(11), 117004 (2006). [CrossRef]

20. S.-C. Park, M.-K. Park, and M.-G. Kang, “Super-resolution image reconstruction: A technical overview,” IEEE Signal Process. Mag. 21–36May , (2003). [CrossRef]

21. A. Stern and B. Javidi, “Three-dimensional image sensing and reconstruction with time-division multiplexed computational integral imaging,” Appl. Opt. 42, 7036–7042 (2003). [CrossRef] [PubMed]

22. K. Nitta, R. Shogenji, S. Miyatake, and J. Tanida, “Image reconstruction for thin observation module by bound optics by using the iterative backprojection method,” Appl. Opt. 45, 2893–2900 (2006). [CrossRef] [PubMed]

23. K. Choi and T. J. Schulz, “Signal-processing approaches for image-resolution restoration for TOMBO imagery,” Appl. Opt. 47, B104–B116 (2008). [CrossRef] [PubMed]

24. M. Shankar, R. Willett, N. Pitsianis, T. Schulz, R. Gibbons, R. Te Kolste, J. Carriere, C. Chen, D. Prather, and D. Brady, “Thin infrared imaging systems through multichannel sampling,” Appl. Opt. 47, B1–B10 (2008). [CrossRef] [PubMed]

25. A. V. Kanaev, D. A. Scribner, J. R. Ackerman, and E. F. Fleet, “Analysis and application of multiframe superresolution processing for conventional imaging systems and lenslet arrays,” Appl. Opt. 46, 4320–4328 (2007). [CrossRef] [PubMed]

26. J.-H. Park, Y. Kim, and B. Lee, “Elemental image generation based on integral imaging with enhanced resolution,” Information Optics and Photonics Technology, Photonics Asia, Proc. SPIE5642, Beijing, China, 184–196, Nov. (2004).

27. Y. Wang and Y. Piao, “Super-Resolution Processing of Computational Reconstructed Images,” Proc. Signal Processing (ICSP), 2010 IEEE 10th International Conference1033–1035 (2010).

28. D. Levin, “The approximation power of moving least-squares,” Math. Comp. 67(224), 1517–1531 (1998). [CrossRef]

29. D.-H. Shin, B. Lee, and J.-J. Lee, “Occlusion removal method of partially occluded 3D object using sub-image block matching in computational integral imaging,” Opt. Express 16, 16294–16304 (2008). [CrossRef] [PubMed]

30. G. H. Golub and C.F. Van Loan, Matrix Computations (Johns Hopkins University Press, 1991).

Image	Distance	Method	PSNR	SSIM

Ewha	4	Bilinear	25.9342	0.9041
		Bicubic	26.7843	0.9230
		Proposed	28.3254	0.9508

	8	Bilinear	22.6038	0.8872
		Bicubic	22.3259	0.8992
		Proposed	24.9953	0.9165

Cupcake	7	Bilinear	27.3806	0.9878
		Bicubic	28.5665	0.9904
		Proposed	32.2902	0.9960

	11	Bilinear	24.8591	0.9763
		Bicubic	25.1015	0.9776
		Proposed	27.3308	0.9871

Cup	7	Bilinear	27.3806	0.9632
		Bicubic	27.2791	0.9684
		Proposed	28.4352	0.9751

	11	Bilinear	25.7536	0.9363
		Bicubic	25.7572	0.9531
		Proposed	27.5614	0.9669

Superresolution of 3-D computational integral imaging based on moving least square method

Abstract

1. Introduction

2. Related works

2.1. Computational integral imaging reconstruction

2.2. Superresolution

2.3. Properties of the moving least square method

3. Proposed method

3.1. Overall system diagram

3.2. Proposed ED-MLS based superresolution

3.3. Computational cost

4. Experimental results

5. Conclusion

Acknowledgments

References and links

Cited By

Figures (13)

Tables (1)

Equations (11)

Optics Express