This paper describes a generalized framework for single-exposure acquisition of multi-dimensional scene information using integral imaging system based on compressive sensing. In the proposed system, a multi-dimensional scene containing a plurality of information such as 3D coordinates, spectral and polarimetric data is captured by integral imaging optics. The image sensor uses pixel-wise filtering elements arranged randomly. The multi-dimensional original object is reconstructed using an algorithm with a sparsity constraint. The proposed system is demonstrated with simulations and feasible optical experiments based on synthetic aperture integral imaging using multi-dimensional objects including 3D coordinates, spectral, and polarimetric information.
© 2013 Optical Society of America
Multi-dimensional imaging is challenging because conventional imaging optics project an object onto a two-dimensional detector array. A cooperative design of optics and signal processing, which is called computational imaging, has been used for multi-dimensional imaging. For example, stereo camera and integral imaging have been applied to 3D imaging, including depth acquisition [1–8]. Those systems have multiple cameras or lenses to observe the object from different perspective views and computationally estimate the depth with the parallaxes between the captured elemental images.
Such imaging systems have been extended to acquisition of multi-dimensional objects including spectral, polarization, etc. [9–12]. In those systems, the lateral pixel count of the multidimensional object is the same as that of the image sensor as shown in Fig. 1(a). It means that the number of the elements in the original multi-dimensional object is larger than that of the pixels of the image sensor. In other words, typical multi-dimensional imaging systems with a single-exposure are ill-posed. The computational process is based on rearrangement of the captured pixels and the full-size object cannot be reconstructed in this case. Thus, conventional single-exposure acquisition of the multi-dimensional information compromises the space-bandwidth product of the image sensor.
A framework called compressive sensing (CS) can solve such ill-posed problem with randomized sampling and reconstruction algorithms employing a sparsity constraint as shown in Fig. 1(b) [13–15]. Some generalized multi-dimensional imaging systems with CS have been proposed [16–19]. In this paper, we propose a new modality of multi-dimensional integral imaging to alleviate some drawbacks in the previous works.
2. Proposed multi-dimensional imaging system
The proposed multi-dimensional imaging system has an integral imaging optics and an image sensor with randomly arranged pixel-wise filtering elements as shown in Fig. 2, where x and y show the lateral axes and z shows the longitudinal axis, respectively. It has been inspired by compressive Fresnel holography , where pixel-wise multimodal filtering elements on the image sensor are used for randomized sparse sampling in holographic imaging . The holographic imaging process is replaced by integral imaging to alleviate some stringent requirements for holographic imaging such as the requirements for active illumination (coherent illumination), speckle noise degradation, difficulty in capturing an outdoor scene by coherent light, multiple coherent sources for multispectral illumination, etc. Furthermore speckle noise and random phase in general objects are difficult to be treated by CS with single-exposure imaging because they are not compressive . An advantage of the proposed system over the other previous multi-dimensional imaging systems is the potential to have a compact optical sensor. The previous systems require additional optical elements to be used with the imaging optics [16–18]. On the other hand, the filtering elements of the proposed system can be integrated onto the image sensor. In the proposed system, the integral imaging optics composed of multiple lenses or cameras projects the multi-dimensional object onto the image sensor with randomly arranged pixel-wise filtering elements. A single lens or camera is called elemental optics in this paper. The object is reconstructed with a CS algorithm employing a sparsity constraint.
Firstly an imaging process of a conventional single aperture imaging system without any coding and filtering optics is described. In this paper, normal small letters are continuous variables, normal capital letters are integer variables, and calligraphic capital letters are functions, respectively. y-axis is omitted and ideal pinhole optics and ideal detector sampling are assumed for simplicity. The imaging process is written as
In the proposed scheme, integral imaging and pixel-wise filters realize a depth- and channel-variant impulse response, respectively. The parallax between the captured elemental images and the transmittance of the pixel-wise filters depend on the depth and the channel, respectively. The imaging process in Eq. (1) can be modified for the proposed integral imaging system asTable 1.
The imaging process of our proposed system can be expressed linearly as shown in Eq. (2). The process in the K-th elemental optics expressed by the equation is also rewritten with matrix operators. Here bold lower case letters are column vectors and bold capital letters are matrices, respectively. This process is written asFig. 2, respectively. ℝa×b is an a ×b matrix with real numbers. The matrix H′K can be decomposed as Eq. (4) based on Eq. (2). Here, QK ∈ ℝMX×(MX×NC) is a matrix indicating the filter response in the K-th elemental optics. The matrix QK can be written as Fig. 2. As mentioned above, (MX × LX) << (NX × NZ × NC) is assumed in this paper. It means that this system is ill-posed. Equation (9) indicates that a column in the entire system matrix H has multiple nonzero elements due to the multiple elemental imaging processes with the filtering in Eq. (5) and the downsampling in Eq. (6). Randomness can be introduced by modifying the filtering in Eq. (5) and the translation in Eq. (8). These mean that the system matrix H approximately satisfies certain properties dictated by CS theory .
To solve the inversion of Eq. (10), a CS algorithm called Two-step iterative shrinkage/thresholding (TwIST)  is used for the multi-dimensional object reconstruction. TwIST solves the following problem shown as24] for each Z-th plane of each C-th channel is chosen as the regularizer, which is: Eq. (12), the elements in the original vector f are rearranged with lexicographic order to express the multiple dimensions and f(X, Y, Z, C) denotes the X, Y, Z, C-th rearranged element of the vector f.
3. Simulations of sparse samplings
In the proposed system, signals on each channel are sparsely sampled with an integral imaging sensor as shown in Fig. 2 and Eq. (2). Both regular and irregular sparse samplings in conventional single aperture imaging and integral imaging, which has multiple apertures, are compared by simulations in this section. Figures 3(a)–3(c) show an object image, which is the Shepp-Logan phantom, and regular and irregular sparse sampling patterns, which correspond to the filter response in Eq. (5), respectively. The size of the object and the two patterns are 150 × 150 pixels each. That is, the size of the object is 150×150×1×1 (=NX ×NY ×NZ ×NC) pixels and the size of the captured image is 150 ×150 (=(MX ×LX) ×(MY ×LY)) pixels, respectively. In the regular sparse sampling pattern, the pattern is divided by blocks with 3 ×3 pixels. A same combination with three pixels in these blocks is set as 1 (white) and the other pixels are 0 (black) as shown in Fig. 3(b). In this case, 33.3 % of the whole image are 1. This ratio is called the sampling ratio in this paper. The irregular sparse sampling pattern is a randomized binary distribution, where 33.3 % of pixels are 1 and the others are 0 as shown in Fig. 3(c). That is, 33.3 % of diagonal elements of the filtering sub-matrix Q′C,K in Eq. (5) are 1 and the others are 0 in both of the sampling patterns.
The object was projected onto the image sensor plane by either a single aperture imaging optics or an integral imaging optics. In the case of the single aperture imaging optics, there was a single elemental image, that is, 1×1 (=LX ×LY), with 150×150 (=MX ×MY) pixels. In the integral imaging case, there were 5×5 (=LX ×LY) elemental images with 30 ×30 (=MX ×MY) pixels for each elemental image. The projected signals by the two optics cases were multiplied with the sampling patterns in Figs. 3(b) and 3(c). In the integral imaging case, the sampling patterns were divided into 5 ×5 regions for the multiplication. The resultant signals, which are the captured images, with the two sampling patterns in the two cases are shown in Figs. 4(a)–4(d), respectively. The signal-to-noise ratio (SNR) in the measurements was 40 dB. The reconstruction results with TwIST algorithm are shown in Fig. 5. By comparing Figs. 5(a)–5(d), it is evident that the integral imaging with irregular sparse sampling, whose reconstruction is shown in Fig. 5(d), has a higher reconstruction fidelity than the other cases. The other cases lose many pixels and/or have degradations in their reconstructions. The integral imaging with irregular sparse sampling shows the robustness of the proposed imaging system for under-sampling. The randomness introduced to sparse sampling, that is, the filtering in Eq. (5), has improved the reconstruction performance of integral imaging. The peak SNR (PSNR) was measured for comparison. The PSNR can be calculated asFig. 3(a) and the image reconstructions in Figs. 5(a)–5(d) were 20.6 dB, 19.4 dB, 21.7 dB, and 30.4 dB, respectively. Those PSNRs also indicate the advantage of integral imaging with irregular sparse sampling.
The relationships between the sampling ratios and reconstruction PSNRs of single aperture imaging and integral imaging with regular and irregular sparse samplings are plotted in Fig. 6. Integral imaging with irregular sparse sampling has higher PSNRs than the others when the projected signals by the optics are sampled with low sampling ratios. The lower bound of the sampling ratio in integral imaging with irregular sparse sampling may be around 30 %. As shown in the simulations of this section, the proposed integral imaging can reduce the number of sampling points or detectors on the image sensor. It is also useful in applications of multi-channel data acquisition because each channel may be under-sampled in this case as shown in Fig. 1.
In this section, the concept of the proposed system is also verified by optical experiments based on synthetic aperture integral imaging. It can be directly applied to various types of integral imaging. In the experiments, a color camera was used to obtain elemental images. The focal length of the camera lens is 50 mm, the sensor size is 36 mm ×24 mm, and the pixel count is 1248 ×832, respectively. The camera was scanned along the horizontal and vertical directions by a two-axis translation stage. The number of the captured elemental images in the scan was 6 ×6 (=LX ×LY).
4.1. Spectral integral imaging
In the first experiment, compressive spectral integral imaging with the proposed system is performed. A sign and a car object were located at 270 mm and 330 mm from the sensor. The objects’ elemental images were captured by the translated camera. The pitch of the translation was 5 mm×5 mm. The entire captured elemental data is shown in Fig. 7 and a sample elemental image is shown in Fig. 8, respectively. The size of the captured elemental images was reduced to 175 ×79 (= MX × MY) pixels. The reduced elemental images were multiplied with the filter responses 𝒬C,K in Eq. (2) to emulate randomly arranged pixel-wise color filters. Three single band-pass filters (red, green, and blue) were assumed. One of the three filters was randomly selected on each detector. In this case, 33.3 % of the diagonal elements of each filtering sub-matrix Q′C,K in Eq. (5) are 1 and the other elements are 0. The diagonal elements of the each three sub-matrices Q′C,K in the K-th elemental optics were complementary from each channel. The sampled emulated elemental image of Fig. 8 is shown in Fig. 9.
The reconstructions with a back-projection algorithm  using the bicubic interpolation and TwIST algorithm are shown in Fig. 10(a) and Fig. 10(b), respectively. No sparsity constraints were used in the back-projection reconstruction. The size of the reconstructed object was 1044×480×2×3 (=NX ×NY ×NZ ×NC) pixels. Thus, the compression ratio was 6.0. The reconstruction planes were set at 270 mm and 330 mm from the sensor, respectively. TwIST removed the defocused object in the back-projection reconstruction and enhanced the contrast and lateral resolution of the reconstructed object.
4.2. Spectral and polarimetric integral imaging
In the second experiment, compressive spectral and polarimetric integral imaging is demonstrated. Polarimetric imaging has been used in medical imaging, remote sensing, industrial control, etc. [26–28]. The scene includes two plants at 430 mm, a truck at 550 mm, and a sign at 810 mm from the sensor. The scene was captured by the translated camera without and with a polarizer at a single polarization angle . The pitch of the translation was 10 mm ×10 mm. The entire captured elemental data of intensity images and linearly polarized images are shown in Figs. 11(a) and 11(b), respectively. A sample captured elemental images of both cases are shown in Figs. 12(a) and 12(b). The elemental images were resized to 247 ×151 (= MX × MY) pixels. The resized images were multiplied and integrated with the filter responses 𝒬C,K to emulate randomly arranged pixel-wise three color filters (red, green, and blue) and pixel-wise polarizers. One of the three color filters was randomly selected on each detector, and one or no polarizer was randomly arranged on each of them. A detector with or without the polarizer captures intensity or linearly polarized spectral information, respectively. In this case, 16.7 % of the diagonal elements of the each filtering sub-matrix Q′C,K are 1 and the others are 0. The diagonal elements of the six (three spectrums times two polarizations) sub-matrices Q′C,K in the K-th elemental optics were complementary from each channel. The sampled emulated elemental image is shown in Fig. 13.
The reconstructions with the back-projection and TwIST algorithms are shown in Figs. 14(a) and 14(b), respectively. The size of the reconstructed object was 1000 ×610 ×3 ×(3 ×2) (=NX ×NY ×NZ ×NC) pixels. Thus the compression ratio was 8.2. In each figure, the reconstruction planes are 430 mm, 550 mm, and 810 mm from the sensor. The first and second rows in Figs. 14(a) or 14(b) show the intensity images and the linearly polarized images, respectively. The polarimetric imaging was verified with the reflection on the body of the truck object on the second plane in both of the reconstructions. TwIST algorithm suppressed the defocused objects which exist in the back-projection reconstruction. Also, TwIST enhanced the contrast and lateral resolution of the reconstructed object. In this experiment, the scene has a small occlusion of the sign by the plants as shown in Figs. 11 and 12. The occlusion does not affect the reconstructions and is negligible in this case. However the impact may be large when the occlusions are not small. It can be alleviated based on a method proposed for occlusions in compressive Fresnel holography .
In this paper, we have proposed and demonstrated a multi-dimensional integral imaging system with compressive sensing. We have described a generalized framework for single-exposure acquisition of multi-dimensional scene information using compressive integral imaging. The system is capable of handling multi-dimensional scene containing a plurality of information such as 3D coordinates, spectral and polarimetric data, dynamic range, high speed imaging, etc. In the proposed system, a multi-dimensional object is captured with the integral imaging optics. The projected signals by the imaging optics are filtered by pixel-wise optical elements on the image sensor, and the resultant signals are integrated. The original object was reconstructed with TwIST algorithm using the total variation as the regularizer. Using both simulations and feasible optical experiments based on synthetic aperture integral imaging, we have demonstrated processing of multi-dimensional scene including 3D coordinates, spectral, and polarimetric information acquisition and image reconstruction. The demonstrated concept can be directly applied to integral imaging systems.
The proposed multi-dimensional integral imaging shown in Fig. 2 and Table 1 is realizable with currently available or near future technologies. For example, pixel-wise color filters and polarizers  are commercially produced and CMOS image sensors use a line-wise shutter called rolling shutter. We may be able to integrate these components directly into the system even if some randomness should be added to them. Spatial light modulators (SLMs) are also useful to implement pixel-wise polarizers, neutral density filters, and shutters.
A future study for designing and constructing the proposed system should be investigated. The system models introduced in Section 2 assume an ideal imaging process. The difference between the ideal models and the real physical phenomena degrades the reconstruction performance. These models should be improved with considering more realistic imaging process containing defocusing, aberration, etc. Furthermore, a boundary condition on the reconstruction performance and a reasonable system design, e.g. filter pattern, considering the performance should be studied. As mentioned in Section 2, the proposed system was inspired by compressive Fresnel holography [19, 20]. The condition on the reconstruction performance in compressive Fresnel holography has been investigated recently . This approach may be applied to the future issues of the multi-dimensional integral imaging system proposed in this paper.
The authors wish to thank the anonymous reviewers for their comments and suggestions.
References and links
1. M. Okutomi and T. Kanade, “A multiple-baseline stereo,” IEEE Trans. Pattern Anal. Mach. Intell. 15, 353–363 (1993). [CrossRef]
2. G. M. Lippmann, “La photographie integrale,” Comptes-Rendus Academie des Sciences 146, 446–451 (1908).
3. C. B. Burckhardt, “Optimum parameters and resolution limitation of integral photography,” J. Opt. Soc. Am. 58, 71–74 (1968). [CrossRef]
5. F. Okano, J. Arai, K. Mitani, and M. Okui, “Real-time integral imaging based on extremely high resolution video system,” Proc. IEEE 94, 490–501 (2006). [CrossRef]
6. M. Cho, M. Daneshpanah, I. Moon, and B. Javidi, “Three-dimensional optical sensing and visualization using integral imaging,” Proc. IEEE 99, 556 –575 (2011). [CrossRef]
7. R. Horisaki, S. Irie, Y. Ogura, and J. Tanida, “Three-dimensional information acquisition using a compound imaging system,” Optical Review 14, 347–350 (2007). [CrossRef]
10. R. J. Plemmons, S. Prasad, S. Matthews, M. Mirotznik, R. Barnard, B. Gray, V. P. Pauca, T. C. Torgersen, J. van der Gracht, and G. Behrmann, “PERIODIC: Integrated computational array imaging technology,” in “Computational Optical Sensing and Imaging,” (2007), p. CMA1.
12. R. Horstmeyer, G. Euliss, R. Athale, and M. Levoy, “Flexible multimodal camera using a light field architecture,” in “Proc. ICCP09 ,” (2009), pp. 1–8.
13. D. L. Donoho, “Compressed sensing,” IEEE Trans. Info. Theory 52, 1289–1306 (2006). [CrossRef]
14. R. Baraniuk, “Compressive sensing,” IEEE Sig. Processing Mag. 24, 118–121 (2007). [CrossRef]
15. E. J. Candes and M. B. Wakin, “An introduction to compressive sampling,” IEEE Sig. Processing Mag. 25, 21–30 (2008). [CrossRef]
16. R. Horisaki, K. Choi, J. Hahn, J. Tanida, and D. J. Brady, “Generalized sampling using a compound-eye imaging system for multi-dimensional object acquisition,” Opt. Express 18, 19367–19378 (2010). [CrossRef] [PubMed]
18. R. Horisaki and J. Tanida, “Multidimensional TOMBO imaging and its applications,” in Proc. SPIE (2011), 8165, pp. 816516. [CrossRef]
20. Y. Rivenson, A. Stern, and B. Javidi, “Compressive Fresnel holography,” J. Display Technol. 6, 506–509 (2010). [CrossRef]
22. E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Info. Theory 52, 489–509 (2006). [CrossRef]
23. J. M. Bioucas-Dias and M. A. T. Figueiredo, “A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Proc. 16, 2992–3004 (2007). [CrossRef]
24. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60, 259–268 (1992). [CrossRef]
31. T. Sato, T. Araki, Y. Sasaki, T. Tsuru, T. Tadokoro, and S. Kawakami, “Compact ellipsometer employing a static polarimeter module with arrayed polarizer and wave-plate elements,” Appl. Opt. 46, 4963–4967 (2007). [CrossRef] [PubMed]