Open Access
Add to BibSonomyAbstract
Color-imaging methods with an integrated compound imaging system called TOMBO (Thin observation module by bound optics) are presented. The TOMBO is a compact optoelectronic imaging system for image capturing based on compound-eye imaging and postdigital processing. First, a general description of the TOMBO system is given, and then two configurations for color imaging are described. Experimental comparison of these configurations is made by use of an experimental TOMBO system. The characteristics and the performance on the proposed methods are briefly discussed.
©2003 Optical Society of America
1. Introduction
In Japan, the tremendous spread of cell phones equipped with imaging devices has revealed potential markets for compact imaging devices. Compactness of the imaging devices provides the various merits of portability, dense functional integration, etc. For the implementation, optoelectronic integration is an important factor. However, the size of a typical imaging device, i.e., a lens, is more than several millimeters, and the working distance also requires the same order, whereas that of an elemental electronics device on a semiconductor chip occupies less than several micrometers. This fact suggests that treatment of the imaging lens is a key issue in hardware compaction of the imaging system.
The authors have proposed an optoelectronic imaging system called TOMBO (Thin observation module by bound optics) [1, 2]. The TOMBO is based on compound-eye image capturing and postdigital processing. Use of compound-eye imaging is a relatively common method to reduce the working distance of an imaging system [3, 4, 5, 6]. In addition, we use digital processing to compensate signal degradation caused by reduced aperture of the individual lenses. Combining the specific hardware structure and the flexible processing, we achieve a thin configuration of an image-capturing device with excellent imaging performance.
In this paper, we propose two methods for color imaging with the TOMBO system. After a brief explanation of the principles, we provide experimental results to compare the capability of the proposed methods, and then finally, we discuss issues that relate to the proposed methods.
2. TOMBO System
Figure 1 shows the hardware configuration of the TOMBO system. The TOMBO consists of a microlens array, a signal separator, and an imaging device. An elemental imaging system on a microlens is called a unit, and the captured image is referred to as the unit image. The signal separator prevents cross talk between the adjacent units without increase of hardware thickness. The imaging device detects a set of unit images collectively. A set of unit images is called a compound image. The TOMBO is characterized by the number of the unit, µ, and that of the photodetector cells per unit, ν. For an imaging device with N pixels, N=µν is satisfied. Note that a typical TOMBO system is quite suitable for semiconductor integration.
Processing executed in the TOMBO is performed to retrieve a high-resolution image from a set of low-resolution ones captured as a compound image. For this purpose we developed and utilized an image reconstruction procedure called the pixel rearrange method for our system [7, 8]. Figure 2 describes the principle of this method. In this processing, the pixels in all the unit images are mapped onto a virtual image plane with estimation of the registration parameters.
The procedure is as follows: Shading effect and spatial variation of sensitivity on the captured compound image are corrected. Unit images are extracted from the corrected compound image. The registration parameters describing the relation between the individual unit images and the target object are estimated based on a presumed model of the TOMBO system. For example, the correlation between unit images or the maximum likelihood estimation can be used for the parameter estimation. With the estimated registration parameters, the pixels of all the unit images are mapped onto a virtual image plane. Interpolation is applied to the blank pixels on the virtual image. Optionally, image filtering is adopted to compensate degradation by the point-spread function of the imaging system.
3. Color Imaging
Color imaging is one indispensable requirement for current imaging devices. In the TOMBO system, two implementation methods are considered. One is color separation by pixels, and the other is color separation by units. The former method is to assign the red, green, and blue channels to the individual pixels, and the latter is to assign these color channels to the individual units, as shown in Fig. 3. A major advantage of color separation by pixels is applicability of a commercial color CCD for the imaging device. On the other hand, the color separation by units contributes one, to easy fabrication of color filters attached to the imaging device and two, to relaxed imaging condition.
As an effect of separation of the color channels, distribution of the observable points of each color channel are changed according to the object distance. Figure 4 illustrates the observation direction of the individual pixels separated into different color channels. As seen from the figure, the method of the color separation by units can capture the color information uniformly at plane A, whereas that of the color separation by pixels observes coagulated color information for a distant object. For the color separation by pixels, plane B, whose distance is 1/2 that shown in plane A, is suitable for color imaging. In this configuration, some points can be observed by all the color channels, which suggests extension to multispectral imaging.
Fig. 3. Two methods for color imaging for µ=4 and ν=8. For the color channels, green1, green2, red, and blue are assumed.
Fig. 4. Observation direction of individual pixels. Color separation (a) by units and (b) by pixels.
4. Experimental Results
The principle of color imaging is verified by an experimental TOMBO system, as shown in Fig. 5. For the imaging device, a CMOS (complementary metal oxide semiconductor) image sensor consisting of 500×500 pixels whose size is 10 µm×10 µm is used. For the signal separator, square windows are perforated on a stainless-steel plate whose thickness is 50 µm, and 21 of the plates are cumulated. Each microlens has a 500-µm aperture and a 1.3-mm focal length. The entire system is composed of 10×10 units, and 45×45 pixels detect the image signals for each unit. To emulate both configurations of the color imaging, red, green, and blue filters (Fujifilm, BPB60, BPB53, BPB45) are alternatively set in front of the microlens array. The captured signals are converted into three images of the red, green, and blue channels, with consideration of the correspondence between the pixel address and the filter arrangement, as shown in Fig. 3. The object is a color transparency illuminated from the back side.
Figure 6 is the experimental result of color imaging with the color separation by units. For this case, the object is set at the distance of 52 cm where 8×8 units can capture the color information uniformly. Figure 6(a) is the captured compound image, and Fig. 6(b) shows some of the magnified unit images. To illustrate the color information, we attached pseudocolor information onto these images. The reconstructed image is obtained by rearrangement and interpolation of the pixel data as shown in Fig. 6(c). The pixel number of the reconstructed image is set as 360×360, which is the same as that of the compound image.
Figure 7 shows the result of color imaging with the color separation by pixels. As shown in Fig. 4, the object distance is set as 26 cm, half that of the former case. To keep the same image size on the photodetector array, we shrunk the object by one half. Figure 7(a) is the captured compound image, and Fig. 7(b) shows some of the magnified unit images. The reconstructed image is obtained by rearrangement of the pixel data, as shown in Fig. 7(c). Since a set of four pixels (green1, red, green2, and blue) are assigned to one color pixel, the pixel numberof the reconstructed image is 180×180. Although the pixel number is reduced one half, as compared with the method of the color separation by units, no interpolation process is required in this case. As a result, exact color distribution of the observing points can be captured in this configuration.
5. Discussion
One of the important features of the TOMBO system is the functional extendability provided by configuration flexibility. Color-imaging methods presented here are instances of such functional extension. To clarify the difference between two methods proposed in this paper, we captured a color chart using both methods. Figure 8 shows the experimental results; Fig. 8(a) is the target object; and Figs. 8(b) and 8(c) are the reconstructed images by the method of the color separation by units and that of the color separation by pixels, respectively. The object is located at the distance of 52 cm for the case of the color separation by units and at the distance of 26 cm for the case of the color separation by pixels. Those distances satisfy the condition that 8×8 units can capture the color information uniformly.
As seen from the results, the angle of view is quite different in both cases. This difference suggests a method for changing the angle of view with the same device specifications. In general, the method of the color separation by units is more suitable for normal color imaging because of the long observation distance and the easy fabrication of color filters. On the other hand, the color separation by pixels is suitable for precise spectral measurement at the observation points. When the distance is set as 1/2n(n=1,2, …) of that of the color separation by units, it is possible to observe the same point with (n+1)2 channels. Based on this idea, one can construct a compact multispectral imaging camera.
Dependence of the observable points on the object distance is a characteristic property of the TOMBO system. As seen from Fig. 4, the best observation plane where the color information is uniformly observed is determined implicitly. To capture an object located at the other plane requires interpolation of pixel data. Although such a process degrades image quality, our experiments show that the interpolation is quite effective for relaxing the imaging condition and that the degradation is not serious.
Fig. 6. Experimental result of color imaging with the color separation by units. (a) Compound image, (b) unit images, and (c) reconstructed image.
Fig. 7. Experimental result of color imaging with the color separation by pixels. (a) Compound image, (b) unit images, (c) reconstructed image.
Fig. 8. Experimental result of color imaging for a color chart: (a) target object, (b) reconstructed image by the method of the color separation by units, (c) reconstructed image by the method of the color separation by pixels.
Generally, with the TOMBO system, the issue of image quality improvement is important. Image blur is a characteristic degradation that appears in the TOMBO system, as shown in Fig. 8. To analyze the retrieved images, we calculated the signal-to-noise ratio (SNR) for divided parts of the image region. The image region with 180×180 pixels is trimmed and divided into 9 parts. The SNR is defined as follows:
where Ei is the squared errors of channel i, Smax is the maximum value of the signal, and N is the pixel number in the part. For the current case, i is any one of the red, green, or blue channels; Smax is 255; and N is 3600. Table 1 summarizes the SNR, which is expressed as percentages. As seen from the table, both methods show similar performance, but approximately 10% of the error signal is overlapped owing to image blur.
Table 1. SNR for the Nine Parts of the Reconstructed Images *
Major sources of the remaining noise on the reconstructed images are considered aberrations of the microlens and misalignment of the system components. Although the minimum resolution of the microlens used in the experimental TOMBO system is approximately 3 µm on the optical axis, the value becomes several tens of micrometers on the marginal region. According to the conventional usage of microlens, microlens function is enough to collect the optical power. However, the TOMBO system requires severe imaging performance of the microlens, which seems a huge challenge in microlens development. System alignment is another issue in the TOMBO system. It is known that various misalignments cause gridlike noise on the reconstructed images. Digital processing can compensate misalignment in the system construction, but precise adjustment greatly reduces the amount of calculation.
6. Conclusion
As a functional extension of the TOMBO system, two methods for color imaging have been presented. After the principles were explained, experimental results by the TOMBO system were shown. Features of the two methods are compared and characterized. For the future research, important issues such as microlens improvement and system adjustment should be considered.
Acknowledgments
This research was supported by the Development of Basic Tera Optical Information Technologies, Osaka Prefecture Collaboration of Regional Entities for the Advancement of Technology Excellence (CREATE) and is currently supported by the Ultra-Thin Image Capturing Module in Regional Science Promotion Program (RSP), Japan Science and Technology Corporation.
References and links
1. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. 40, 1806–1813 (2001). [CrossRef]
2. J. Tanida, Y. Kitamura, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N. Kondou, D Miyazaki, and Y. Ichioka, “Compact image capturing system based on compound imaging and digital reconstruction,” in Micro- and Nano-optics for Optical Interconnection and Information Processing, Proc. SPIE4455, 34–41 (2001). [CrossRef]
3. S. Ogata, J. Ishida, and T. Sasano, “Optical sensor array in an artificial compound eye,” Opt. Eng. 33, 3649–3655 (1994). [CrossRef]
4. J. S. Sanders and C. E. Halford, “Design and analysis of apposition compound eye optical sensors,” Opt. Eng. 34, 222–235 (1995). [CrossRef]
5. K. Hamanaka and H. Koshi, “An artificial compound eye using a microlens array and its application to scale invariant processing,” Opt. Rev. 3, 264–268 (1996). [CrossRef]
6. B. A. Wandell, A. El Gamal, and B. Girod, “Common principles of image acquisition systems and biological vision,” Proc. IEEE 90, 5–17 (2002). [CrossRef]
7. J. Tanida, R. Shogenji, K. Yamada, M. Miyamoto, and S. Miyatake, “Functional extension of thin observation module by bound optics (TOMBO),” in Optics in Computing, OSA Technical Digest (Optical Society of America, Washington DC, 2003), pp.152–154.
8. Y. Kitamura, R. Shogenji, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N. Kondou, D. Miyazaki, J. Tanida, and Y. Ichioka, “Reconstruction of high-resolution image on a compound-eye image capturing system,” Appl. Opt. (submitted).
