New imaging modality through deep compressed sensing enables high speed imaging
Schematics of DeCIOPS (deep compressed imaging via optimized pattern scanning). The object is sampled block by block by an illumination pattern. ISTA-Net, a compressed-sensing-inspired deep neural network is used to reconstruct the object. The illumination pattern and the parameters in ISTA-Net are optimized through an end-to-end trained auto-encoder.
Optical imaging typically relies on a camera with millions of photodetectors. By contrast, imaging can also take place with a single detector. This is particularly advantageous for imaging at non-visible spectrum where the conventional pixelated cameras lose the sensitivity or become very costly for a good performance.
A most straightforward way to image through a single detector is to scan the object and measure it pixel by pixel. Yet, another interesting approach, commonly known as single-pixel camera, is to leverage compressed sensing. Instead of being sampled pixel by pixel, the object is sampled through the sum of different combinations of pixels. Typically, the object is illuminated with a pattern, and the detector captures the sum intensity of the object pixels illuminated by the pattern. By performing the measurement through different patterns, the entire object can be reconstructed.
In both the conventional camera and the point-scanning-based system, the actual number of measurements is the same as the total number of pixels, though the former measures all pixels at once through different detectors and the latter measures individual pixel sequentially through a single detector. However, the number of measurements, i.e. the number of illumination patterns, can be much smaller than the number of pixels (for example 10~100 times smaller) in compressed sensing. The computation algorithm can then recover the object in high quality through a small number of measurements, by exploiting the natural relationship between adjacent pixels. As a result, compressed sensing has a very high measurement efficiency. The fewer measurements make it promising to reduce the data acquisition time and thus increase the imaging speed.
However, the existing implementation of compressed sensing in imaging relies on switching different illumination masks, and the limited speed in the switching device (digital mirror device) restraints the overall speed. In many scenarios, the imaging speed of the switching-mask-based single-pixel camera is actually lower than that based on point scanning or conventional cameras.
The research group led by Prof. Weijian Yang from University of California, Davis recently demonstrated a new compressed imaging modality, termed "deep compressed imaging via optimized pattern scanning" (DeCIOPS), which could overcome the challenges in the existing single-pixel cameras, and greatly improve the imaging speed of compressed imaging. The research results are published in Photonics Research, Vol. 9, No. 3, 2021 (Kangning Zhang, Junjie Hu, Weijian Yang. Deep compressed imaging via optimized pattern scanning[J]. Photonics Research, 2021, 9(3): 03000B57 ).
In this new imaging modality (DeCIOPS), a block of pixels, weighted by an illumination pattern, are sampled and summed into a single measurement. By scanning the illumination pattern across the object, different blocks of pixels could be measured. A reconstruction algorithm based on a compressed-sensing-inspired deep neural network was developed to recover the high resolution image from this seemingly low-resolution block-by-block measurement. Since the number of measurement blocks can be much smaller than the number of total pixels, the measurement speed can be very high.
In addition, the researchers modeled DeCIOPS through an auto-encoder, which was trained in an end-to-end manner and jointly optimized the illumination pattern for image acquisition and the deep neural network for object reconstruction. The end-to-end optimization ensures an overall optimal performance of the entire imaging system and thus a high quality in object reconstruction.
Essentially, DeCIOPS synthesizes the strengths of the high acquisition speed of point scanning and the high measurement efficiency of compressed sensing. Compared to the point scanning system, DeCIOPS increases the imaging speed (by a few folds to one order of magnitude) through a highly efficient sampling scheme. Compared to the conventional compressed imaging which requires switching between many illumination patterns, DeCIOPS scans a single illumination pattern across the object, and could increase the overall acquisition speed by orders of magnitude.
"DeCIOPS represents a nice example in computational imaging with a tight integration of the imaging system and the computation algorithm," said Prof. Weijian Yang. He believes that the high imaging speed offered in DeCIOPS makes it very promising for applications in biomedicine, surveillance, and consumer electronics.
但是，利用压缩感测的单像素相机需要使用DMD（Digital Micromirror Device）去切换不同的照明图案，而这个切换速度限制了整体的成像速度。从而导致在许多情况下，这种单像素相机的成像速度低于基于点扫描或常规相机的成像速度。
针对上述研究现状，美国加州大学戴维斯分校的杨暐健教授课题组在Photonics Research 2021年第3期（Kangning Zhang, Junjie Hu, Weijian Yang. Deep compressed imaging via optimized pattern scanning[J]. Photonics Research, 2021, 9(3): 03000B57 ）发表的文章中展示了一种新的压缩成像方式（DeCIOPS）。该成像方式通过在物体上投影并扫描一个被经过优化的图案，再进行深度压缩成像，可以解决现有单像素相机中遇到的困难，大大提高压缩成像的成像速度。