The Computational Optical Sensing and Imaging (COSI) conference has a long history, being held over the past 15 years. One of its original aims has been to develop and promote integrated optical, electronic, and algorithmic designs of optical systems. After years of development, this field has been expanded from topics such as wavefront sensing and compressive coding to much broader scope, including computational holography, computational microscopy, compressive imaging, NLOS imaging, phase imaging, and novel applications with new types of sensors. The 2021 COSI conference was held virtually from July 19 to 23. The virtual delivery allowed the meeting to take place in spite of on-going travel and sanitary restrictions. Fourteen invited talks and over 120 contributed papers were presented at the event. In addition to the annual conferences, COSI organizers have published several feature issues in the Applied Optics and Optics Express journals over the years, the most recent one being in Optics Express in 2020. After a two-year break, due to the COVID 19 pandemic, the conference now has, for the first time, a joint feature issue with Optics Express and Applied Optics. We are excited to publish 36 papers in this feature issue, highlighting the latest works in this field.

Among the 36 papers in this feature issue, there are two invited reviews devoted to deep learning on holography [1] and ptychography [2]. Another invited research article focuses on an angle measuring system [3]. The contributed papers [4–36] cover image systems engineering, algorithms, devices, and mathematical models mainly in the following research areas: computational microscopy, non-line-of-sight imaging(NLOS), compressive imaging, ghost imaging/single-pixel imaging, imaging through scattering media, phase retrieval, and 3D imaging. In multiple areas, the use of deep learning to solve challenging inverse problems has become widely adopted.

In the following we summarize the papers in this issue. The invited reviews and articles are presented first. Then we present the contributed papers based on the research topics ordered as appeared on the conference website. These papers can be found in the feature issue website in Optics Express (https://opg.optica.org/oe/virtual_issue.cfm?vid=503) and the 9th issue of 2022 in Applied Optics.

1. Invited reviews and articles

Holographic imaging has important applications in microscopy, metrology, and other domains. Deep learning–based reconstruction techniques have recently emerged as versatile and powerful tools for digital holography across these application areas. The review paper [1] surveys recent deep learning approaches in this area and introduces a taxonomy that makes this topic accessible to the novice while providing comprehensive and in-depth coverage of the field.

Ptychography is a well-known computational lensless imaging technique, however its application to extreme ultraviolet microscopy and wavefront sensing is relatively new. The invited review [2] covers the state of the art in extreme ultraviolet (XUV) ptychography with tabletop high-harmonic generation sources. Different hardware options including illumination optics and detector concepts as well as algorithmic aspects in the analysis of multispectral ptychography data are considered. Cutting-edge technological applications of XUV ptychography such as multispectral wavefront sensing, attosecond pulse characterization, and depth-resolved imaging, and the perspectives for its development are discussed. This comprehensive review is an excellent guide for researchers working in the area as well as for graduate students.

The invited research article in this feature issue is on a high-performance, optoelectronic angle measuring system capable of high accuracy and high precision angle measurements [3]. The proposed system uses a diversity of diffractive and refractive optical elements, combined with custom analog and digital electronics, to achieve six degrees of freedom localization and angular velocity estimation.

2. Computational microscopy (including ptychography), digital holographic microscopy

In [4], the authors demonstrate in simulation and experiment a multi-lens microscopic imaging system that overlaps multiple independent fields of view on a single sensor for automated specimen analysis based on CNN.

In [5], the authors describe a computational light-sheet microscope for the hyperspectral acquisition at high spectral resolution. The method uses a structured light pattern to acquire and reconstruct the spatial dimension orthogonal to the slit of the used spectrometer. The paper demonstrates the feasibility of the method and reports the first in vivo results for hydra specimens labeled using two fluorophores.

In [6], the authors present a computational method for fast single-photon counting of directly sampled time-domain fluorescence lifetime imaging microscopy data. It is capable of accurate fluorescence lifetime and intensity measurements while acquiring over 160 Mega-counts-per-second with sub-nanosecond time resolution between consecutive photon counts.

In [7], the authors propose and experimentally verifies a method to improve the image quality of the synthetic aperture imaging interferometric microscopy system based on analysis of cross-correlation between two consecutive sub-images in the overlapping regions.

In [8], the authors report on a simple, low cost method for quantitative phase imaging based on in-line Gabor holography realizable in a standard bright-field microscope with coherent sensing capabilities.

In [9], the authors present an approach to enhance cryo-electron microscopy (cryo-EM) postprocessed maps based on a multiscale tubular filter. The method locally determines the tubularness measure and uses this information to enhance elongated local structures and to attenuate blob-like and plate-like structures, which contributes to an improvement of the obtained reconstructions.

3. Compressed sensing

In [10], the authors develop a neural network-based reconstruction method for spatial compressive imaging. The method incorporates the sensing matrix information in the form of degraded maps as input to the network, resulting in high-quality and fast reconstructions at low compression rates.

In [11], the authors propose a hardware modification to the rolling shutter (RS) mechanism found in CMOS detectors by shuffling the pixels in every scanline. With the improved sampling of the space-time datacube and sophisticated reconstruction methods including a neural network-based method, the proposed shuffled RS approach provides an alternative for snapshot temporal imaging, if ever implemented in hardware.

4. Imaging through scattering and turbid media

In [12], the authors use a phase plate model to analyze the morphology and statistics of speckles produced by a point-like source with wide spectrum.

In [13], the authors develop a wavefront shaping method based on adaptive stochastic parallel gradient descent optimization with the Hadamard basis to focus light through scattering soil samples.

In [14], the authors combine speckle-correlation pre-processing with neural network-based reconstruction to achieve color object recovery through unknown opaque scattering layers by training with only one diffuser.

5. Lensless imaging, coherent diffraction imaging

In [15], the authors develop a new reconstruction method for lensless inline holography. This method can produce high-resolution images of the amplitude and phase of a thin sample over a large field of view using aliased intensity measurements taken at a lower resolution.

In [16] the authors report a maximum likelihood estimation based framework for holographic coherent diffraction imaging that provides improved image reconstruction results for various practical settings, e.g., missing low frequency data due to occlusion from a beamstop apparatus, or data that is highly corrupted by Poisson shot noise. The developed mathematical framework is also applicable beyond holographic coherent diffraction imaging.

6. Machine learning for computational imaging

Machine learning approach has been very popular in almost all imaging problems. In this feature issue there are 8 papers using neural networks for multiple problems. Besides the papers [1,10,11,14,20,25] which have been or will be introduced under more specific topics, the following are 2 more works.

In [17], the authors use a simple fiber-optic tip sensor to identify liquid using a neural network. The fiber-droplet and the droplet-air interfaces work together as a EFPI with a liquid cavity. As the droplet evaporates, the length of the cavity reduces. Thus, using a probing light source, the evaporation event can be captured. A CNN network is used for the classification task.

In [18], the authors propose a backpropagation neural-network-based method for improving the measurement accuracy of four-quadrant detectors with a small amount of real data.

7. Novel applications of holography

In [19], the authors explore a new approach to overlay metrology for semiconductor chip alignment. For this purpose, they develop a compact dark-field digital holographic microscope that uses only a single imaging lens. Aberrations from this optical systems are calibrated and corrected using nano-sized point scatterers on a silicon substrate together with computational wavefront imaging techniques.

8. Phase retrieval and its applications

In [20], the authors demonstrate the use of deep neural networks in combination with coded diffraction patterns to solve the phase retrieval inverse problem efficiently and with surprising accuracy.

In [21], the authors develop a general frequency-shifting technique to pixelwise retrieve the absolute phase in fringe projection profilometry without any phase unwrapping.

In [22], the authors propose and demonstrate a computational method capable of performing single-shot phase retrieval, achieving pixel-level super-resolution in a compact optical system.

9. Polarization imaging and sensing

In [23] the authors integrate a microscope objective and a lenslet array with a polarization mask, then use this optical element as a replacement of the conventional lens in a camera for polarimetric microscopy.

10. Topics in 3D imaging (structured illumination, ToF Sensing, LiDAR, light fields)

In [24], the authors propose a novel method for the center detection of laser lines in line triangulation setups. The method exploits the different spectral filters provided by the camera’s Bayer pattern to synthesize a high dynamic range image of the projected laser line. This leads to an improvement in line center detection which yields an improved 3D surface reconstruction.

In [25], the authors describe the use of neural networks to perform decomposition of LiDAR signal waveforms, resulting in significant computational efficiency gains. Different network versions are proposed and analyzed, the applicability of each version depends on the SNR of the input LiDAR signals.

In [26], the authors propose a method capable of correcting phase and amplitude errors caused by mirror movement uncertainties in mechanically scanned structured illumination imaging.

11. Unconventional imaging modalities (intensity interferometry, ghost imaging and mutual intensity imaging)

In [27], the authors use simulation to demonstrate a method for three-dimensional ghost imaging that integrates the differential-correlation-sampling technique and a modulated continuous-wave laser source and allows suppressing the effect of dynamic ambient light and electrical noise.

In [28], the authors propose a forgery attacking algorithm for grayscale single-pixel intensity values in computational ghost imaging. Then the generalized forgery attack scheme is also discussed for the well-known double random phase encoding system in Fourier domain and Fresnel domain.

In [29], the authors introduce arbitrary-order fractional derivative operations into single-pixel imaging that offers a good tradeoff between image SNR and performance of edge enhancement.

In [30], the authors propose an image-free target tracking scheme based on the discrete cosine transform and single-pixel detection. The tracking speed can reach 208 fps at a spatial resolution of 128 $\times$ 128 pixels with a tracking error of no more than one pixel.

12. Computer generated holography and computational displays for AR/VR

In [31], the authors present an alternative algorithm for single-shot color hologram generation, which offers more flexibility when applied to imaging with holographic displays.

13. Event-driven computational imaging using neuromorphic sensors

In [32], the authors present a novel approach in the analysis of motion using speckle imaging, demonstrating that event-driven cameras can be effectively used for that purpose in a lens-free optoelectronic system.

14. Non-line-of-sight imaging

In [33], the authors study the problem of white light–illuminated NLOS imaging. The authors incorporate speckle correlation-based model into a deep neural network, and use a two-step strategy to learn the optimization of the scattered pattern autocorrelation and then object image reconstruction.

In [34], the authors propose physical and virtual NLOS image acquisition systems for capturing simulated and realistic images under various ambient light conditions which are used further to train/fine-tune a multi-task CNN to perform simultaneous background illumination correction and NLOS object localization.

In [35], the authors propose a new approach to localizing 3D objects outside the line of sight of a camera. Their method operates in the thermal infrared wavelength range and it works in a passive manner, i.e., it does not require active light sources. The authors demonstrate experimental results of their non-line-of-sight 3D localization approach.

15. Optical computing

In [36], the authors develop an optical computing engine for performing fractional Fourier transform (FRFT) on non-stationary temporal signals, which can replace the computer-based FRFT computation in a swept-source optical coherence tomography system.

The above are our summary of all papers in this feature issue. We hope readers enjoy these exciting works in this field. We also would like to thank the Editor in Chief James Leger, Gisele Bennett and Senior Deputy Editor Chris Dainty for the opportunity to organize this feature issue. We thank Ms. Carmelita Washington, Ms. Nicole Williams-Jones, Ms. Kelly Cohen, and Ms. Rebecca Robinson from the OSA Manuscript Office for their assistance through the manuscript review process. In the end, we would like to announce that, the next Computational Optical Sensing and Imaging conference will take place in Vancouver, British Columbia, Canada July 11–15, 2022. It will be a hybrid meeting to accommodate both in-person and online participation.