1. INTRODUCTION

In the human visual system, the wavelength and intensity of light are perceived as color and brightness, respectively. As with the human eye, conventional image sensors sample the intensity of light reflected from objects in a scene. The human visual system and conventional image sensors, however, lack the ability to perceive one fundamental property of light: its polarization. The polarization of light reflected from a scene contains information that can be useful for different applications. This is because the state of polarization of the reflected light is not necessarily correlated with its intensity and spectral composition. Polarization imaging has proven useful for applications such as material identification [1], contrast enhancement in images [2], and detection of nonmelanoma skin cancers [3].

Polarization imagers convert the polarization information of light into intensity values, which can be sampled using photodetector arrays. The intensity of light transmitted through different polarization filters can then be used to calculate the Stokes parameters, which describe the total intensity, degree of polarization, and angle of polarization of the light. The first three Stokes parameters are sufficient for determining the angle and degree of linear polarization. Because linearly polarized light is most common in nature, the majority of polarization imagers are often limited to the determination of the first three Stokes parameters. These parameters can be calculated, for instance, by sampling the light intensity transmitted through four different polarization filters (e.g., 0, 45, 90, and 135 deg filters). Assuming $I_0$, $I_{45}$, $I_{90}$, and $I_{135}$, are the light intensities transmitted through these four filters, the first three Stokes parameters can be calculated using Eqs. (1)–(3):

The polarized light intensity information can be sampled in diverse ways. A review of some of these methods is presented in [4]. The two prevalent architectures for sampling polarization intensity are “division-of-time” and “division-of-focal-plane” sensors. In division-of-time polarization imagers, a CMOS or CCD image sensor is coupled with a polarization filter designed such that it can be mechanically, electrically, or acousto-optically modulated to change the light transmission axis. This allows different polarization intensities to be captured from frame to frame. The drawback of these sensors is that the frame rate is limited, and motion during imaging causes image blur and distortion. Division-of-focal-plane imagers, on the other hand, use a polarization filter array over the pixels such that each pixel samples the intensity of light transmitted through one polarization filter. In a similar manner to the interpolation done in color images, the missing polarization intensities at each pixel location can be recovered from neighboring pixels.

Regardless of the architecture used, polarization imagers typically feature a combination of a conventional CMOS or CCD sensor and a polarization filter unit. The polarization filter may be fabricated separately and then bonded to the image sensor chip. Alternatively, the filter and image sensor may be fabricated monolithically if the filter can be formed using the standard CMOS process. The performance of linear polarization imagers has been greatly improved largely due to improvement in polarizer performance. Monolithic integration of the sensor and polarization filter array also has allowed higher-resolution sensors with good filter–pixel alignment making several applications possible. However, as polarization imaging develops into a mainstream imaging modality for different applications, a number of challenges and obstacles still need to be addressed.

One fundamental handicap of conventional polarization imagers is their inability to efficiently and simultaneously capture color and polarization images. A common approach for obtaining color and polarization information involves the modulation of the color filter over an array of polarization-sensitive pixels, to sequentially capture different color frames in addition to the polarization intensities [5]. This method is limited by the need for electrical or mechanical modulation of the filters. Another method, disclosed in [6], involves stacking a polarization filter array over a color filter array. In this method, color frames and near-infrared polarization frames are sequentially captured by modulating the light reaching the focal plane array. A more recent innovation presented in [7] involves the use of three vertically stacked p–n junctions at different depths in silicon to sample the short, medium, and long wavelengths transmitted through the polarization filter. Because a silicon coefficient decreases with wavelength, shorter wavelengths will be absorbed more in the top p–n junctions, while longer wavelengths will dominate the signal in the bottom p–n junction. This allows full color reproduction, but interpolation is required to obtain missing polarization intensities at each pixel location. The complexity of the readout circuitry as well as poor color separation of the three junctions reduces the appeal of this method.

Another challenge that polarization imagers have to contend with is the trend of decreasing pixel sizes in CMOS image sensors. The decrease in pixel sizes is largely motivated by the goal of increasing resolution. However, this trend has reached a point where pixel sizes are smaller than the diffraction limit of the sensor optical system. For these diffraction-limited systems, the smallest resolvable point can be represented by the diameter of the Airy disc pattern, which is dependent on the light wavelength, $\lambda$, and the $f$-number, $F$, of the optical system [8]:

In this paper, we propose the use of the Quanta image sensor (QIS) concept for polarization imaging to overcome the limitations of current polarization sensors, which employ conventional CMOS image sensors. The QIS concept was proposed to mitigate the issues of low full-well and dynamic range in subdiffraction limit (SDL) pixels [8,9]. In this concept, the image is greatly oversampled in the spatial and temporal domains using specialized SDL pixels known as jots. The extremely small size of the jots (∼a few hundred nanometers) facilitates spatial oversampling, while a fast readout speed ($>1000\text{fields}/\mathrm{s}$) allows temporal oversampling of the image. Figure 1 shows the level of oversampling in a 1 μm pixel compared with a 250 nm jot when a lens (F/2.8) is used to focus the light. The number of pixels/jots enclosed in the 3.7 μm diameter Airy disc gives an indication of the level of oversampling in the image.

 

Fig. 1. Airy disc drawn over (a) 1 μm pixels and (b) 250 nm pixels, showing level of oversampling. Incident illumination has wavelength 550 nm, and a $f/2.8$ lens is used.

Download Full Size | PDF

The jots are designed such that they have an extremely low noise floor, which allows the voltage change due to a single photoelectron to be resolved. A preliminary iteration of such a jot reported in [10] achieved a read noise level less than a third of the signal generated by a single electron. This jot demonstrated single photoelectron sensitivity at room temperature without avalanche multiplication. With single photon sensitivity, these jots may produce a single-bit output upon detection of a single photoelectron or multibit output corresponding to the integration of a few photoelectrons. A pixel in the final image is formed from the outputs of a spatiotemporal kernel/cube of jots in the QIS device. High full-well can be attained by summing more jots to produce a pixel. The low noise floor and high full-well capability equate to a high dynamic range. This imaging concept also significantly increases the range of post-capture processing that can be performed on the image because the spatial and temporal information from incident photons is captured. For instance, motion blur can be minimized by shifting frames in the temporal domain. The pixel sizes in the final image can also be dynamically changed by varying the size of the kernel of jots used in forming pixels [9].

2. POLARIZATION IMAGING USING THE QIS CONCEPT

The application of this concept to polarization imaging is a necessary next step as pixel sizes continue to decrease. One important benefit of polarization information is to provide enhanced contrast information about scenes in low light conditions where intensity contrast is extremely low. However, with the level of light attenuation caused by polarization filters, conventional polarization imagers lack the sensitivity to provide such contrast information in low light. The single-photon sensitivity of the QIS concept will allow polarization imaging at much lower illumination levels.

Also, polarization information may be used to supplement the intensity and color information in conventional monochrome and color images for applications such as object tracking and surveillance. The QIS concept provides an advantage in this domain by virtue of its dynamic pixel size. For this application the pixel sizes in the polarization image can be dynamically changed by varying the number of jots summed in the spatial or temporal domain to form the pixel. This allows the resolution to be changed in a specific region to keep track of a target. It also allows the frame rate to be dynamically changed by reducing the number of jots summed in the temporal direction to track fast-moving events or objects in the region of interest.

As suggested in [11], binary polarization information may be used to track variation in polarized light intensity caused by object motion toward the sensor. This allows motion estimation using the binary polarization data. When a QIS imager is used to sample polarization information, temporally oversampled binary data is produced. This makes it possible for different motion estimation methods, including temporal differencing, to be used for improved motion detection.

Furthermore, the benefits of enhanced dynamic range and increased flexibility in post-capture control of different parameters makes this concept attractive for polarization imaging. With increased dynamic range in the sensor, the range of polarization information that can be represented will be increased. In order to implement a polarization-sensitive QIS imager, a number of key issues must be addressed. Because of the size constraint of the jots/pixels in the QIS concept, the choice of polarization imaging architecture and polarization filter is extremely important. The formation of the polarization image from single-bit jot images also needs to be addressed. Finally, the problem of simultaneous acquisition of color and polarization information needs to be addressed.

3. POLARIZATION FILTERS FOR SDL PIXELS IN QIS

For the QIS concept, which requires a high readout speed for temporal oversampling, the “division-of-time” architecture for polarization imagers is nonideal because of its frame rate limitation. The division-of-focal plane architecture is well suited for the QIS implementation because it allows single-shot acquisition of polarization information so that high frame rates can be achieved. Because the image is also highly spatially oversampled with the specialized SDL pixels/jots discussed earlier, the division-of-focal plane architecture also will not degrade the resolution.

The choice of polarization filters is equally as important. Polarization filter performance is often discussed with regard to two important parameters: the polarization extinction ratio (PER) and transmission efficiency. The PER measures the ratio of maximum transmitted light polarized in the direction of the polarization axis of the filter to the transmitted light polarized perpendicular to the filter’s polarization axis. A high PER indicates better polarization contrast. The transmission efficiency measures the maximum transmittance of light polarized along the axis of the filter.

Polymer film polarizers have been used for polarization imaging with relative success [12]. However, polymer films are difficult to manufacture with uniform thickness across the entire array, especially when thicknesses less than 10 um are required [13]. They are therefore not suitable for subdiffraction limit pixels because filters with a thickness much greater than the pixel size can cause an increase in cross talk between pixels. Because a thick filter increases the optical path length, light incident over one pixel at an angle can easily reach a neighboring pixel by the time the photons emerge from the filter. Complex fabrication techniques are also required for integrating these polymer filters in a standard CMOS process.

The nanowire grid polarizer is an ideal candidate for linear polarization imaging with the QIS concept because it is compatible with standard CMOS fabrication and provides high transmittance and high PER. For the implementation of a polarization-sensitive QIS device, the filter thickness must be decreased to submicrometer levels to reduce the device form factor as well as the cross talk between jots. Wire grid polarizers fit this criterion because filters with thickness on the order of a hundred nanometers can be realized with high transmittance and PER.

The wire grid polarizer transmits electromagnetic waves with electric field perpendicular to the wires [i.e., transverse magnetic wave (TM waves)] and reflects waves with electric field parallel to the direction of the grid wires [i.e., transverse electric (TE waves)]. TE-polarized light causes electron motion along the length of the wire. This electron motion generates an opposite field, which causes reflection of the incident light. On the other hand, TM-polarized light causes electron vibrations in the transverse direction; however, their motion is restricted by the width/pitch of the wire. The reflectance and transmission of these grids are dependent on the size of the wires as well as the period of the grid and the metal properties. As the wire grid period reduces, the transmission of the TM-polarized wave increases and TE-polarized light transmission diminishes. Narrowing the wires decreases the room available for lateral motion by electrons due to TM waves, thus minimizing attenuation of the TM waves. For TE waves, the narrow wires are closer together, which makes it easier for electrons to be excited up and down the wires to generate an opposing field, which reflects the incident light.

Wire grid polarizers with a linewidth as low as 50 nm and period of 100 nm have been modelled and fabricated and have shown very good performance with transmission and PER values of 85% and 2000, respectively [14]. However, further reduction in pitch is limited by the minimum feature size of available standard CMOS technology. It is expected that, as the minimum feature size decreases in advanced technology nodes, wire grid polarizers with much smaller pitches can be realized. It is possible to model the performance of extremely small pitch wire grid polarizers using FDTD electromagnetic simulation.

Figure 2 shows an aluminum wire grid over silicon oxide substrate simulated using technology computer aided design (TCAD) software from Sentaurus. Periodic boundary conditions were used on the sides and perfectly matched layers on the top and bottom to absorb excess scattered light. The wire grid structure is illuminated from the top by TM and TE polarized light of different wavelengths, and the TM transmission and extinction ratio are calculated. The width of the aluminum wires is varied from 5 to 50 nm with a period twice the linewidth. The results of simulation with aluminum thickness of 50 nm are shown in Fig. 3.

 

Fig. 2. Aluminum wire grid over silicon oxide with thickness of 600 nm simulated in TCAD. Wire grid shown has line width of 5 nm, period of 10 nm, and wire thickness of 50 nm.

Download Full Size | PDF

 

Fig. 3. Simulation results showing the TM-polarized light transmission (top) and polarization extinction ratio (bottom) of aluminum wire grid polarizers with wire linewidths from 5 to 50 nm. The period is twice the linewidth, and wire thickness is 50 nm.

Download Full Size | PDF

From Fig. 3, it is clear that sufficiently high light transmission and extinction ratio can be attained by the aluminum wire grid polarizer even at very low metal linewidths for the short, medium, and long wavelengths in the visible light regime. Blue light, which has the lowest wavelength-to-pitch ratio, also records the lowest transmission and PER, while the red light has the highest. At metal thickness as low as 50 nm, there is sufficiently high attenuation of the TE-polarized wave and transmission of the TM-polarized light, resulting in extinction ratios above 15 dB. Higher extinction ratios can be attained by increasing the metal thickness; however, this also reduces the transmission efficiency of the TM waves.

The wire grid polarizer’s transmittance and extinction ratio also depend on the light incidence angle. TM-polarized light transmission increases while TE-polarized light transmittance decreases with increasing angle of incidence. When the light incidence angle is greater than the Brewster angle at a given wavelength, the TM transmission decreases. Figure 4 shows the angular dependence of the TM transmission and extinction ratio in simulations performed for a polarizer with linewidth of 5 nm. Sufficiently high transmission can be attained for angles less than 60 deg. For larger angles, the incident light has a higher chance of penetrating into neighboring pixels. This increases the cross talk between pixels and corrupts the polarization information.

 

Fig. 4. Simulation results showing the dependence of TM-polarized light transmission (top) and polarization extinction ratio (bottom) on the light incidence angle. Linewidth, period, and metal thickness for this simulation are 5, 10, and 50 nm, respectively.

Download Full Size | PDF

The nanowire polarization filter is therefore ideal for the implementation of a polarization-sensitive QIS imager, which requires extremely small jot sizes. However, the formation of oxides may limit the width of aluminum nanowires that can be realized during fabrication and significantly deteriorate performance at such low linewidths. At higher widths of the metal wire, the formation of the oxide decreases the width of the aluminum wire and can actually increase light transmission. However, for linewidths on the order of a few nanometers, the wires could be entirely oxidized. The oxidation of the aluminum wires could, however, be greatly minimized during fabrication by first patterning a dielectric grating and then filling in the grooves with the aluminum. This greatly minimizes the oxidation of the sidewalls of the aluminum wire because only the top of the metal wire is exposed.

Even when such small-sized nanowire polarization filters are fabricated, another important hurdle that has to be surmounted is the alignment between the filters and the small jots. As previously mentioned, jots used in QIS are expected to be a few hundred nanometers in pitch. Alignment of polarization filters over individual jots may prove challenging at this size. Misalignment of filters over jots will reduce the light transmission and decrease the signal captured. Such misalignment may also result in incorrect sampling of the intensity of polarized light transmitted because light, which passes through one polarization filter, may be absorbed in the wrong jot. This results in increased cross talk between jots, which decreases the accuracy in calculation of the state of polarization and also reduces the polarization contrast.

To address this issue, a single polarization filter may be formed over multiple jots in the array. This can be done because the QIS concept employs a high level of oversampling. Formation of a single polarization filter over multiple jots therefore will not result in aliasing in the sampling of any of the polarization intensities. Figure 4(a) shows the conventional polarization imager with a single polarization filter over each pixel. In Fig. 5(b), a single polarization filter is disposed over a group of $m\times n$ jots. This approach increases the tolerances required for filter fabrication and decreases the level of misalignment between jots and the polarization filters. The jots under a single polarization filter may be summed to produce a single pixel in the final image. This reduces the effect of cross talk from one jot to the other for jots located under the same filter.

 

Fig. 5. Polarization filter array pattern formed over image sensor with (a) a single polarization filter over each pixel (or jot). (b) Single polarization filter formed over a region ($m\times n$) of pixels (or jots).

Download Full Size | PDF

4. RECOVERING THE POLARIZATION IMAGE FROM THE BINARY JOT IMAGE

In conventional polarization imagers, the polarized light intensities obtained by integrating photoelectrons over an integration period are used to calculate DoLP and AoP. Such outputs may be analog or digital signals with a resolution of 8 bits or more because conventional CMOS or CCD image sensors are used. Polarization-sensitive QIS imagers, however, may produce single-bit outputs, so the calculation of the state of polarization from single-bit jot output has to be addressed. It is evident that, if the intensity of light transmitted through the four different polarization filters is quantified by single-bit/binary output, only a few polarization angles can be computed. In fact, with single-bit outputs, using Eqs. (1)–(5), only eight different angles can be calculated regardless of the polarization of the incident light. Therefore, a kernel of jots must be used to reconstruct the intensity information for calculation of the DoLP and AoP.

Figure 6 shows the result of a Monte Carlo simulation in which the angle of polarization is calculated from the output of single-bit jots. In Fig. 6, the pixels underneath the four polarization filters are reconstructed by summing $m\times n\times t$ cubicle/kernel of jots. This kernel consists of $m\times n$ jots in the spatial domain summed over $t$ frames in the time domain. In these simulations, it is assumed that on average one photon is incident over four ideal polarization filters placed over an array of single-bit jots. This average number of photons is known as the Quanta exposure (H). The light transmitted by the polarization filters is dependent on the incident light’s intensity $I_o$ and angle of polarization $\varphi$, as well as the transmission angle of polarization filter $\theta$:

 

Fig. 6. Simulation results of the polarization angles calculated from pixels formed by summing $m\times n\times t$ single-bit jots. The incident photons are varied from jot to jot using a Poisson random function with mean equal to the Quanta exposure, $H=1$ photon.

Download Full Size | PDF

Figure 7 shows a simulation in which the angle of polarization is calculated for several runs assuming a reference light with polarization angle of 60 deg. It is clear from the histogram of calculated polarization angles that computation of polarization information from four jots with single-bit output is not feasible. With a single bit representing the output of each of the four polarization intensities, only a few different angles can be represented. As more jots are summed to represent the intensity of each of the different polarizations, more angles can be calculated, and the accuracy of the calculation improves. For multibit jots, which integrate a few photoelectrons, fewer jots will need to be summed compared with single-bit jots. The signal-to-noise ratio improves as more jots are summed to represent each intensity. It should be mentioned that, because oversampling is done in the temporal and spatial domains, jots can be summed in the temporal dimension as well. The size of the spatiotemporal kernel of jots summed to produce the pixel values therefore depends on the application. Applications requiring very high frame rates may use a large summation kernel size in the spatial domain so that fewer frames are summed in the temporal direction.

 

Fig. 7. Simulated results of polarization angles calculated from pixels formed by summing outputs of $m\times n\times t$ single-bit jots illuminated by 60 deg polarized light. Simulation is repeated 1000 times, and the histogram shows the distribution of calculated polarization angles. Photon arrival is varied using a Poisson random function.

Download Full Size | PDF

5. MIXED COLOR POLARIZATION IMAGING WITH QIS

The use of QIS also has the added benefit of allowing simultaneous spectral (color) and polarization imaging. Presently, time-multiplexed capture of consecutive color frames with a focal plane array polarization imager is the common means of concurrently obtaining polarization and color information. This method is limited by the need for electrical or mechanically modulation of the filters [4]. Other methods of simultaneously acquiring color and polarization information discussed earlier have not achieved much success because of the complexity in their implementation.

The intuitive method of simultaneous color and polarization imaging is to include both polarization and color filters in a single array so that interpolation methods can be used to retrieve full color and polarization images. Even though this concept of mixed-polarization color filter arrays has been considered [15], it has not been employed mainly because including color and polarization filters in the array would reduce the frequency with which the polarization and color information can be sampled. Using such mixed filter patterns in conventional image sensors may result in insufficient sampling of either color or polarization information resulting in aliasing-related artifacts. Another important limitation is the increase in stack thickness associated with the stacking of polarization filter and color filter layers. For instance, polyvinyl alcohol filters have a typical thickness of about 10 μm and above [13]. Increased stack thickness will result in high cross-talk levels between pixels, rendering color and polarization information potentially unusable.

With the QIS concept, the very high level of oversampling in the spatial domain allows polarization and color filters to be included without risk of aliasing in sampling either quantity. Again, a single color or polarization filter may be formed over multiple jots to ease the tolerances for filter fabrication and reduce the cross-talk impact. Sample mixed polarization color filter arrays that can be used are depicted in Fig. 8. Several other color and polarization filter variations can be used. In the mixed polarization color filter shown in Fig. 8(a), the white/panchromatic color filter pixel may be simultaneously used for recovering color information and providing the total intensity required for the polarization calculation.

 

Fig. 8. Sample mixed polarization-color filter patterns. (a) Pattern of red, blue, and white/panchromatic filters along with 0 and 45 deg polarization filters. (b) Pattern with red, green, and blue color filters and 0, 45, 90, and 135 deg polarization.

Download Full Size | PDF

For such a mixed polarization color filter array, the wire grid polarizer may be an ideal candidate. Fabrication of the jots in the QIS concept requires advanced process nodes such as 65 nm and below. This will allow wire grids filters with high extinction ratios to be fabricated using metal layers in the standard CMOS process. In a front-side illuminated sensor, the front-side metal layers may be used for patterning the polarization filters. In a backside illuminated (BSI) sensor, the metal layer used for defining the aperture of each pixel can be co-opted for the patterning of the wire grid polarization filters.

In fact, drawing from the recent trend of embedded color filters [16], a mixed polarization color filter array can be realized with low stack thickness. In backside illuminated sensors, these embedded color filters are formed by filling in the back metal aperture grid with color filter resist. Using this technology, the aperture-defining metal layer can be used first to pattern the wire grid polarization filters over polarization-sensitive pixels. Then, the region above the color-sensitive pixels can be etched, and color filter resist can then be embedded. A sample process for realizing such a mixed polarization color filter array is shown in Fig. 9. Alternatively, the mixed polarization color filter array may be formed entirely in the metal layer by implementing polarization and color filters with subwavelength metal grid patterns. The feasibility of color filter formation with subwavelength metal grids has been demonstrated [17]. Better filter performance is expected, as technology nodes scale down and smaller grid/grating periods can be realized.

 

Fig. 9. Fabrication steps for a mixed polarization color filter array pattern on the aperture-defining metal layer in a backside illuminated image sensor.

Download Full Size | PDF

In summary, polarization imaging systems will ultimately be limited by diffraction as pixel sizes decrease. The application of the Quanta image sensor concept to polarization imaging is therefore the next logical step as pixel sizes decrease further in such imagers. The benefits of using QIS for polarization imaging presented in this paper include improved full-well capacity and dynamic range as well as enhanced low light polarization imaging. High spatial oversampling in the QIS concept can also be leveraged for mixed polarization color imaging, by including color- and polarization-sensitive pixels in the same array. The wire grid polarization filter has been identified as a suitable candidate for implementation of a polarization-sensitive QIS imager. Improvement of light transmission and polarization extinction ratio of these filters with decrease in grid pitch and their ease of fabrication in standard CMOS make them particularly attractive. At the extremely small feature sizes anticipated, careful selection of the metal used as well as the dimensions of this grid may also promote surface plasmon resonance enhanced absorption in the substrate underneath these wire grid filters. Anticipated barriers to the implementation of QIS polarization imagers also have been identified, and innovative solutions have been proposed to surmount these obstacles.