Open Access
Add to BibSonomy
Abstract
Despite recent advances, customized multispectral cameras can be challenging or costly to deploy in some use cases. Complexities span electronic synchronization, multi-camera calibration, parallax and spatial co-registration, and data acquisition from multiple cameras, all of which can hamper their ease of use. This paper discusses a generalized procedure for multispectral sensing using a pixelated polarization camera and anisotropic polymer film retarders to create multivariate optical filters. We then describe the calibration procedure, which leverages neural networks to convert measured data into calibrated spectra (intensity versus wavelength). Experimental results are presented for a multivariate and channeled optical filter. Finally, imaging results taken using a red, green, and blue microgrid polarization camera and the channeled optical filter are presented. Imaging experiments indicated that the calculated spectra’s root mean square error is highest in the region where the camera’s red, green, and blue filter responses overlap. The average error of the spectral reflectance, measured of our spectralon tiles, was 6.5% for wavelengths spanning 425-675 nm. This technique demonstrates that 12 spectral channels can be obtained with a relatively simple and robust optical setup, and at minimal cost beyond the purchase of the camera.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Spectral imaging is helpful in many applications, including remote sensing [1–6] and agricultural plant phenotyping [7–10]. While many snapshot imaging spectrometer architectures exist [11], many are complex, expensive, or challenging to access or optimize. Recently, microgrid polarization cameras have been used to create imaging spectrometers as per Ono [12], who developed and demonstrated a division of aperture approach, or Schonbrun [13], who established a phase-shifting approach using a multiwave chiral disperser. Both approaches enable color detection through the use of a microgrid polarization camera and polarization modulation. However, while Ono’s [12] division of aperture (DoA) approach was able to detect 9 spectral channels, the use of DoA is complex for many users to implement on their own. Additionally, it experiences a reduced signal-to-noise ratio (SNR) as the number of spectral bands is increased (proportional to 1/N, where N is the number of spectral channels), and it undergoes parallax when imaging nearby targets. Additionally, the use of chiral dispersers, as with Schonbrun [13], can be challenging to find and implement and typically have lower birefringence compared to linear retarders.
In this paper, we focus on expanding the accessibility of spectral imaging sensors [14–16] through the use of polymer retarders and arbitrary Solc filters [17–19], combined with either monochrome or red, green, and blue (RGB) microgrid polarization cameras. The use of polarization offers multiple advantages, including: (1) Commercially available and rugged microgrid polarization cameras are now available at moderate cost; (2) Filter designs can be easily customized with existing software; (3) Filters can be assembled with moderate expertise; (4) Polymer film retarder sheets are inexpensive and readily purchased; and (5) At least two types of filters can be created - either 4-channel multivariate filters as sampled using a monochrome microgrid camera or a 12-channel multispectral filter using an RGB microgrid polarization camera. This paper is organized as follows: in Section 2, we overview the system layout and organization, Section 3 reviews the simulation and optimization method, Section 4 details experimental results, and Section 5 details our laboratory results captured using the 12 channel spectrometer.
2. System layout
The system relies on using polarization to create a multivariate filter response. Unlike a bandpass filter, which transmits a relatively narrow range of wavelengths around the passband, multivariate filters allow many wavelengths to transmit simultaneously. Such a filter can enable higher signal to noise ratios for targets that contain multiple spectral features, in that the multivariate filter’s response can be tailored to it [20]. A view of the system layout is depicted in Fig. 1. Specifically, Fig. 1(a) illustrates a configuration used to achieve a multivariate filter response at each of the focal plane array’s (FPA’s) 4 unique microgrid pixels. Light from the scene transmits through a horizontal linear polarizer (LP0) before traversing a stack of K polymer film retarders. Each of the retarders (WP${_1}$ through WP${_K}$) can contain differing retardance values, as indicated by the cyan or magenta-colored components, in addition to unique fast axis orientations ($\mathrm {\theta }_{1}$ through $\mathrm {\theta }_{\textrm {K}}$). After transmitting through the filters, the light is focused by an objective lens onto a monochrome microgrid FPA.
Fig. 1. System schematic of the (a) multivariate filter; and (b) multiple passband filter. Acronyms: Focal Plane Array (FPA); Linear Polarizer (LP); Waveplate (WP); and Achromatic Quarter Waveplate (AQWP).
Meanwhile, Fig. 1(b) depicts a configuration used to obtain multiple distinct spectral passbands. In this configuration, films of equal retardance are laminated together with their fast axes aligned to create a customizable multi-wave (4-8 wave) retarder. An achromatic quarter-wave plate (QWP) is oriented, with a fast axis at 45 degrees, to convert the linear retarders’ eigenstates to circular polarization. This enables each of the FPA’s pixels to view a phase-shifted version of the spectral transmission response function, similar to Schonbrun [13] as applied to 4 band imaging and, generally, typical of geometric phase shifters [21,22]. In the case of Schonbrun, the microgrid’s transmission axis changed the geometric phase, which was mapped onto the wavelength of the spectral filter; thus, a blue, green, or red filter was produced by selecting different linear polarizer transmission axis orientations. When combined with an RGB microgrid polarization camera, each color’s superpixel is able to detect a greater number of unique spectral passbands, similar to the division of aperture approach demonstrated by Ono [12]. However unlike Schonbrun, Ono encoded individual narrow spectral bandpass filters onto different linear polarization states, which were then demultiplexed by the microgrid camera and software. Our method deploys similar concepts, such that when combined with appropriate characterization, calibration, and reconstruction algorithms, complete spectra (intensity versus wavelength) can be extracted from the measured data. Alternatively, algorithms can use the data directly from the sensor to perform classification tasks, avoiding the need to convert raw data into spectral intensity [23,24].
Finally, it should be noted that a multiwave retarder could also be selected for the design in Fig. 1(b); however, for cost and accessibility purposes, we opted to laminate polymer films together. This strategy also increases the field of view over retarders that use two plates with orthogonal fast-axes when the retarder is not field-widened [25].
3. Optimization and simulations
Simulations were performed to optimize the multivariate filter response function and to model the filter design with multiple passbands.
3.1 Multivariate Solc filters
The birefringent elements were modeled using Mueller calculus [26]. The Mueller matrix of a diattenuator can be expressed as
Similarly, the Mueller matrix of a retarder with an arbitrary retardance and fast axis orientation can be defined as
It should be noted that for our results, we generally set MSP = 12 when working with an RGB polarization camera since the two green Bayer filter values are redundant. Finally, Eq. (5) with ${\delta _Q}$ of ${0^{\circ }}$ can be used to model the system depicted in Fig. 1(a), while a ${\delta _Q}$ of ${90^{\circ }}$ effectively models the system depicted in Fig. 1(b). In the former case, a retardance of zero produces an identity matrix. Finally, all values are implicitly dependent on wavelength ${\lambda }$.
Multivariate filter optimization consisted of the following steps:
- 1. Load or define the target spectra that need to be classified;
- 3. The optimizer will run Nitert times, randomly selecting:
- (a) The number of layers for the given permutation, defined as being less than or equal to K;
- (b) The Solc layers’ fast axis orientations; and
- (c) The retardance of each layer, as chosen randomly from available films (OPDs of 560, 280, 165, and 140 nm are readily available).
For each combination of the above parameters, the optimizer then calculates:
- (d) The spectral response of each FPA microgrid pixel is calculated using Eq. (5) to produce a set of spectral response functions for each of the pixels within a superpixel. An example illustrating the spectral response functions calculated for a multivariate filter is provided in Fig. 2 for each of the 4 pixels within a superpixel of a monochrome polarization camera.
- 5. Each filter response is spectrally band-integrated with shot noise at a fixed spectral irradiance ${ph/s/nm/superpixel}$. This ensures that designs with wider spectral passbands (e.g., larger full-width at half maximum spectral passbands) have relatively higher SNR compared to designs with narrower passbands [11]. Signals are then modeled by$$I\left( m \right) = \frac{1}{{{M_{SP}}}}\int_{{\lambda _{\min }}}^{{\lambda _{\max }}} {{T_{LP,m}}\left( \lambda \right)\left[ {\sum_{n = 1}^{{N_{\max }}} {{a_n}{B_n}\left( \lambda \right)} } \right]d\lambda } + {\mathbf{n}} \, ,$$where ${B_{n}}$ is the response of each basis spectra n and n is a vector of length ${\lambda }$ containing shot noise.
- 6. A subset of the simulated signals were used to train a shallow neural network classifier Nrep times. The root-mean-square (RMS) error was then calculated, for each network, using data that was not used for training. The minimum RMS error was chosen to represent the filter’s maximum performance at classifying the basis spectra.
- 7. If the minimum RMS error is less than a prior result, the design is stored and the protocol returns to step 3 Nitert times to generate and analyze a new Solc filter design.
Fig. 2. An optimal randomly generated multivariate filter response, depicting each polarization pixel’s spectral response versus wavelength alongside four example target spectra.
Using the above optimization steps, we created a multivariate design optimized to detect four narrowband signals at spectral locations of 500, 545, 656, and 700 nm. The calculated filter spectra are presented in Fig. 2 alongside the simulated target spectra. Simulation results are presented in Section 3.3. Overall, we maintained a constraint that only 10 layers should be used in the design in order to minimize the fabrication complexity.
3.2 Multiple passband filter using a thick retarder
Additionally, we simulated the filter response of a thick multi-order retarder when joined with a phase-shifting achromatic quarter-wave plate and an RGB microgrid FPA. The RGB bands were modeled after the Sony IMX477, which, while not a microgrid FPA, has a similar RGB color response. This choice also enabled us to compare the polarization filters’ performance to that of a standard RGB camera, without a microgrid, per Section 3.3. The response of each linear polarizer on the microgrid is depicted in Fig. 3(a). Each transmission function generates a sinusoidal pattern versus wavenumber, which appears to have a chirped frequency when plotted versus wavelength [27]. These spectra have a notably higher frequency versus wavenumber compared to Schonbrun [13], such that they would have a reduced ability to extract individual color spectra if sensed with a monochrome camera. However, when these spectral responses are combined with an RGB camera as per Fig. 3(b), the separability of the individual peaks increases. Despite this, there still exists significant crosstalk between adjacent spectral channels within a given Bayer filter’s full width spectral bandwidth. As described in Section 4, crosstalk is corrected for using a neural network-based reconstruction approach [28,29].
Fig. 3. multiple passband design. (a) Spectral response of each linearly polarized pixel generated by a single high order retarder followed by an AQWP; (b) Response of the system when combined with an RGB microgrid pattern.
3.3 Simulation results
Regression accuracy was calculated using a cascade forward neural network in MATLAB R2021a. This network was trained to estimate the abundances in the vector ${\textbf {A}}$ (each ${a_n}$ in Eq. (7)) for each basis spectrum ${B_n}$ in Eq. (8). The network’s input was either 3, 4, or 12 unique intensities for the RGB, Solc, or multiple passband filters, respectively, as calculated using Eq. (8). A total of 10,000 training spectra were used to create 20 neural networks, from which the best performing network was used in our analysis. The best network was based on the minimum RMS error, calculated using 1,000 validation spectra. Signal power ranged from 1 ph/s/nm/superpixel up to 49.7 ph/s/nm/superpixel. Shot noise was applied to the band integrated signals before training and inference.
A view of the results of this simulation are depicted in Fig. 4. Provided is the coefficient of determination ${R^2}$ of the estimated abundances, as calculated by the neural network, versus the true abundances, as defined in the simulation. The coefficient ${R^2}$ is provided for each of the 500, 540, 656, and 700 nm basis spectra per Fig. 2. Overall, the RGB sensor is reasonable at distinguishing abundances in the 500 and 542 nm spectra (${R^2}$ > 0.9 for signal > 15 ph/s/nm/superpixel), given that these spectra are separable into the camera’s blue or green channels, respectively. Conversely, the RGB camera is unable to accurately calculate the abundances of the 656 nm and 700 nm target spectra (${R^2}$ < 0.5 for all signal levels). Meanwhile, the Solc filter is better at distinguishing the 656 nm and 700 nm spectra (${R^2}$ of 0.75-0.85) but worse at distinguishing the 500 and 540 nm spectra (${R^2}$ of 0.50-0.62) for signal > 15 ph/s/nm/superpixel.
Fig. 4. Coefficient of determination for simulated signals detected using 3 different systems. Results are presented for the multivariate Solc filter (solid line), off-the-shelf RGB camera response (dotted line), a RGB polarization camera with retarder (dashed line), and an RGB polarization camera with the multivariate Solc filter (dash-dotted line) for the 500, 542, 656, and 700 nm target spectra.
Despite optimization, this specific set of basis spectra may be challenging for a Solc filter with 4 spectral channels to detect. This was assessed by calculating the condition number of a measurement matrix Q that is defined in the sensor’s measurement space [23]. This matrix relates the detected intensity to each abundance vector by
where I is a vector of length MSP containing the superpixel’s intensities and A is the abundance vector. Each element of Q is calculated by where $I\left ( {m} \right )$ is calculated using Eq. (8) without noise n for each basis spectrum n, and n, m are the row and column indices within Q. The condition number of Q was 283 for the RGB camera, 31.3 for the Solc filter using a monochrome polarization camera, and 4.51 using a thick retarder and RGB filter. Thus, the relatively lower performance of the Solc filter was likely due to its higher condition number, caused by greater crosstalk, when compared to the thick retarder using an RGB polarization camera.To remedy the lower performance of the Solc filter, we also simulated its performance using an RGB polarization camera. This provided a condition number of 4.65. While this is similar to that of the thick retarder, it comes without the need for an (often more expensive) achromatic QWP. The updated performance of this new configuration is presented alongside the data in Fig. 4(a-d) and generally achieves performance similar to, but less than, that of the thick retarder, with ${R^2}$ > 0.8 for signal > 15 ph/s/nm/superpixel for all spectra n. This analysis demonstrates the complex tradespace that this sensor offers when deployed for specific use cases, in that there are many degrees of freedom for optimization that balances cost (e.g., the use of an achromatic QWP), performance, and assembly complexity (e.g., alignment of a multivariate filter’s Solc stages vs. simply stacking films, at the same fast axis orientation, to form a thick retarder). Assembly complexity and performance is demonstrated in our experimental results per Section 5.
4. Calibration and reconstruction
The sensor was calibrated by quantifying the measurement matrix H using a monochromator and an integrating sphere. A view of the characterization setup is illustrated in Fig. 5(a). It consisted of a 75 W xenon arc lamp (Optical Building Blocks) sourcing a monochromator (Horiba MicroHR with a 140 mm focal length). The monochromator illuminated an integrating sphere, which was positioned to fill the sensor’s field of view. A photo of the sensor is provided in Fig. 5(b), which consisted of rotation mounts for the linear polarizer (Bolder Vision UHC), waveplate films (American Polarizers Inc), and achromatic quarter waveplate (Bolder Vision AQWP3), followed by the objective lens (Navitar 12 mm focal length lens, Thorlabs MVL12M23) and polarization camera (Lucid Vision Labs Trition TRI050S1-PC). The filter assembly as built, including the rotation stages and optomechanics, was approximately 100 mm in length. However, the linear polarizer, filter films, and achromatic quarter waveplate, if stacked into a monolithic device, would only be 4 mm thick and could be mounted with the objective lens’s filter threads, increasing compactness.
Fig. 5. (a) Schematic of the calibration setup for characterizing the spectral response of the system. (b) Photo of the system on the benchtop during calibration.
Calibration imagery was measured for wavelengths spanning 400-700 nm in 2 nm increments at a spectral full width at half maximum bandwidth of 3 nm. Additionally, the spectrum of the xenon arc lamp and monochromator was measured separately using a calibrated radiometer (Thorlabs PM100D with an S142C silicon photodetector) to quantify the input power separately. This enabled the integrating sphere’s images to be normalized to the spectral response of the arc lamp and monochromator by
As based on our prior work, we trained a cascade forward neural network to reconstruct data from the sensor using MATLAB [28,29]. Synthetic spectra were created based on either low-pass filtered random uniform distributions or by using modified sigmoid functions. Each synthetic spectrum had a length of $N_\lambda =151$ elements that corresponded to each wavelength that was sampled using the monochromator. Approximately 20,000 synthetic spectra were generated using the following procedure:
- 1. A measurement matrix H was extracted from a randomly selected spatial location inside the sensor’s field of view. This ensured that the reconstruction algorithm can accommodate for the small field of view dependence in the retarder’s spectral transmission versus angle [25]. This produced a matrix$${\mathbf{H}}\left( {{x_r},{y_r},\lambda ,m} \right) = {I_{cal}}\left( {{x_r},{y_r},\lambda ,m} \right)$$where $x_r$ and $y_r$ are randomly selected x and y coordinates and m is the superpixel’s index as defined previously in section 3.1 under step 3 part (b) of the simulation. H has dimensions $M_{SP} \times N_\lambda$ and m spanned 1 to $M_{SP} = 12$ for our calibration since the green filter is duplicated twice on our RGB polarization camera.
- 2. A random spectrum was generated by either:
- (a) Applying a low pass filter to a uniform random number distribution versus wavelength. The hamming filter’s full width was randomly selected to be within the range of optical path difference’s spanning 1 to 3 $\mu$m, such that the spectral resolution of any created features exceeded 130 nm. This created a numerically generated vector $\mathbf {v}\left ( \lambda \right )$.
- (b) Creating a sigmoid based on the function$$\mathbf{v}\left( \lambda \right) = {\left[ {1 + \exp \left( {\frac{{ - \lambda + {\lambda _0}}}{d}} \right)} \right]^{ - 1}},$$where d is the function’s slope and $\lambda _0$ is the center wavelength (50% power point). Both d and $\lambda _0$ were uniform random variables with ranges spanning 10 to 100 nm and 430 to 675 nm, respectively. It should be mentioned that the spectral bandwidth needed to change from 10% to 90% of full, given this range of d, was spanned 45 to 440 nm.
Spectra were then selected uniformly from one of the two above categories with a probability of 50%.
- 3. The sensor’s signal was then calculated using the measurement matrix by where $\mathbf {I}$ is a vector of length $M_{SP}$ that represents the intensity measured by each pixel inside the superpixel. Noise was then added by normalizing I to 1 and scaling it by a randomly selected number of photons spanning 1000 to 20000. Poisson noise was then applied to the signal and the signal was re-normalized back to a maximum value of 1 (normalization better facilitates the neural networks’ training).
- 4. The spectrum from (2), which had 151 spectral samples, was then down-sampled using spline interpolation to have $M_{SP}$ samples.
Since the network was trained from scratch (e.g., transfer learning was not used), the network’s architecture was selected by training 1000 randomly generated networks. Approximately 18,000 of the 20,000 spectra were used for training and the remaining 2,000 spectra were used for validation. The network’s architecture was determined for each iteration based on a random number of layers, spanning 1 to 5, and a random number of nodes, spanning 2 to 15. Networks with the best architecture were selected based on the minimum mean square error in the validation set. The training function used was based on the scaled conjugate gradient technique [30]. Finally, training was accomplished on the central processing unit (AMD 3900X, 90 seconds / network) to demonstrate that a GPU is not necessarily a prerequisite for deploying this technique; however, training on the GPU was generally faster (Nvidia GeForce 3090, 40 seconds / network).
A view of the network architecture is depicted in Fig. 6. Generally, networks with 5 layers performed better than networks with fewer layers. This resulted in a cascade forward neural network with 5 hidden layers comprising, from input to output, of 10, 6, 12, 14, and 16 nodes. There were 12 input nodes, corresponding to the number of used superpixels, and 12 output nodes, corresponding to the number of wavelengths in the reconstruction.
Fig. 6. Cascade forward neural network architecture that was used to convert measurements from the sensor’s modulated measurement space to spectral intensity versus wavelength.
Fig. 7. Results for the multivariate filter. The (a) originally designed (O), fabricated (F), and measured (M) filter responses; and (b) the original fast-axis orientation angles, in degrees, as compared to their fabricated values. Fabricated values were calculated by fitting the measured and theoretical transmission spectra.
5. Experimental results
5.1 Filter fabrication
Experiments were conducted in the lab to demonstrate aspects of both the multivariate and multiple passband design concepts. First, the multivariate design was optimized to produce a prescription such that the OPD of each layer, for light traveling from left to right (starting at 280 nm), was OPD = [280, 560, 140, 560, 560, 560, 140, 280, 140, 560] (in nm) with fast axis orientations of [17, 14.5, 17.5, 6.5,−7, 16, 17, 7.5, 5.5, 10] (in degrees). Films of appropriate OPD were laminated together and inserted between a fixed linear polarization generator at 45 degrees and a rotating linear polarizer. Measurements were then taken with an Ocean Optics spectrometer (USB2000) through analyzer linear polarization states of 0, 45, 90, and 135 degrees. A view of the originally designed theoretical (O) and experimentally measured (M) spectral transmittance of this design are depicted in Fig. 7(a). Generally, the shape of the transmission functions is preserved; however, discrepancies exist in the I0 channel around 640 nm and in the magnitude of the peaks in the I45, I135, and I90 channels for wavelengths spanning 425-600 nm. These discrepancies are likely caused by misalignment of the fast axis during the filter’s assembly, which was done by hand. To test this, we fit the filter’s model (Eq. (8)) to the measured transmission functions using the fast axis orientations as variables. This produced the fabricated (F) transmission functions provided in Fig. 7(a), which generally follow the measured transmittances within $\pm$2%. The original fast axis orientation, in degrees, is plotted alongside the fitted (fabricated) values Fig. 7(b). Overall, absolute fabricated alignment differs from the desired alignment by $3.5^{\circ }$.
We also designed a thick film retarder and tested its spectral response using a RGB microgrid camera. The camera and retarder were positioned in front of an integrating sphere, which was sourced by a monochromator connected to a xenon arc lamp. Wavelengths were scanned from 400 to 700 nm in 2 nm increments to build the spectral response of each pixel inside the RGB superpixel (note that each superpixel is ${4\times 4}$, containing a repeated pattern of ${2\times 2}$ polarization superpixels in which there are 2 green superpixels, 1 red, and 1 blue). A view of these data are depicted in Fig. 8(a-d), illustrating the response of each colored superpixel within all 4 microgrid polarization states.
Fig. 8. Spectral response of the high order retarder with an RGB microgrid camera measured with a monochromator. Response for the (a) red; (b) blue; and (c,d) green superpixels. Note that each colored superpixel has all 4 migrogrid linear polarization states (0, 45, 90, and 135 degrees) associated with phase shifts of 0, 90, 180, and 270 degrees, respectively.
5.2 Imaging and validation
In addition to characterizing fabricated filters, we also collected imagery data using the sensor in the laboratory. Two sets of targets were quantified: (1) NIST-traceable spectralon color reflectance tiles (LabSphere); and (2) A commercial color checker board (X-rite ColorChecker Classic XL).
RGB imagery from the camera system of these two targets are presented in Fig. 9. Figure 9(a) depicts the NIST-traceable spectralon reflectance standards, which consisted of a white 99% reflectance tile alongside a blue, green, yellow, and red color tile. Each tile has an independent spectral characterization provided by LabSphere. Additionally, Fig. 9(b) depicts a commercial color checker board. The colored tiles, indicated by a row and column index in the image, were characterized independently at near-normal incidence using our Ocean Optics spectrometer (model HR4000). Relative reflectivity was then calculated by dividing the labeled tiles, e.g. (2,2), by that of the white tile in the corner of the color checker. Note that the gray tiles were not used in our analysis since the normalized spectral reflectance is uniform across these objects.
Fig. 9. RGB images of the tile setups used to characterize the calibration’s accuracy. (a) The NIST-traceable white and colored spectralon reflectance standards. (b) A standard color checker board with specific tiles indicated by row and column indices.
A spatial reconstruction of the 2-dimensional spectra are depicted in Fig. 10 while normalized reflectance spectra, calculated from these images, are provided in Fig. 11 alongside their normalized NIST values. Inference speed was measured to be 200 ms per image on the CPU (AMD 3900X) and 30 ms on the GPU (Nvidia GeForce 3090) in MATLAB. Since the neural network reconstruction technique does not always maintain the absolute intensity between the input and output, both the measured and NIST spectra were normalized to the spectrum’s maximum value. The root mean square error between the blue, green, yellow and red spectra, averaged over wavelengths spanning 425-650 nm, were calculated to be 5.6%, 8.2%, 7.6%, and 4.8%, respectively, making the average root mean square error 6.5% for these tiles.
Fig. 11. Normalized reflectance of the blue, green, yellow, and red NIST-traceable spectralon reflectance standards (solid lines) compared to the measured values. Error bars represent the standard deviation across a ${11\times 11}$ pixel spatial patch on the tile.
Additionally, the normalized spectral reflectance calculated from the reconstructed color checker images are provided in Fig. 12. With reference to the row and column indices in Fig. 9(b), the normalized reflectance for tiles (2,1) through (3,3) are provided in Fig. 12(a) while the results for tiles (3,4) through (4,6) are provided in Fig. 12(b).
Fig. 12. Normalized reflectance calculated for each color checker tile (denoted as ’x’) as compared to a independent measurement taken with an Ocean Optics spectrometer (solid line). Row and column indices follow those indicated previously in Fig. 9(b). Error bars represent the standard deviation across a ${11\times 11}$ pixel spatial patch on the tile.
6. Results discussion
As observed in Fig. 11 and Fig. 12, overall performance is lower for wavelengths spanning 480-510 nm and 580-620 nm. This is especially evident when a tile’s reflectance transitions between its maximum and minimum value (or vice versa) within these spectral regions. This observation was confirmed by calculating the average root mean square error between the sensor’s reconstructed spectrum and that taken with the Ocean Optics spectrometer using the data in Fig. 12. As depicted in Fig. 13, the maximum error occurs within these two spectral regions.
Fig. 13. Average RMS error versus wavelength calculated from the measurements of the color checker tiles (left axis, solid black line) plotted with the camera’s relative spectral response for its red, green, and blue filters (right axis, dashed lines).
It is anticipated that the higher error in these regions is caused by the increased crosstalk created by the spectral overlap in the Bayer filters’ spectral transmission. The filters’ response, as calculated using the data in Fig. 8, is also provided alongside the root mean square error in Fig. 13. Highest error occurs for any reconstructed wavelength where there is a significant response from more than one filter (e.g., 486 nm, 507 nm, 588 nm, and 609 nm). Such behavior is expected since the RGB filters are being used to reduce the crosstalk to enable more channels to be reconstructed when compared to a monochrome polarization camera.
Additionally, there is also higher error at the two shorter wavelengths (486 nm and 507 nm) when compared to longer wavelengths (588 nm and 609 nm). We expect that this is caused by the higher density of the modulations at shorter wavelengths, as indicated in Fig. 8(b), as compared to the lower frequency modulations at the longer green and red wavelengths, as indicated in Fig. 8(c) and (a), respectively. This means that the calibration algorithm will have more channel crosstalk to resolve at these shorter wavelengths. It is anticipated that performance can be improved at these shorter wavelengths, if desired, by reducing the thickness of the retarder at the potential cost of reduced spectral resolution. Such a tradespace will be studied in our future research efforts.
Finally, the average full width at half maximum (FWHM) resolution of the filter’s spectral modulation, as observed in Fig. 8, varies from approximately 25 nm in the blue (Fig. 8(b)) to 75 nm in the red (Fig. 8(a)). We quantified the FWHM spectral resolution in the reconstructions on the order of 60 nm in both the blue, green, and red wavelengths, where the higher spectral resolution of the shorter wavelength modulations is likely suppressed by the crosstalk between the multiple modulation peaks within the blue and green Bayer filter channels. However, this resolution is sufficient for characterizing the spectral reflectivity of slowly varying intensity versus wavelength per our experimental results in Figs. 9 and 12. This is sufficient for many plant or agricultural sensing applications, where the spectral bandwidth of the vegetative red edge is similar to that (approximately 50 nm) of the NIST red tile in Fig. 9. Meanwhile, classification of higher spectral resolution targets may still be possible using a classifier-based approach as per Section 3.1. When using a classifier, we observe similar advantages to e.g., "smashed filters" as per Ref. [24] in that high spectral resolution features inside the spectrum can be identified, even when the spectrum can not be recovered at the desired resolution.
7. Conclusion
This paper has provided results regarding a readily accessible and implementable spectral classification method using either a customizable multivariate filter or a channeled polarization filter. When implemented using birefringent polymer films in a arbitrary Solc filter, the multivariate response can be optimized to maximize a classifier’s performance in response to the target spectra. Similarly, using the same films with aligned fast axes, multiple distinct spectral passbands can be reconstructed using the measured data. Results indicated that the root mean square error, averaged versus wavelength, was approximately 6.5% for the spectralon tiles. Generally error was highest for wavelengths of 486 nm, 507 nm, 588 nm, and 609 nm, which corresponded to wavelengths with higher crosstalk between the red, green, and blue color filters. Error was also highest for the shortest wavelengths in the regions containing crosstalk (486 and 507 nm) due to the higher frequency spectral modulations that occur in a channeled filter at shorter wavelengths. Our future work will be to refine our calibration methods and experimental results and to publish the software needed to perform the reconstructions presented in Fig. 11. Such a system could then be broadly applied and supported to improve agricultural and remote sensing applications, chemical identification, and classification. Future work will focus on expanding the spectral sensitivity into the near infrared spectral region, as well as deploying the sensor from drone platforms.
Funding
North Carolina State University GRIP4PSI Program (573000); National Science Foundation (1809753).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. Y. Knyazikhin, M. A. Schull, P. Stenberg, M. M ottus, M. Rautiainen, Y. Yang, A. Marshak, P. L. Carmona, R. K. Kaufmann, P. Lewis, M. I. Disney, V. Vanderbilt, A. B. Davis, F. Baret, S. Jacquemoud, A. Lyapustin, and R. B. Myneni, “Hyperspectral remote sensing of foliar nitrogen content,” Proc. Natl. Acad. Sci. 110(3), E185–E192 (2013). [CrossRef]
2. F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Trans. Geosci. Remote Sensing 42(8), 1778–1790 (2004). [CrossRef]
3. P. Latorre-Carmona, Y. Knyazikhin, L. Alonso, J. F. Moreno, F. Pla, and Y. Yan, “On hyperspectral remote sensing of leaf biophysical constituents: Decoupling vegetation structure and leaf optics using chris–proba data over crops in barrax,” IEEE Geosci. Remote Sensing Lett. 11(9), 1579–1583 (2014). [CrossRef]
4. C. P. Warren, D. Even, W. Pfister, K. Nakanishi, A. Velasco, D. Breitwieser, S. Yee, and J. Naungayan, “Miniaturized visible near-infrared hyperspectral imager for remote-sensing applications,” Opt. Eng. 51(11), 111720-1 (2012). [CrossRef]
5. X. Ceamanos and S. Doute, “Spectral smile correction of CRISM/MRO hyperspectral images,” IEEE Trans. Geosci. Remote Sensing 48(11), 3951–3959 (2010). [CrossRef]
6. M. M. Lunagaria and H. R. Patel, “Evaluation of PROSAIL inversion for retrieval of chlorophyll, leaf dry matter, leaf angle, and leaf area index of wheat using spectrodirectional measurements,” Int. J. Remote. Sens. 40(21), 8125–8145 (2019). WOS:000472112000004. [CrossRef]
7. A.-K. Mahlein, M. T. Kuska, S. Thomas, M. Wahabzada, J. Behmann, U. Rascher, and K. Kersting, “Quantitative and qualitative phenotyping of disease resistance of crops by hyperspectral sensors: seamless interlocking of phytopathology, sensors, and machine learning is needed!” Curr. Opin. Plant Biol. 50, 156–162 (2019). [CrossRef]
8. A.-K. Mahlein, E.-C. Oerke, U. Steiner, and H.-W. Dehne, “Recent advances in sensing plant diseases for precision crop protection,” Eur. J. Plant Pathol. 133(1), 197–209 (2012). WOS:000302232600013. [CrossRef]
9. F. Fiorani and U. Schurr, “Future scenarios for plant phenotyping,” Annu. Rev. Plant Biol. 64(1), 267–291 (2013). [CrossRef]
10. L. Li, Q. Zhang, and D. Huang, “A review of imaging techniques for plant phenotyping,” Sensors 14(11), 20078–20111 (2014). [CrossRef]
11. N. Hagen and M. W. Kudenov, “Review of snapshot spectral imaging technologies,” Opt. Eng. 52(9), 090901 (2013). [CrossRef]
12. S. Ono, “Snapshot multispectral imaging using a pixel-wise polarization color image sensor,” Opt. Express 28(23), 34536–34573 (2020). Publisher: Optical Society of America. [CrossRef]
13. E. Schonbrun, G. Möller, and G. D. Caprio, “Polarization encoded color camera,” Opt. Lett. 39(6), 1433–1436 (2014). Publisher: Optical Society of America. [CrossRef]
14. R. Habel, M. Kudenov, and M. Wimmer, “Practical Spectral Photography,” Comput. Graph. Forum 31(2pt2), 449–458 (2012). [CrossRef]
15. D. S. Jeon, S.-H. Baek, S. Yi, Q. Fu, X. Dun, W. Heidrich, and M. H. Kim, “Compact snapshot hyperspectral imaging with diffracted rotation,” ACM Trans. Graph. 38(4), 1–13 (2019). [CrossRef]
16. S.-H. Baek, I. Kim, D. Gutierrez, and M. H. Kim, “Compact single-shot hyperspectral imaging using a prism,” ACM Trans. Graph. 36(6), 1–12 (2017). [CrossRef]
17. J. W. Evans, “A birefringent monochromator for isolating high orders in grating spectra,” Appl. Opt. 2(2), 193 (1963). [CrossRef]
18. A. M. Title and W. J. Rosenberg, “Tunable birefringent filters,” Opt. Eng. 20(6), 206815 (1981). [CrossRef]
19. J. W. Evans, “Solc birefringent filter,” J. Opt. Soc. Am. 48(3), 142 (1958). [CrossRef]
20. O. Soyemi, D. Eastwood, L. Zhang, H. Li, J. Karunamuni, P. Gemperline, R. A. Synowicki, and M. L. Myrick, “Design and Testing of a Multivariate Optical Element: The First Demonstration of Multivariate Optical Computing for Predictive Spectroscopy,” Anal. Chem. 73(6), 1069–1079 (2001). Publisher: American Chemical Society. [CrossRef]
21. J. Kim, Y. Li, M. N. Miskiewicz, C. Oh, M. W. Kudenov, and M. J. Escuti, “Fabrication of ideal geometric-phase holograms with arbitrary wavefronts,” Optica 2(11), 958 (2015). [CrossRef]
22. J. Anandan, “The geometric phase,” Nature 360(6402), 307–313 (1992). [CrossRef]
23. B. D. Maione, C. Baldridge, and M. W. Kudenov, “Microbolometer with a multi-aperture polymer thin-film array for neural-network-based target identification,” Appl. Opt. 58(27), 7285–7297 (2019). [CrossRef]
24. M. A. Davenport, M. F. Duarte, M. B. Wakin, J. N. Laska, D. Takhar, K. F. Kelly, and R. G. Baraniuk, “The smashed filter for compressive classification and target recognition,” in Computational Imaging V, vol. 6498 (International Society for Optics and Photonics, 2007), p. 64980H.
25. M. Francon and S. Mallick, “Polarization interferomers: Applications in microscopy and macroscopy,” Wiley Interscience (1972).
26. D. H. Goldstein, Polarized light (Chemical Rubber Company, 2011).
27. M. W. Kudenov, N. A. Hagen, E. L. Dereniak, and G. R. Gerhart, “Fourier transform channeled spectropolarimetry in the MWIR,” Opt. Express 15(20), 12792–12805 (2007). [CrossRef]
28. D. Luo and M. W. Kudenov, “Neural network calibration of a snapshot birefringent Fourier transform spectrometer with periodic phase errors,” Opt. Express 24(10), 11266 (2016). [CrossRef]
29. B. D. Maione, D. Luo, M. Miskiewicz, M. Escuti, and M. W. Kudenov, “Spatially heterodyned snapshot imaging spectrometer,” Appl. Opt. 55(31), 8667 (2016). [CrossRef]
30. M. F. Møller, “A scaled conjugate gradient algorithm for fast supervised learning,” Neural Networks 6(4), 525–533 (1993). [CrossRef]
