1. Introduction

Passive polarimetric imaging has emerged in recent years as a useful sensing modality for detection and recognition [1]. The added information provided by polarization can aid in clutter reduction and object characterization. Imagery from microgrid polarimeters is obtained by incorporating a mosaic of pixel-wise micropolarizers on a focal plane array (FPA). Such imagers are akin to Bayer pattern color cameras, except the mosaic pattern is for polarization state, rather than color. With a typical microgrid polarimetric sensor, four polarimetric state images are acquired using a 2 × 2 pattern of polarizers. In this manner, each polarization image is obtained by subsampling the full FPA image by 2 in each dimension. Thus, the effective pixel pitch for each channel is twice that of the native FPA (1/2 the sampling frequency). Because of the desire for a wide field of view and other factors, many imaging systems are equipped with optics that allow some undersampling for a given FPA. By employing the microgrid with the same optics and FPA, undersampling and resulting aliasing artifacts are increased [2, 3].

Multi-frame super-resolution (SR) enhancement methods have been widely demonstrated to improve the effective sampling rate of undersampled imagers [4]. By using multiple low-resolution (LR) frames from video, these SR algorithms exploit sub-pixel motion between frames to effectively enhance the sampling rate for the native FPA. By doing so, these SR methods effectively trade temporal resolution for spatial resolution. In addition to enhancing the sampling rate and reducing aliasing, the SR algorithms typically perform restoration to reduce noise and blur. Generally speaking, iterative image reconstruction based SR methods are among the best performing algorithms, but come with a relatively high computational cost [4]. The simplest SR algorithms, in a computational sense, are the interpolation-restoration approaches [4]. Here we shall consider these two classes of SR algorithms as applied to polarimetric imagery. Because of their ability to help overcome undersampling, we believe multi-frame SR algorithms are a powerful match for microgrid polarimeters. To the best of our knowledge, the only publications to date exploring this connection were the result of preliminary work led by two of the current coauthors [5, 6]. We believe the current paper represents the first comprehensive treatment of the subject. In [5], Ratliff et al used the iterative multi-frame SR algorithm in [7] applied to the four demosaiced polarimetric channels independently. LeMaster explored a simple and fast multi-frame SR algorithm in [6] that registered and interlaced samples from multiple frames, also using independent channel processing. The work in [5, 6] demonstrated that SR can be used to enhance the resolution of polarimetric data from a microgrid imager, but neither fully exploited the redundant information available in the microgrid array.

Here we present two new multi-channel multi-frame SR algorithms specifically designed for processing imagery from microgrid polarimeters. The new SR methods are designed to provide improved performance by exploiting correlation between the polarimetric channels and by using a detailed observation model that includes an accurate point spread function (PSF). One of the two new SR algorithms presented here uses a form of regularized least squares (RLS) related to that in [7] and has an iterative solution. The proposed RLS algorithm is novel in that here we jointly estimate all of the polarimetric images and use a new regularization function that exploits correlation among the polarimetric channels. The other proposed algorithm is a new type of adaptive wiener filter (AWF) SR method [8] designed for microgrid image data. The microgrid AWF method leads to a faster implementation that is more readily suitable to real-time processing with appropriate hardware than the RLS method. We study the performance of the two new microgrid SR methods using imagery from a long-wave infrared (LWIR) microgrid polarimeter. We also present results using simulated microgrid data derived from a visible polarimetric imaging system. These simulated data are used to allow for a new quantitative performance analysis. Our results demonstrate that there can be significant benefits to using SR with microgrid polarimeters. We also show how multichannel SR processing that exploits the correlation between the polarimetric channels yields improved performance over independent channel processing.

The remainder of this paper is organized as follows. In Section 2, background on polarimetric imagery is briefly reviewed and the observation models are presented. The new microgrid RLS SR algorithm is presented in Section 3 and the new microgrid AWF SR method is presented in Section 4. Experimental results are presented in Section 5 to illustrate the efficacy of the proposed methods. Finally, we offer conclusions in Section 6.

2. Observation model

The sensor we are focusing on is equipped with four polarization analyzers oriented at angles θ ₁ = 0°, θ ₂ = 45°, θ ₃ = 90°, and θ ₄ = 135°. The microgrid configuration for acquiring these four polarimetric images is shown in Fig. 1. As shown, the microgrid image can be deinterlaced to produce the four individual polarimetric images. The choice of angles is designed to make the computation of Stokes vector images convenient [1, 9]. In general, let the ideally-sampled high-resolution (HR) polarimetric image, with P channels and N pixels per channel, be represented in lexicographical notation as p = [p ₁, p ₂,..., p_NP]^T. Breaking this down into individual polarimetric channels, we get $\mathbf{\text{p}}\hspace{0.17em}=\hspace{0.17em}{\left[{\mathbf{\text{p}}}_{{\theta }_1}^T,\hspace{0.17em}{\mathbf{\text{p}}}_{{\theta }_2}^T,\hspace{0.17em}\dots ,\hspace{0.17em}{\mathbf{\text{p}}}_{{\theta }_P}^T\right]}^T$, where p _θi = [p_{θi ,1}, p_θi,₂,..., p_θi,N]^T. For such a four-angle sensor, the first three Stokes images [1] can be estimated as

 

Fig. 1 Microgrid pattern for the polarimetric imager considered here. Four polarization images are acquired using a 2 × 2 repeating pattern of micropolarizers on the FPA.

Download Full Size | PDF

For multi-frame SR, let the number of observed LR microgrid frames be K. Let the LR observed pixels from frame k be expressed in lexicographical notation as g(k), where k = 1, 2,...,K. Breaking down each observed frame by polarimetric channel we get $\mathbf{\text{g}}(k)\hspace{0.17em}=\hspace{0.17em}{\left[{\mathbf{\text{g}}}_{{\theta }_1}^T\hspace{0.17em}(k),\hspace{0.17em}{\mathbf{\text{g}}}_{{\theta }_2}^T\hspace{0.17em}(k),\hspace{0.17em}\dots ,{\mathbf{\text{g}}}_{{\theta }_P}^T\hspace{0.17em}(k)\right]}^T$, where g _θi (k) = [g_θi,₁ (k), g_θi,₂ (k),..., g_θi,_M (k)]^T. Let the complete set of all observed pixels in lexicographical notation be specified by g = [g ^T (1), g ^T (2),...,g ^T (K)]^T = [g ₁, g ₂,..., g_KMP]^T.

A block diagram illustrating the observation model relating p to the observed frame g(k) is presented in Fig. 2. The ideal polarimatric data in p is assumed to be sampled above the Nyquist rate. The motion between frames is parameterized with the vector s(k). Next, the model includes a linear shift invariant blurring using the discrete PSF for the optical system. This is followed by down-sampling by a factor of L. Finally, the down-sampled images are mosaiced into the pattern in Fig. 1 and noise is introduced to obtain g(k). This observation model can be mathematically expressed such that each observed pixel is a weighted sum of the ideal pixels in p plus noise. The particular weights depend on the system PSF, the motion between frames, and the microgrid layout. In particular, the LR pixels can be expressed in terms of ideal HR pixels as follows

 

Fig. 2 Block diagram illustrating the observation model relating p to the observed frames g(k), for k = 1,2,...,K.

Download Full Size | PDF

The observation model in Eqs. (3) and (4) will be used for the RLS SR algorithm. However, to aid in the development of the fast AWF algorithm here, we consider an alternate observation model based on that in [8]. The alternate observation model is shown in Fig. 3. Here the motion model is commuted with the PSF and integrated into what becomes a nonuniform sampling operator. This alternative observation model is equivalent to the model in Fig. 2 for translational motion. It is also equivalent for rotational motion when the PSF is circularly symmetric [8]. The power of this model is that it leads to the computationally simple interpolation-restoration SR algorithms [8]. Based on this model, nonuniform interpolation can be used to recover noisy, but uniformly spaced, samples of f (x,y,θ_i) from g. Then, restoration is used to estimate the corresponding samples of p(x,y, θ_i).

 

Fig. 3 Alternative observation model valid for translational motion (and rotational motion when the PSF is circularly symmetric). Here the motion model is commuted with the PSF and integrated into what becomes a nonuniform sampling operator.

Download Full Size | PDF

Let us now consider the modeling of the system PSF. We model the optical transfer function (OTF) with three components

The microgrid polarimetric imager used here has the following specifications. It has a spectral bandwidth of λ = 7.8 – 9.8 μm. We use a wavelength of λ = 9 μm for our PSF model. The system uses F/2 optics and has a full FPA pixel pitch of 25 μm with approximately 100% fill factor rectangular detectors. Note that for this microgrid imager, the pixel pitch for pixels of like polarization is actually 50 μm. Thus, the effective active area for the like-polarization sub-array detectors is modeled as spanning half the 50 μm like-polarization pixel pitch in each dimension. This equates to a 25% fill factor (taking into account the skipping of detectors in the FPA as shown in Fig. 1). For abberation modeling, we use W_RMS = 1/14 [11]. Cross sections of the relevant 2-D modulation transfer functions (MTFs) are shown in Fig. 4(a) for this system. The continuous PSF is shown in Fig. 4(b). The discrete impulse invariant PSF, for use in the observation model operating on the HR image, can be found by sampling the continuous PSF with a sampling period of 50/L μm.

 

Fig. 4 (a) Cross sections of theoretical 2-D MTF and its components for the microgrid polarimetric imager used here. (b) Theoretical continuous PSF for L = 4.

Download Full Size | PDF

Note that the Nyquist or folding frequency (i.e., 1/2 sampling frequency) for a single channel is 10 cycles/mm, as shown in green in Fig. 4(a). This is well below the 55.56 cycles/mm cut-off frequency of the optics. Any scene frequency content above the Nyquist frequency is “folded” into lower frequencies, creating aliasing artifacts. Thus, this system is likely to exhibit significant aliasing without SR. Another point to note from the overall MTF is that it has a low pass blurring effect. However, the blurring cannot be combatted with a linear shift-invariant filter without first (or jointly) addressing the undersampling issue. Otherwise, the aliasing artifacts will simply be amplified. Based on the overall system MTF shown in Fig. 4(a), we believe that L = 4 is a practical choice for SR processing for this sensor. The effective sampling frequency is then 80 cycles/mm, with a Nyquist frequency of 40 cycles/mm as shown in yellow in Fig. 4(a). In addition to aliasing reduction, we also hope to restore the imagery from the blurring effects of the overall MTF.

3. RLS SR for microgrid polarimeters

Based on the observation model depicted in Fig. 2 and detailed in Eq. (3), we propose an RLS estimator for p. The output of the RLS algorithm is given by

While this RLS approach is based on that in [7], what makes Eq. (9) novel are the regularization terms. The RLS algorithm in [7] uses only one regularization term, a smoothness term for the single channel data. Here we have multichannel data with physical constraints, like the redundancy exploited in the third term in Eq. (9). Note also, that the smoothness terms are applied here to the Stokes images, and not the original polarimetric channels. This is done for several reasons. First, the Stokes images may be a final product for such an imaging system. By having smoothness terms applied here, we better ensure that this product will be free from high frequency artifacts. Furthermore, {s _{1, j}} and {s _{2, j}} are known to generally have relatively low energy (i.e., low variance). Thus, applying a term favoring small high-frequency energy in these channels is sensible. The Stokes image {s _{0, j}} is the total intensity image. This should contain most of the energy. One may choose to use the smoothness constraint on this term minimally by employing a high σ_{s 0}. The parameters σ_{s 1} and σ_{s 2} may be smaller, because of the expected lower energy in these images. Finally, because the Stokes terms interrelate the different polarization channels, it allows us to exploit their distinct spatial sampling positions better than if we applied a smoothness constraint on each channel independently. The approach of applying smoothness regularization to the Stokes images in polarimetric data may be viewed as akin to using luminance and chrominance coordinates in color processing [12–14]. It is interesting to note that the RLS estimate can be viewed as a maximum a posteriori (MAP) estimator in the case of independent and identically distributed Gaussian noise [15]. In the MAP framework, the quadratic nature of the regularization terms in Eq. (9) can be seen as resulting from a Gaussian prior pdf.

To minimize the cost function in Eq. (9), one may use a gradient descent or conjugate gradient algorithm similar to that presented in [7]. Here we develop the somewhat simpler gradient descent approach. We initialize the RLS estimation process by registering all K frames to a common reference. A convenient choice of reference is the most recent observed frame. Note that we need to register the frames to subpixel accuracy. We have found that we get better registration performance by deinterlacing the channels, applying the registration algorithm in [7], and then averaging the registration parameters obtained from the four channels. With this and the PSF from Section 2, we have a complete observation model. Next, we take the reference image and use single frame interpolation to obtain an initial estimate of p, denoted p̂ ⁰. To improve this estimate using gradient descent, we need the gradient of Eq. (9) with respect to p, denoted ∇_p C(p). This gradient is composed of the partial derivative of the cost function with respect to each of the HR samples in p. This can be expressed as

The optimum step size at each iteration can be found by optimizing C(p̂ ⁿ) as a function of ɛⁿ, given p̂ ^{n −1}. In particular, we set the derivative of C(p̂ ⁿ) with respect to ɛ ⁿ equal to zero, and solve for ɛⁿ. This calculation results in an optimum step size of the form

4. AWF SR for microgrid polarimeters

Unlike the RLS, the AWF SR method [8] is non-iterative and has a potentially faster implementation than the RLS. Here we propose a new type of the AWF SR algorithm [8] for use with a microgrid imager. As in [8], the microgrid AWF uses a moving observation window on the HR grid as shown in Fig. 5. The observation window is shown for L = 4 and is populated with pixels from a single frame for simplicity. With multiple frames of data with motion between them, the observation window will contain additional sets of pixels from each polarization state, positioned according to the motion. In general, let the observation window span W_x HR pixel spacings in the horizontal direction and W_y HR pixel spacings in the vertical direction. All of the LR pixels of any polarization that lie within the span of this observation window are placed into an observation vector g _i = [g_{i ,1}, g_{i ,2},..., g_i,Ji]^T, where i is the window positional index and J_i is the number of LR pixels within the window span. The samples within the observation window will be used to estimate the HR pixels within the generally smaller D_x × D_y sample estimation window at the center of the observation window, as shown in Fig. 5. Thus, the AWF SR method is effectively performing nonuniform interpolation and restoration simultaneously. One of the main differences here, versus the AWF in [8], is that the observation window contains LR pixels from multiple polarizations. Furthermore, at each HR pixel location, we must estimate all of the polarization states, not just a single value.

 

Fig. 5 Observation window for the microgrid AWF algorithm for polarimetric data with L = 4. For simplicity, the window is shown populated with pixels from only a single frame.

Download Full Size | PDF

The HR pixel estimates for the samples within the estimation window are obtained using a weighted sum of the LR pixels in the observation window. This is expressed as

We wish to employ the minimum mean squared error (MSE) filter weights in Eq. (31). The additional challenge we face when processing microgrid data is that we must not only consider the spatial arrangement of samples in a given observation window, but also their polarization state. Given a correlation model for the observed samples, the minimum MSE weights [8] are given by

To fill the matrices in Eqs. (33) and (34), we need a statistical model to provide the needed correlations. Consider a correlation model, specifically designed for microgrid data, that incorporates knowledge of the spatial separation between samples, and the polarization state of each. For the desired data, we denote this correlation function as r_dθdϕ (x,y). The variables θ and ϕ represent the polarization angles of the two samples involved, and x,y represents the spatial separation. Here we propose the following model

Given r_dθdϕ (x,y), it can be shown that the cross-correlation function between p(x,y, θ) and f (x,y, ϕ), as shown in Fig. 3, can be expressed as

As described in [8], there are circumstances where one only needs to compute one set of weights for processing all the observation windows for an entire output frame. This allows for very fast processing. This fortunate case occurs when we have global translational motion and the statistical parameters are assumed to be constant across the image. With global translational motion, the same spatial pattern of LR samples is observed in each observation window, so long as the observation and estimation window sizes are an integer multiple of L. If we have bad pixels, we can simply exclude those from the observation vector g _i, so that they do not contribute to the output. However, this requires that a distinct set of weights be computed around any bad pixels, as it changes the spatial distribution of samples in the observation window. If processing speed is critical, a bad pixel replacement algorithm can be employed prior to AWF filtering [16]. This faster approach is how we address bad pixels for the AWF SR results presented in Section 5.

An example of some polarimetric AWF filter weights is shown in Fig. 6. The brightest pixel corresponds to a weight of 1.41 and the darkest is −0.17. The remaining weights are scaled linearly. For reference, note that the perimeter weights are very close to zero. These weights are computed for K = 1 frame, P = 4 polarimetric channels, and an upsampling factor of L = 2. Thus, we assume we have a single microgrid frame and we form an estimate of each polarization at each FPA location. This can be viewed as demosaicing with restoration. This simple case is shown to make interpreting the weights more straightforward than with mulitple frames and/or more upsampling. In computing these weights, we assume the noise variance is ${\sigma }_n^2\hspace{0.17em}=\hspace{0.17em}1$ and ${\sigma }_p^2\hspace{0.17em}=\hspace{0.17em}100\hspace{0.17em}$. The observation window size is W_x = W_y = 10 and the estimation window size is D_x = D_y = 2. The weights correspond to estimating the upper left pixel in the estimation window. This location is highlighted with a plus sign in Fig. 6. Each of the four columns in Fig. 6 corresponds to a different polarization sample being estimated for the one spatial location, and each row corresponds to using a different α in computing the weights. In particular, the columns from left to right correspond to polarization angles of θ = 0°, 45°, 90° and 135°. The rows from top to bottom correspond to α = 0, 0.7, 1.0. Note that when α = 0 (top row), the weights are only nonzero for pixels on the microgrid of the same polarization as the output. On the other extreme, when α = 1 (bottom row), the weights treat all pixels on the migrogrid array as equivalent, ignoring their polarization differences. Thus, the output for all polarizations is formed with the same weights, yielding the same output. This would be appropriate when the DoLP is assumed to be zero, and all polarization channels should be equal. The case where α = 0.7 (middle row) yields an interesting result, where samples from all polarizations are combined to form each output, especially when estimating at a location where that polarization is not measured (columns 2–4). The advantage of using samples with polarizations other than the one being estimated is that they offer spatial locations not represented by pixels of like polarization. Spatially close pixels of a different polarization may be more correlated to the sample we are estimating than ones farther away that have the same polarization. Thus, using α, we control the weight given to samples with polarizations different than the polarization being estimated relative to those of like polarization. When multiple frames are present, the picture is more intricate as samples of various polarizations are distributed potentially nonuniformly throughout the observation window. However, the principle is the same where α balances spatial correlation with polarization correlation.

 

Fig. 6 AWF filter weights for K = 1 frame, P = 4 channels, W_x = W_y = 10 observation window, D_x = D_y = 2 estimation window and L = 2 upsampling (single frame demosaicing with restoration). The weights correspond to estimating the pixel location with the red plus sign. The columns from left to right correspond to polarization angles of θ = 0°, 45°, 90° and 135°, respectively. The rows from top to bottom correspond to α = 0, 0.7, 1.0.

Download Full Size | PDF

We have found that in some cases we can improve the microgrid AWF results by preprocessing the microgrid to compensate for disparate local statistics of the individual channels. In particular, if we subtract the local means and divide by the local standard deviations of the individual channels, they tend to exhibit greater correlation. This allows us to use a larger α and better exploit the spatial sampling diversity provided by the microgrid as shown in Fig. 1. To better describe this method, let a(i, j) represent a sample from the microgrid array, where i = 1,2,...,M is the 2 × 2 super-pixel index and j is the polarization sample index within the super-pixel. Let j = 1,2,3,4 correspond to θ = 0°, 45°, 90⁰, 135°, respectively. Local means, denoted ā(i, j), are estimated by deinterlacing the channels and applying a Gaussian low-pass filter with standard deviation of σ pixels. Local standard deviations, denoted ã(i, j), are estimated by subtracting the local mean and then using a Gaussian low-pass filter on the squared difference, and taking the square root of the result. In terms of these variables, the preprocessing step is expressed as

5. Experimental results

In this section, we present a number of experimental results to compare and contrast the performance of the AWF and RLS microgrid SR methods with each other and with other single frame methods. The first set of results are from the LWIR microgrid polarimetric imager described in Section 2. We also consider data from a visible polarimetric sensor, where we simulate microgrid sampling. These data allow for quantitative error analysis.

5.1. LWIR microgrid polarimetric data

The first set of results are shown in Fig. 7 and continue in Fig. 8. A single raw LWIR microgrid image is shown in Fig. 7(a). The sensor has a number of bad pixels including a stripe of bad pixels near column 120. The bad pixel map is shown as a red contour in Fig. 7(a). The remaining images in Figs. 7 and 8 are various estimates of p ₀ for L = 4. In particular, the result of using single-frame channel-independent bicubic interpolation is shown in Fig. 7(b). The output using the single-frame frequency-domain method by Tyo et al [3], which is built upon the work in [2], is shown in Fig. 7(c). The output for the single-frame LSF method from Eq. (40) with σ = 1 is shown in Fig. 7(d). The microgrid AWF SR method using only a single frame with ρ = α = 0.7 and σ = 1 produces the result shown in Fig. 7(e). The microgrid RLS SR method using only a single frame with ${\sigma }_n^2\hspace{0.17em}=\hspace{0.17em}1$, ${\sigma }_{s_0}^2\hspace{0.17em}=\hspace{0.17em}100$, and ${\sigma }_{s_1}^2\hspace{0.17em}=\hspace{0.17em}{\sigma }_{s_2}^2\hspace{0.17em}=\hspace{0.17em}{\sigma }_r^2\hspace{0.17em}=\hspace{0.17em}10$ produces the result shown in Fig. 7(f). The selection of the cost function parameters has been informed by quantitative analysis using simulated data as well as subjective evaluation. Note that the bicubic image is quite blurred and shows aliasing artifacts. These artifacts are most pronounced on the starboard wing and vertical stabilizer of the aircraft. Here the leading edges of these surfaces appear jagged. The Tyo and LSF methods do appear to reduce some of the more obvious aliasing artifacts. They do this by exploiting the sampling diversity of all polarizations on the microgrid array. The AWF and RLS outputs using a single frame appear to show further improvements as they also provide some deconvolution of the system PSF.

 

Fig. 7 LWIR microgrid imagery results. (a) Raw microgrid image with bad pixel mask shown in red. Estimates of p ₀ using an upsampling factor of L = 4 are shown for (b) bicubic interpolation of polarimetric channel g ₀(1) (c) Tyo et al method from [3] (d) LSF method (e) single frame AWF with correlation model parameters of ρ = α = 0.7 (f) single frame RLS.

Download Full Size | PDF

 

Fig. 8 LWIR microgrid imagery results continued. (a) Single frame AWF with α = 0.0 (b) single frame AWF with α = 1.0 (c) independent-channel single-frame RLS (d) independent-channel 20 frame RLS (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Download Full Size | PDF

To see the impact of α on the microgrid AWF SR method, we show the single frame result using α = 0.0 and α = 1.0 in Figs. 8(a) and 8(b), respectively. Note that for α = 0.0, the polarimetric channels are treated independently [8] and the result is similar to that of bicubic interpolation. Using α = 1.0, the AWF treats all polarimetric channels as being the same. While this method can sometimes produce a reasonable {s _{0, j}} image, the {s _{1, j}} and {s _{2, j}} Stokes images are zero or have only low spatial frequency content from the local statistics processing in Eq. (39). Note that cross-hatching artifacts can be seen in Fig. 8(b), especially near the cockpit window at the front of the aircraft. These types of artifacts tend to become significantly more pronounced with α = 1.0 when more input frames are used. The single-frame RLS algorithm operating on the polarimetric channels independently [5] is shown in Fig. 8(c). Again, the output looks similar to bicubic interpolation as only samples of one polarization are used. Using the independent-channel RLS with 20 input frames gives rise to the output shown in Fig. 8(d). Here the results are much better. Given a large number of input frames, one can often get a good result using only samples of polarization matching that of the output. The 20 frame microgrid AWF output with ρ = α = 0.7 and σ = 9 is presented in Fig. 8(e). Finally, the 20 frame microgrid RLS output with ${\sigma }_n^2\hspace{0.17em}=\hspace{0.17em}5$, ${\sigma }_{s_0}^2\hspace{0.17em}=\hspace{0.17em}100$, and ${\sigma }_{s_1}^2\hspace{0.17em}=\hspace{0.17em}{\sigma }_{s_2}^2\hspace{0.17em}=\hspace{0.17em}{\sigma }_r^2\hspace{0.17em}=\hspace{0.17em}10$ are shown in Fig. 8(f). The noise variance term is increased here to add robustness to possible registration errors. These multiframe results appear to show the best detail on the roofs of the buildings in the foreground. Many diagonal lines that appear in these images look aliased in most of the other images.

To illustrate the typical convergence behavior of the microgrid RLS algorithm, the cost function and its components are shown in Fig. 9 for the single frame LWIR result in Fig. 7(e). Note that the total cost decreases smoothly with gradient-descent iteration. The observation model (or data) term also declines with iteration. The Stokes regularization terms actually increase with iteration. This is because the image estimate becomes sharper than the interpolated starting image as the iterations progress. These terms help keep the final estimate from becoming overly noisy. Note that the constants in the cost function can be modified to give more or less weight to any of these components in order to shape the characteristics of the final solution.

 

Fig. 9 Microgrid RLS SR algorithm cost function and its components for the single frame LWIR result in Fig. 7(e).

Download Full Size | PDF

The DoLP images calculated from various estimates of p are shown in Fig. 10. Since DoLP involves all of the polarimetric channels, these results reveal characteristics of the estimates in all channels. DoLP also tends to expose sampling errors rather dramatically because it is made up of image differences, as can be seen in Eq. (2). The DoLP images for bicubic interpolation, the Tyo method, and the LSF method are shown in Figs. 10(a)–10(c), respectively. The DoLP image for the independent channel RLS method [5] using 20 frames is shown in Fig. 10(d). Finally, the DoLP images for 20 frame microgrid AWF SR and 20 frame microgrid RLS SR are shown in Figs. 10(e) and 10(f), respectively. Based on subjective analysis, we believe that the multiframe microgrid AWF and RLS methods appear to produce the sharpest results with the least aliasing.

 

Fig. 10 DoLP images for LWIR microgrid data. (a) Bicubic interpolation (b) Tyo et al (c) LSF method (d) independent-channel 20 frame RLS (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Download Full Size | PDF

5.2. Visible polarimetric data with simulated microgrid sampling

To allow for quantitative analysis with truth imagery, we use a visible imaging system equipped with a linear polarization filter. The camera is an Imaging Source DMK 21BU04. This is a 640 × 480 8-bit grayscale camera with a Sony ICX098BL CCD sensor with 5.6μm detectors. The camera is fitted with a Computar 5mm F/1.4 lens. Full frames at each of the four polarization angles are acquired of an outdoor scene containing two vehicles. The polarization filter is rotated in 45° increments between acquisition to provide the four full-frame polarization images. We define these data to be the true p. These data are then put through the observation model with L = 4 and a noise variance of 1. We use a set of randomly generated shifts and the PSF in Fig. 4 to create 20 simulated microgrid images. The Stokes intensity image results for these data are shown in Fig. 11. In particular, one of the simulated microgrid images is shown in Fig. 11(a). The corresponding true {s _{0, j}} Stokes image is shown in Fig. 11(b). Bicubic interpolation yields the result presented in Fig. 11(c). Note that the bicubic interpolation result looks blurred relative to the true image. Since {s _{0, j}} is an average of the four polarimetric channels, some of the aliasing artifacts that are more clearly seen in the individual polarimetric channels are reduced here. The output of the Tyo method is shown in Fig. 11(d) and the 20 frame microgrid AWF output with ρ = α = 0.7 and σ = 9 is shown in Fig. 11(e). Finally, the 20 frame microgrid RLS output with ${\sigma }_n^2\hspace{0.17em}=\hspace{0.17em}1$, ${\sigma }_{s_0}^2\hspace{0.17em}=\hspace{0.17em}100$, and ${\sigma }_{s_1}^2\hspace{0.17em}=\hspace{0.17em}{\sigma }_{s_2}^2\hspace{0.17em}=\hspace{0.17em}{\sigma }_r^2\hspace{0.17em}=\hspace{0.17em}10$ is shown in Fig. 11(f).

Cross-sections of the true image {s _{0, j}} from Fig. 11 as well as several of the corresponding estimate images along Row 234 and about Column 67 are shown in Fig. 12. These cross-sections are centered about a dark line in the sidewalk concrete, providing us with approximate line spread functions. These provide some insight into the resolution characteristics of the various image estimates. Here it appears that the microgrid RLS provides the narrowest line spread, followed closely by the microgrid AWF. The single frame methods clearly have much broader line spread functions. This is to be expected as these methods do not include any deblurring. It is interesting to note that the Tyo method does appear to have a more favorable line spread function than bicubic interpolation.

 

Fig. 11 Visible polarimetric data with simulated microgrid sampling. (a) Simulated microgrid frame (b) true {s _{0, j}} image for an upsampling factor of L = 4 (c) estimate using bicubic interpolation (d) Tyo et al method (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Download Full Size | PDF

 

Fig. 12 Cross-sections of the true image {s _{0, j}} and corresponding estimates along Row 234 and about Column 67 from Fig. 11. These plots represent approximate line spread functions and provide some insight into the resolution of the various estimates.

Download Full Size | PDF

The corresponding DoLP images for the key methods operating on the simulated microgrid data are provided in Fig. 13. The true DoLP image is shown in Fig. 13(a). The DoLP generated with bicubic interpolation is shown in Fig. 13(b). Here, the aliasing artifacts are exposed, especially along edges in the scene content. The DoLP images for Tyo and LSF are shown in Figs. 13(c) and 13(d), respectively. The DoLP image for the 20 frame microgrid AWF is shown in Fig. 13(e) and the DoLP image for the 20 frame microgrid RLS is shown in Fig. 13(f). Based on subjective analysis, we believe that the multiframe AWF and RLS methods appear to provide the best estimates of the true Stokes intensity image and DoLP image.

 

Fig. 13 DoLP images for simulated microgrid data. (a) True DoLP image (b) bicubic interpolation (c) Tyo et al method (d) LSF method (e) 20 frame microgrid AWF SR with ρ = α = 0.7 (f) 20 frame microgrid RLS SR.

Download Full Size | PDF

The MSEs between the true Stokes image and the various estimates are shown in Fig. 14 as a function of the number of input frames. In particular, the MSEs for {s _{0, j}}, {s _{1, j}}, and {s _{2, j}} are shown in Figs. 14(a)–14(c), respectively. Note that the Tyo and LSF methods do outperform bicubic interpolation, as expected. Furthermore, the single-frame microgrid AWF with α = 0.7 and the microgrid RLS methods do better yet. We believe this is in part due to the fact that these methods incorporate knowledge of the PSF and perform some deblurring. As more input frames are used, the microgrid AWF and RLS SR estimates improve, with diminishing returns as the sampling grid gets well populated by numerous shifted frames. Note also that the α = 0.7 result outperforms the independent channel AWF (α = 0.0). The multichannel RLS also outperforms its independent channel counterpart [5]. It is also interesting to note that the AWF with α = 1.0 provides a relatively low MSE on {s _{0, j}}, but does a poor job with {s _{1, j}} and {s _{2, j}} that does not improve with input frames. This is because with α = 1.0, the {s _{1, j}} and {s _{2, j}} estimates are nearly zero, save for the impact of the local statistics processing beginning with Eq. (39). These results suggest that when estimating {s _{0, j}} with a relatively small number of frames, it is beneficial to assume more correlation between channels (i.e., use a higher α) to more fully exploit the spatial sampling diversity offered by the different channels. However, this comes at the expense of the polarization content in {s _{1, j}} and {s _{2, j}}. We also see that when enough frames are available, the benefit of multichannel processing is somewhat reduced, due to the abundance of spatial samples of each polarization.

 

Fig. 14 MSE versus number of input frames for the Stokes images from the simulated microgrid imagery (a) {s _{0, j}} (b) {s _{1, j}} (c) {s _{2, j}}.

Download Full Size | PDF

On whole, the MSE results show that the RLS outperforms the AWF. We believe this is in part due to the finite window size of the AWF as well as the natural way in which the RLS incorporates the redundancy constraint in the cost function. However, the AWF offers a significant computational savings and yields a very competitive result with similar MSE. To illustrate the computational savings, consider the following run times using MATLAB on a PC with an Intel Core i7 CPU with 3.07GHz processor. For the 20 frame AWF, the run time was 0.31s per output frame. For the 20 frame RLS, the run time was 82.46s per output frame. For reference, the run times per output frame were 0.093s, 0.071s, and 0.133s for bicubic, Tyo, and LSF, respectively.

Finally, the MSE results in Fig. 15 show how the α parameter impacts the MSE performance of the microgrid AWF filter on the three Stokes images. These results are for the 20 frame microgrid AWF output with ρ = 0.7 and σ = 9. Note that the MSE on {s _{0, j}} is minimized with an α close to, but distinctly less than, one. The MSE on the other Stokes images is obtained with an α near 0.5. Again, this shows us that if one assumes too much correlation between channels (i.e., high α), the pure polarimetric content in {s _{1, j}} and {s _{2, j}} is degraded. On the other hand, assuming too little correlation predominantly hurts our estimate of {s _{0, j}}. We have found a good compromise in the value used in our earlier results of α = 0.7.

 

Fig. 15 MSE versus α for the 20 frame microgrid AWF output with ρ = 0.7.

Download Full Size | PDF

6. Conclusions

Microgrid imaging systems are likely to suffer from undersampling to a greater extent than their single-channel counterparts. We have presented two new SR algorithms designed for the restoration of microgrid polarimetric data in the presence of aliasing, blur and noise. These are the microgrid RLS and microgrid AWF algorithms. We have also presented the simple LSF demosacing algorithm based on local statistics. Each in their own way, the new microgrid SR methods exploit the sampling diversity offered by the microgrid pattern and correlation between different polarization channels. Unlike simpler independent-channel methods, the new microgrid algorithms use samples from all polarizations to estimate each output at each polarization. This allows the new microgrid algorithms to outperform their independent channel counterparts. By using multiple frames, when available, we further exploit sampling diversity provided by relative motion between the scene and the FPA. Even with a single frame, the AWF and RLS provided lower MSE than the other single frame methods tested here. As the number of input frames increases, the algorithms perform even better. We did observe that independent-channel SR processing can do reasonably well when a large number of input frames are available. However, the microgrid approach using all the polarizations is critical to combatting aliasing with a relatively small number of input frames. Operating with a small number of input frames is desirable in many applications because we are less likely to encounter scene motion, motion parallax, or other changes that make registration more difficult. Finally, the microgrid AWF SR method has a low computational complexity for translational motion. It is much faster than the RLS in that case, but enjoys a very similar performance, as seen in the subjective and quantitative results presented here.