Abstract
High-accuracy spot target detection based on a complementary metal-oxide semiconductor (CMOS) image sensor, such as astronomy magnitude, medicine, and astronomy photometrics, needs accurate pixel response. Because pixels have different silicon structures and read outputting, each pixel has non-uniformity response with specific illumination. The flat-field correction of a CMOS image sensor is crucial before image processing. In this work, a flat-field model and correction method based on spot scale areas of CMOS image sensor pixel response are proposed. Compared with traditional full-plane calibration, this method aims at spot areas to fit most selected normal pixels’ mean response curve with different light intensities and exposure times, which can guarantee spot imaging areas with higher accurate pixel response. Finally, the accuracy of this flat-field correction method is evaluated by the influence on spot target extraction accuracy. The experimental results indicate that using this flat-field correction method can decrease the non-uniform variance from 7.34 (LSB/10 bit) to 1.91 (LSB/10 bit) (improved by 74.1%) and reduce the noise effect on spot extraction accuracy, which improves it from 0.3453 pixel to 0.0116 pixel (). The proposed approach solves the problem of non-uniform pixel response and improves imaging SNR for high-accuracy spot target localization.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. INTRODUCTION
High-accuracy spot target imaging detection is the key technology for applications such as Gaia satellite star precise positioning [1], medical target imaging location [2], and astronomical photometry [3]. The positioning accuracy of these optical applications directly depends on the accuracy of pixel response of the image sensor. With the different silicon structures and readout noise existing between different pixels [4], the spatial non-uniform noise is generated in imagers under the same illumination, which limits further improvement of spot localization accuracy.
Recently, with the development of photoelectric technology, the CMOS active imaging system has advantages of low power consumption, high dynamic performance, and high functional density ratio—the reason it has replaced the traditional CCD imaging system in many cases of detecting spot targets. One of the typical applications is the star tracker system [5,6]. But unlike the unified amplification of CCD imaging systems, there are different fixed pattern noises (FPNs), typical noise, and random noise in CMOS imaging, which brings out the pixel response non-uniformity to affect the image quality. Although some noise such as single-point salt-and-pepper noise and white noise can be filtered out in subsequent processing algorithms, if the noise exists in spot target extraction areas, it cannot be removed by filters and can cause spot location deviation at the same time. For this reason, a pixel response non-uniform correction method based on laboratory illumination is put forward to effectively solve the problem of image sensor non-uniform response during dark and long exposure time spot target imaging processes. In addition, this approach with more convenient operation and easier calculation is different from the traditional large-area calibration, which has problems of inhomogeneous illumination and large computation data [7].
In this paper, an active pixel sensor (APS) CMOS star tracker is researched as the typical application of spot target localization. FPN is the principal part of non-uniform response that can be measured and changes as the exposure time or light intensity changes. In 2005, the German Jena-optronik company used the infinite impulse response (IIR) filter method to reduce FPN noise in the star tracker; this method utilizes the previous frame response to conduct moving average filtering for the new frame response in the scope of object extraction, which leads to existing residual noise information. The noise reduction effect of this method is limited. This method improves the star point localization accuracy only from 0.48 to 0.13 pixel () [8]. In 2013, Eduardo used a quadratic polynomial model calibration method to reduce CMOS image sensor FPN pixel, and the Pearson correlation coefficient was used to evaluate the accuracy of the fitting model. This method has a 99.95% fit Pearson coefficient of flat-field calibration [9]. In 2016, Fiuz aimed at star tracker imaging system applications to establish a CMOS column amplifier noise model and proposed a fitting calibration correction method based on a linear model. The global response variance of column pixels reduced from 0.2758 to 0.043 (LSB/12bit) [10]. Michael and others respectively conducted the CMOS image sensor noise experiment to analyze the fitting variance of noise of the linear pixel response model, piecewise linear model, and quadratic model [11,12]. The results show that piecewise linear is the better model function of CMOS image sensor FPN noise fitting. From the above, for elimination of CMOS FPN and non-uniform pixel response, there are mainly two methods, including filtering and calibration correction. The filter method cannot completely conduct noise filtering in the spot target extraction region, so the better way to reduce non-uniformity of pixel response is the calibration correction method.
This paper proposes an image sensor non-uniformity flat-field calibration method for high-accuracy spot target localization that can effectively eliminate the non-uniformity response of image sensors and improve spot positioning accuracy from 0.1 pixel to 0.01 pixel.
2. CMOS IMAGE SENSOR NOISE ANALYSIS AND PIXEL RESPONSE MODEL
In this section, the non-uniformity calibration method for a CMOS image sensor’s pixel response and the pixel response model are proposed. The calibration procedures are based on the established model. The influence of non-uniformity pixel response in star spot extraction is described in Fig. 1. This shows the necessity and advantage of the proposed calibration method compared with the traditional filtering method.
A. Pixel Response Noise Analysis and Correction Modeling
First, the CMOS image sensor pixel response model is established to conduct the flat-field calibration correction function. For the ideal CMOS image sensor, the pixel output (e) with the light intensity signal is proportional to the relationship (e/unit light intensity) as
However, the actual CMOS image sensor pixel response includes two parts of noise [13–15] that lead to deviation from ideal response, and it can be described as is the FPN, which can be predicted by measuring experiment, and is the random noise that is different between each frame and cannot be predicted. In this paper, the FPN part is measured by the proposed method, and the random noise can be reduced only through the subsequent filter. can be given as contains mainly three parts (the temperature is fixed as 25°C; is the pixel position) [16]:- (1) : CMOS image sensor offset noise, which relates to the internal amplifier technology structure and changes with the temperature . The typical offset noise is the temperature noise caused by the random motion of free electrons in the resistance : where is the Boltzmann constant, is the Kelvin temperature, is the noise equivalent bandwidth, and is the channel resistance.
When the temperature is fixed, the measurable offset noise can be expressed by
- (2) : dark current noise is generated from non-uniform dark current . Crystal material surface defects produce leakage current and intrinsic dark current that changes with the crystal structure, exposure time , pixel area , unit electronic , analog gain level , band gap , impurity energy gap , Kelvin temperature , Boltzmann’s constant , and scaling factor for current generation for each pixel . The dark current model can be given as According to Eq. (6), the dark current can be uniformly measured when the parameters of the imager are firm. But an actual CMOS detector has lag, column mode noise, and semiconductor materials that lead to the non-uniformity of pixel response. For this reason, is different between pixels and is no longer a direct proportion function and can be described as
We set light intensity signal as fixed and change exposure time within short exposure time (1 ms–100 ms) so that the nonlinearity is not obvious [19]. A linear function is used to model then in part 3, which will be proved reasonable as
In conclusion, the flat-field calibration method is proposed in this paper; the noise pixel response measurement model is established for a linear functional form [Eqs. (5), (7), and (9)] as From Eq. (10), a linear model is established for FPN, and a calibration model can be given as Equation (11) is the proposed calibration model to complete the non-uniform correction. First, through changing the light intensity, the non-uniform response is calculated point to point by the standard pixel response , which is calculated by pixels area of the correction pixel position. Then the gain of can be obtained from the response. Second, the corresponding pixels and are calibrated by the standard pixel response and the non-uniform response . Finally, the corrected non-uniform response corresponds to the standard .B. Correction Area Selection and Pixel Classification
The general centroid algorithm is the gray centroid method using the specific calculation window. The calculated target signal is obtained from total energy, which removes the background value. Due to the optical system error (spherical aberration, comatic aberration, color aberration, field curvature) and other errors (imager systemic error), the responses of the light intensity and background in the different pixel areas of the imager plane are non-uniform [19,20]. Hence the selection method of a non-uniform calibration window for specific FPN pixel position is put forward to calculate the correction standard of different areas. It is difficult to handle huge amounts of total pixels with limited storage space, so pixel classification needs to be discussed to find the preferential correction pixels.
If the correction area is chosen too large, the correction standard will be inaccurate for the calibration pixel point. Therefore, the size of the area close to the calculated window of the gray centroid method is appropriate.
The static star spot extraction maximum area is usually . But when the satellite moves, the star spot image will spread more than pixels area [21,22]. When the star tracker moves with the speed along one coordinate, the spread area can be estimated. The 2°/s dynamic performance will be required to track the star. If the dynamic star spot wants to be accurately calculated, the pixels’ response of spread star spot area should be uniform. The length of the spread area will be affected by star trackers and optical parameters (show in Table 1). When exposure time , the length of the spot area is maximum, the centroid moves with 2°/s coordinate angular velocity, and can be given as
The spread spot area is shown in Fig. 2. Hence, in order to guarantee the 2°/s dynamic performance, the pixel uniform area is chosen as , and the standard response is calibrated by , , in order to cover the possible area of the star point for the centroid calculated.A simulation based on the parameters in Table 1 is conducted to analyze the influence of noise value on star spot centroid extraction. The impact of FPN pixel on spot target localization is shown in Fig. 3. The FPN pixel passes by the localization area that generates the error of gray value response.
The simulation results indicate that FPN has great influence on spot centroid localization related to the distance and value shown in Fig. 4. The maximum centroiding error is 0.37 pixel () with 80 pixel response noise value and 2 pixel distance to real point centroid. These simulation results can be the reference pixel classification to ensure pixel response correction.
On the image plane, the number of pixels is too many and the computation quantity of this algorithm is too large, which makes it not applicable to calculate all pixels, whereupon some specific pixels need to be selected to conduct calibration. The statistic number of all noise pixels () of the image sensor is shown in Fig. 5. 0.025177% (330) pixels are selected to be corrected, which affects the centoriding error by 0.05 pixel (18 LSB/10bit) according to the simulation result.
3. PIXEL RESPONSE NON-UNIFORMITY CORRECTION AND SPOT CENTROID LOCALIZATION ACCURACY ANALYSIS
The laboratory non-uniformity correction experiment is conducted to further show the applicability of the proposed approach in Section 3.A. The influence of the FPN experiment based on the Section 3.A non-uniformity correction experiment on spot centroid extraction is conducted in Section 3.B.
A. Pixel Response Non-Uniformity Correction Experiment
Using the pixel response model function conducted in Section 2, a CMOS image sensor non-uniformity correction calibration experiment is conducted in the laboratory. The experimental apparatus is shown in Fig. 6, and parameters are shown in Table 2. The light intensity of stars is generated by an adjustable uniform irradiation simulator that illuminates on the image sensor. Then the different light intensities and exposure times (1–100 ms) are adjusted to get a series of calibration correction sample points.
First of all, the overall analysis is carried out on the image plane. Then the necessary correction non-uniformity pixel response point is found, which has great influence on sampling star spot centroid extraction. Then around the noise point within pixels area, the scope of normal pixels’ response is calculated to make the standard response as abscissa x coordinate ( and the gain are calculated). The polynomial fitting method shown in Eq. (11) (according to the linear model ) is used to get the pixel non-uniformity of response measurement curves shown in Fig. 7(a) and correction curves shown in Fig. 7(b). In order to show the fitting process, the curves of some different exposure times are shown in Fig. 7(c).
The measurement fitting curve results show that non-uniformity responses change with exposure time and light intensity, and the noise pixels’ response is higher than normal response through the calibration model correction, which can be corrected to normal pixel response range. The correction process can be performed in all pixels that show such non-uniformity performance. The calibration curves under some specific exposure times are shown in Fig. 7(c).
With exposure time , the standard pixel response curve ( spot area) and non-uniformity pixel response curves are shown in Fig. 8. The curves show that each pixel’s response has a different linear function [different and cause the uniformity pixel response].
Because the proposed non-uniformity method experiment is conducted with fixed illumination intensity and changed exposure time from 1 ms to 100 ms, the calibration results can be analyzed under different exposure times () using 2D curves shown in Fig. 7(a). The different exposure time curves fitting result is shown in Fig. 7(c), and the corresponding residual error is shown in Fig. 9. The above pixel response curves’ correction non-uniform error is
is area pixels’ average output, is each area pixel’s response, and is pixel number () in this experiment.The oral pixels’ mean residual error for the all exposure times is 7.3391 (LSB/10 bit), while the proposed correction method reduces the error to 1.9015 (LSB/10bit), improving the performance by 74.1%. The promotion effect is remarkable.
In the conducted model before, the non-uniform noise nonlinear effect was ignored. It is necessary to calculate the influence extent. Let the polynomial fitting put forward the residual curve shown in Tables 3 and 4. From the calculation result, compared with the residual error 1.6002 LSB when is 1, the residual error is 1.5936 LSB when is 2 and it is 1.5541 LSB when is 3, which shows little improvement and increases much computational complexity. Hence, the linear model is selected for the non-uniform calibration.
B. Impact of the Star Spot Centroid Extraction Accuracy Analysis Experiment
A non-uniform image sensor pixel will affect the precision of star tracker centroid extraction. If the noise point moves into the star spot extraction area (typical star extraction is area ), the traditional method of average filtering denoising will not be able to eliminate the effect of noise on accuracy of centroid extraction. The correction method in this paper can effectively reduce the effects of non-uniform pixels on centroid localization accuracy. The experiment system is composed of three parts: the star tracker, three-axis rotary table, and star simulator, as shown in Fig. 10.
In the lab, the star simulator illuminates on the NST-1 star tracker, then the three-axis turntable turns to capture the images. The focal length of the star tracker is 25 mm and the size of the image sensor is 5.3 μm, so the corresponding rotating angle of each pixel can be calculated as about . In the experiment, the single rotation angle of the turntable is set as 0.001°, which means the star image moves across one pixel when the turntable rotates nearly 13 times. The area is used to calculate the star point center of gray mass, and the accuracy curve is shown in Fig. 110. The experimental accuracy curves have the S-curve [23] with cycle type because the image sensor’s sampling precision is 1 pixel; however, the star point’s movement is 0.1 pixel. In the process of discretization for continuous spot target energy, the pixel’s geometric center replaces the continuous energy center that will cause the system error of the gray centroid algorithm.
According to the accuracy measuring curves, if the noise point moves into the star spot area, it will cause serious deviation without flat-field calibration. The maximum error critical points are the entrance into the area and the departure from the area. Localization error, respectively, reaches 0.303 pixel () and 0.345 pixel (), which has the serious influence on the attitude measuring calculation of the star tracker. That condition will be unable to meet the high accuracy requirement of the star tracker centroid localization.
The traditional centroid algorithm cannot reduce the FPN of pixels, which makes the accuracy very low. Using the proposed non-uniform correction method for spot localization can effectively eliminate the effects of the noise pixel. Using this method can improve the star spot localization accuracy to 0.012 pixel (). Experimental results show that the proposed flat-field calibration method can be effective for the correction of the noise pixel and reducing the effects of noise on star localization accuracy.
4. CONCLUSION
In this paper, a flat-field calibration method for high-accuracy CMOS spot target image sensor pixel response non-uniformity is put forward. By analyzing the noise’s effect on star spot localization, a proper calibration correction model is set up. Then by using uniform illumination, the experiment of the image sampling is conducted that noise pixels can be effectively processed by using the flat-field calibration method. The average fitting residual after correction is 1.91 LSB (improved by 74.1%). Compared with the conventional average filter method and full plane calibration, this method is of the pixel gray response level, which is not affected by the surrounding pixels change and has the advantages of convenience and higher precision in the spot area. The experimental results of star spot localization accuracy indicate that this method can effectively eliminate the noise effect on centroid extraction accuracy, which can improve centroid extraction accuracy from 0.302 pixel () to 0.012 pixel (). The approach is an effective flat-field calibration method of high accuracy pixel response that satisfies the requirement of high accuracy spot localization accuracy.
Funding
National Natural Science Foundation of China (NSFC) (51522505, 51827806, 61605099); Key Research and Development Program of China (2016YFB0501201).
Acknowledgment
The authors acknowledge support from TY-Space Technology (Beijing) Ltd. for its cooperation in the experiment.
REFERENCES
1. A. Mora and A. Vosteen, “Gaia in-orbit realignment: overview and data analysis,” Proc. SPIE 8442, 84421Q (2012). [CrossRef]
2. P. Kask, K. Palo, C. Hinnah, and T. Pommerencke, “Flat field correction for high-throughput imaging of fluorescent samples,” J. Microsc. 263, 328–340 (2016). [CrossRef]
3. N. Padmanabhan, D. J. Schlegel, D. P. Finkbeiner, J. C. Barentine, M. R. Blanton, H. J. Brewington, J. E. Gunn, M. Harvanek, D. W. Hogg, Z. Ivezic, D. Johnston, S. M. Kent, S. J. Kleinman, G. R. Knapp, J. Krzesinski, D. Long, E. H. Neilsen, A. Nitta, C. Loomis, R. H. Lupton, S. Roweis, S. A. Snedden, M. A. Strauss, and D. L. Tucker, “An improved photometric calibration of the Sloan digital sky survey imaging data,” Astrophys. J. 674, 1217–1233 (2008). [CrossRef]
4. J. Li, A. Mahmoodi, and D. Joseph, “Using polynomials to simplify fixed pattern noise and photometric correction of logarithmic CMOS image sensors,” Sensors 15, 26331–26352 (2015). [CrossRef]
5. M. S. Wei, F. Xing, and Z. You, “A real-time detection and positioning method for small and weak targets using a 1D morphology-based approach in 2D images,” Light Sci. Appl. 7, 18006 (2018). [CrossRef]
6. S. Zhang, F. Xing, T. Sun, Z. You, and M. S. Wei, “Novel approach to improve the attitude update rate of a star tracker,” Opt. Express 26, 5164–5181 (2018). [CrossRef]
7. S. E. Park, J. G. Kim, M. A. A. Hegazy, M. H. Cho, and S. Y. Lee, “A flat-field correction method for photon-counting-detector-based micro-CT,” Proc. SPIE 9033, 90335N (2014). [CrossRef]
8. S. Uwe, “Intelligent error correction method applied on active pixel sensor based star tracker,” Proc. SPIE 5964, 154–161 (2005). [CrossRef]
9. M. Fiuzy, M. Hashemi, and S. K. M. Mashhadi, “A novel fixed pattern noise reduction algorithm in CMOS detector for LEO satellite application,” in 4th International Conference on Control, Instrumentation, and Automation (ICCIA) (2016), pp. 419–424.
10. E. S. Pereira, “Determining the fixed pattern noise of a CMOS sensor: improving the sensibility of autonomous star trackers,” J. Aerosp. Technol. Manage. 5, 217–222 (2013). [CrossRef]
11. M. Schoberl, C. Senel, S. Fossel, H. Bloss, and A. Kaup, “Non-linear dark current fixed pattern noise compensation for variable frame rate moving picture cameras,” in 17th European Signal Processing Conference (2009), pp. 268–272.
12. A. El Gamal, B. Fowler, H. Min, and X. Q. Liu, “Modeling and estimation of FPN components in CMOS image sensors,” Proc. SPIE 3301, 168–177 (1998). [CrossRef]
13. S. H. Lim and A. E. Gamal, “Gain fixed pattern noise correction via optical flow,” IEEE Trans. Circuit Syst. 51, 779–786 (2004). [CrossRef]
14. H. Tian, B. Fowler, and A. El Gamal, “Analysis of temporal noise in CMOS APS,” Proc. SPIE 3649, 177–185 (1999). [CrossRef]
15. N. V. Loukianova, H. O. Folkerts, J. P. V. Maas, D. W. E. Verbugt, A. J. Mierop, W. Hoekstra, E. Roks, and A. J. P. Theuwissen, “Leakage current modeling of test structures for characterization of dark current in CMOS image sensors,” IEEE Trans. Electron Devices 50, 77–83 (2003). [CrossRef]
16. J. Lukas, J. Fridrich, and M. Goljan, “Digital camera identification from sensor pattern noise,” IEEE Trans. Inf. Forensics Security 1, 205–214 (2006). [CrossRef]
17. B. Fowler, A. El Gamal, D. Yang, and H. Tian, “A method for estimating quantum efficiency for CMOS image sensors,” Proc. SPIE 3301, 178–185 (1998). [CrossRef]
18. M. Schoberl, S. Fossel, and A. Kaup, “Fixed pattern noise column drift compensation (CDC) for digital moving picture cameras,” in IEEE International Conference on Image Processing (2010), pp. 573–576.
19. M. Asadnezhad, A. Eslamimajd, and H. Hajghassem, “Optical system design of star sensor and stray light analysis,” J. Eur. Opt. Soc. 14, 9 (2018). [CrossRef]
20. S. Chang, “Aberrations of a slightly inclined Gaussian beam in a symmetric optical system,” Optik 123, 1190–1195 (2012). [CrossRef]
21. T. Sun, F. Xing, Z. You, and M. S. Wei, “Motion-blurred star acquisition method of the star tracker under high dynamic conditions,” Opt. Express 21, 20096–20110 (2013). [CrossRef]
22. J. Y. Yan, J. Jiang, and G. J. Zhang, “Dynamic imaging model and parameter optimization for a star tracker,” Opt. Express 24, 5961–5983 (2016). [CrossRef]
23. X. G. Wei, J. Xu, J. Li, J. Y. Yan, and G. J. Zhang, “S-curve centroiding error correction for star sensor,” Acta Astronaut. 99, 231–241 (2014). [CrossRef]