Open Access
Add to BibSonomyAbstract
Quantification of skin changes due to acanthosis nigricans (AN), a disorder common among insulin-resistant diabetic and obese individuals, was investigated using two optical techniques: diffuse reflectance spectroscopy (DRS) and colorimetry. Measurements were obtained from AN lesions on the neck and two control sites of eight AN patients. A principal component/discriminant function analysis successfully differentiated between AN lesion and normal skin with 87.7% sensitivity and 94.8% specificity in DRS measurements and 97.2% sensitivity and 96.4% specificity in colorimetry measurements.
©2011 Optical Society of America
1. Introduction
Obesity is one of the major health problems in the USA. According to one overweight and obesity prevalence estimation, over two thirds of American adults are overweight and over one third of adults are obese [1]. Among insulin-resistant diabetic patients who also suffer from obesity, acanthosis nigricans (AN) is a very common associated skin disorder causing skin darkening and roughening mostly occurring in the posterior and lateral folds of the neck, the axilla, inframammary, groin, and other areas. However, the exact relationship between insulin, obesity, and insulin resistance of diabetic patients is not yet fully understood [2]. Acanthosis nigricans can occur due to several other conditions such as a result of glandular disorder, Addison disease (deficiency of hormones from the adrenal gland), disorder from the pituitary gland, low level of thyroid hormones, and oral contraceptives. However, obesity remains the major cause [3]. Acanthosis nigricans is typically diagnosed visually by a dermatologist or other physician. An all-optical transdermal diagnostic based on light scattering that could assist in the diagnosis of acanthosis nigricans and quantitatively monitor changes in skin darkening and thickening would be extremely useful for diagnosis and assessing compliance to therapy.
Skin’s surface structure, chromophore composition, and compositional variation with depth can strongly influence the characteristic features of a spectrum of light scattered from a skin sample. When light is incident on a sample reflection, scattering, absorption, and transmission through the sample can occur. In simple media, the reflection of light obeys the law of reflection and the transmission of light obeys Snell’s law. The Fresnel equations describe reflection and transmission of light through multiple media having different indices [4].
Reflection can be defined as either specular or diffuse. Specular reflection is the reflection off of smooth or glossy surfaces in which an incoming ray is reflected into a single outgoing direction obeying the law of reflection. Conversely, when parallel rays of light are incident on a rough surface, the direction of the reflected light rays may differ due to different orientations of the surface normals for the various incident rays. As well, once the light enters into the skin multiple scattering can occur internally due to the different components present in the skin structure. Thus, the scattered light emerging from the skin may have different orientations. This type of non-specular reflection is called diffuse reflection.
Many theoretical models have been developed to describe the diffuse reflectance phenomenon. The model proposed by Farrell et al. in 1992 [5] is quite successful. This model allows a determination of tissue optical properties from diffuse reflectance spectra, using only the shape of the reflectance curve. This model was improved by Zonios et al. in 1999 [6] and was applied to biological tissues by introducing four main parameters: hemoglobin concentration, hemoglobin saturation, effective scatterer density, and effective scatterer size of the tissue. Hemoglobin and melanin are the main chromophore absorbers in the skin. Absorption (scattering) can be explained using the characteristic parameter called absorption (scattering) coefficient μa (μs) which is defined as the probability of photon absorption (scattering) per unit path length in a medium. This coefficient depends on the cross section and the number density of the absorbers (scatterers) [7]. Since hemoglobin can be either in the form of oxyhemoglobin or deoxyhemoglobin, their concentrations can be estimated using the known absorption coefficients of biological tissues at different wavelengths. Subsequently, quantitative assessment of human skin melanin, hemoglobin, and light scattering properties were determined using diffuse reflectance spectroscopy in the visible and near-infrared ranges based on the same analytical model [8]. Since all these experiments were done using fiber optic probes, attempts to design a ball lens coupled fiber optic probe for depth resolved spectroscopy of epithelial tissues were also reported [9,10]. Katika et al. in 2006 [11] investigated optical properties of human skin using steady state directional diffuse reflectance through numerical simulations using a seven-layer skin model. Diffuse reflectance spectra of warts, vitiligo, thrombus, and angioma were analyzed using principal component analytical methods and were able to differentiate and characterize these skin conditions [12].
A chromameter (used for colorimetry) is a type of spectrophotometer which can be used for complex color analysis with high precision and can accurately determine the spectral reflectance at each wavelength [13]. Several studies have reported the quantification of skin color and pigmentation using a colorimeter [14–16]. Different skin areas of healthy adults were tested using three colorimeters; a chromameter, a dermaspectrometer, and a mexameter were able to quantify small skin color changes [14] including erythema, darkening and skin blanching. The intensity of erythema [15], reaction of physical and allergic stimuli, effect of depigmentation of sunscreen, and bleaching agents have been reported [16].
No studies had directly compared colorimetry and diffuse reflectance spectroscopy (DRS) as applied to the same patient group in a clinical application prior to our 2010 work to investigate the usefulness of these two techniques for the diagnosis of AN. In our previous study, we concluded that the darkness of the skin as determined by colorimetry is a reliable indicator of AN relative to normal skin and is more efficacious than the concentration of melanin as determined by DRS [17]. No conclusion about the ability of these techniques to discriminate AN lesion from non-AN lesion darkened skin (i.e. due to tanning) was drawn. In this paper, we utilize a chemometric approach to analyze the data obtained from these AN patients by colorimetry and DRS to evaluate whether a classification of “lesion” skin or “healthy” skin can be made more reliably using a chemometric model which utilizes more of the data at one time compared to a straight-forward calculation of a skin chromophore concentration or the use of individual color parameters (e.g. melanin concentration or L* value related to skin darkness).
Another important aspect of the chemometric analysis of the DRS data is its use of only the raw diffuse reflectance (or absorbance) spectra instead of a reliance on calculated model-dependent melanin, oxyhemoglobin, and deoxyhemoglobin chromophore concentrations to differentiate normal healthy skin from AN lesion. The scattering of light from a medium as heterogeneous as skin is a complex phenomenon, and thus any model of this scattering is at best an approximation. Moreover, there were also limitations with using this DRS system for measurement of AN lesions since the original algorithm was designed to evaluate smooth normal skin. The use of the entire raw spectrum eliminates this need for an approximate model and the calculation of any chromophore concentration.
2. Materials and Methods
2.1. Spectroscopy
The DRS apparatus consisted of a HL-2000 Ocean Optics deuterium tungsten halogen lamp for skin illumination, a broadband spectrophotometer (USB 2000 light detector; BWTEK, Inc., USA) capable of detecting absorbance in the wavelength range 350-850 nm, a bifurcated fiber optic probe for light delivery and scattered light collection, and a computer. The output end of the bifurcated fiber bundle which was placed in contact with the skin was 2.5 mm. The absorbance spectra were calculated from the measured scattered light spectrum by a custom Labview v 8.0 (Labview Inc.) program (Johnson and Johnson, NJ, USA).
The colorimeter apparatus consisted of a CM-2600d spectrophotometer (Konica Minolta CM-2600d, Osaka, Japan) which utilized a Xenon arc lamp for skin illumination and a computer. Scattered light is collected by the CM-2600d through an integrating sphere whose internal surface is coated with a barium sulfate coating to make the light diffuse uniformly [13]. This instrument uses the standard tristimulus color analysis method utilizing the L*a*b* color system. The numerical parameters L*, a*, and b* represent, respectively, a color’s darkness to lightness, its green to red color component, and its blue to yellow color component. A circular patch of skin 8 mm in diameter was illuminated by the CM-2600d instrument. Both instruments were corrected for detector dark current and calibrated with a standard white disk prior to measurements on every patient. These instruments are shown in Fig. 1 .
Fig. 1 Left (top to bottom): DRS setup with laptop, spectrophotometer, halogen light source, calibration plate and bifurcated fiber optic cable; close-up of DRS probe showing 2 mm fiber optic core; DRS probe applied to patient’s posterior neck; Right (top to bottom): colorimeter; close-up of colorimeter 8 mm aperture; colorimeter measuring patient’s posterior neck.
2.2. Data collection
DRS and colorimetry measurements were obtained from four areas on the individual patient; two considered to be healthy tissue and used as a control and two areas of an acanthosis nigricans lesion on the neck (Fig. 1). The two control sites were the inner forearm approximately 10 cm from the wrist and the flat part of the upper shoulder several inches from the neck on the clavicle. Spectra from the AN lesion were collected from the median and lateral areas of neck. DRS and colorimetry data were collected from eight patients with acanthosis nigricans over an eight month period. Three colorimetry and ten DRS absorbance spectra measurements were taken from each site. The higher number of DRS measurements was necessary due to the smaller area of tissue sampled by the smaller DRS probe area. Since the probing pressure of the human skin can change the diffuse reflectance spectra [18], care was taken to insure uniform pressure on the skin over the course of the study. All measurements were taken at the same room temperature and humidity. For documentation and comparison, photographs of the lesion area were also taken every visit using a digital camera with a cross-polarized filter.
2.3. Chemometric techniques
In our previous study, differences between the lesion and healthy skin were determined by calculating average values of L*, a*, b*, melanin concentration, oxyhemoglobin concentration, and deoxyhemoglobin concentration, calculating the standard deviation of the measurements, and determining the average difference between healthy and lesion tissue [17]. Average values that differed by more than 3 standard deviations (3-sigma) were considered to be valuable predictors of the presence of lesion tissue. In this work, colorimetry data consisting of the L*, a*, and b* values and the raw absorption spectra of skin were analyzed separately using two chemometric methods known as principal component analysis (PCA) and discriminant function analysis (DFA).
PCA is a statistical method which correlates all the n variables of a data set and then determines which variables carry the most significant information in the data set. Thus, the original data set is transformed into a new data set having a reduced number of variables that carry the maximum variance in the data set [12]. In the PCA of the DRS data, the entire broadband DRS spectrum (226 channels spanning 450 nm with a 2 nm channel width) was used as an input data set. The PCA reduced the 226 variables to 4 variables which captured 99.7% of the variance in the data. These new variables were then used as the input to the DFA. The colorimetry data, consisting of only 3 parameters or variables (L*, a*, and b*) were not analyzed with a PCA. The raw spectra were not obtainable from the CM-2600d instrument.
The DFA method classifies the data into the independent groups present in the data by minimizing the variations within the groups and maximizing the variations between the groups [19]. The DFA constructs N-1 discriminant functions (DFs) for discrimination amongst N user-defined groups. Each spectrum can then be classified by N-1 DF scores. The DF scores were then used to allow a classification of an unknown sample as “lesion” or “healthy”. Typically each data set possessed only two DF scores, as we were attempting to differentiate between only three user-defined groups: control (forearm), control (neck), and AN lesion. Although median and lateral AN lesion measurements were obtained, a careful analysis showed no difference in these measurements and no utility in separating these measurements. Therefore all these lesion measurements were combined in the analysis.
To lessen the between-patient scatter of the measurements due to the inherent differences in skin coloring and composition of the patients, prior to analysis both colorimetry data and DRS data were normalized for every patient. This was done by first calculating the average of the patient’s forearm control measurements. The scatter of the forearm control measurements about this average allowed us to characterize the anticipated scatter of the lesion measurements which was expected to be even greater due to the heterogeneity of the lesion tissue. All subsequent measurements were then divided by that patient’s average forearm control to insure that data obtained from the neck control or neck lesion tissue were really differences from that patient’s normal skin coloring or composition.
3. Results and Discussion
The mean DRS absorbance spectra of the three measurement sites for one patient are shown in Fig. 2 . Absorbance is a unitless quantity obtained from the diffuse reflectance spectrum [20]. Each of the three spectra shown in Fig. 2 is the average of the all the measurements made at that particular site.
Fig. 2 Mean absorbance spectra of forearm control, neck control, and lesion. Lesion tissue demonstrates significantly greater absorption/weaker scattering.
Regions of large absorbance indicate strong absorption or weak scattering of the light at that wavelength, sometimes indicative of the presence of a specific chromophore. In Fig. 2 a clear increase in the absorbance of the lesion can be seen compared to the controls (forearm and neck). This is indicative of an increase in the concentration of the chromophores melanin, oxyhemoglobin, and deoxyhemoglobin, as was shown in our previous analysis of these patients. Aside from this overall increase in absorbance, statistically meaningful and reproducible spectral differences are hard to quantify in the three spectra shown in Fig. 2.
Chemometric techniques provided a more reliable way to obtain quantitative classification of the differences in the spectra. Figure 3 shows the first two discriminant function scores of the DFA performed on (a) the normalized DRS data (subject to PCA first, as explained above) and (b) the colorimetry data obtained from all eight patients spanning all patient visits. In the DFA results, DF1 expresses the maximum variance of the data and DF2 contains the rest of the variance in the data. It is evident that while there is some overlap of individual measurements, the center of mass of the lesion data distribution (shown as the dark square in Fig. 3) is well separated from the centers of mass of the distribution for the two controls. The reversal of the DF1 scores for the two modalities (i.e. positive DF1 lesion scores for DRS and negative DF1 scores for colorimetry) has no physical significance and is purely a computational artifact. The fact that all the patients’ data can be clustered significantly implies that the variations of the lesion tissue from the control sites are reproducible and similar in all patients.
Fig. 3 Discriminant function plots for (a) DRS and (b) colorimetry data obtained from all patients over the course of this study.
A leave-one-out (LOO) classification was also performed for both data sets. In a LOO method, each data point is treated as an unknown case and is classified against a data set consisting of all the other data points. The accuracy of that classification is then compared to the known identity of the data point to create a truth table. The truth tables for both spectral methods are shown in Table 1 . The interpretation of these truth tables is as follows: true positives (TP) indicate a lesion measurement was correctly classified as lesion. True negatives (TN) indicate a normal control measurement was correctly classified as normal. False positives (FP) indicate a normal control measurement was incorrectly classified as lesion. False negatives (FN) indicate a lesion measurement was incorrectly classified as normal.
Table 1. Truth tables for leave-one-out classification results of DRS and colorimetry.
Colorimetry data showed the highest true positive and true negative results, 98.4% and 96.1% respectively, which confirmed our earlier result that colorimetry seems to be more efficacious for diagnosing AN than DRS as we currently perform it [17]. Impressively, when analyzed with the PCA/DFA the DRS data showed more than 91% sensitivity (TP/TP + FN) and specificity (TN/TN + FP), which was not the case for the standard statistical analyses as we have reported earlier [17].
In the previous leave-one-out analysis, an unclassified measurement was classified with a set of discriminant functions constructed using all the other data points, including other data points from that same patient. However, if either of these techniques is to be used for patient screening, it is not realistic to expect the discriminant functions to have been constructed with any prior data from that patient. Therefore we performed a DFA excluding one patient at a time from the analysis. Unclassified lesion and neck control data from that patient were then input to the DFA (which contained none of that patient’s other data) and were classified according to the library of results from other patients. The truth tables for this analysis are shown in Table 2 . As expected the rates of true positives and true negatives declined, although only by a small amount (by 3.7% and 0.3% for DRS and 1.2% and 1.5% for colorimetry) and rates of false positives and false negatives increased (by 0.4% and 3.7% for DRS and 1.5% and 1.2% for colorimetry). These results show a more realistic truth table for the techniques if they were to be used to screen previously unexamined patients for AN.
Table 2. Truth tables for patient exclusion classification results of DRS and colorimetry.
One of the benefits of using a chemometric approach is shown in Fig. 4 which shows the DF plot of the patient exclusion analysis for patient 8 in our previous study. Patient 8 was the only case in our previous study that did not show a 3-sigma standard deviation between the calculated melanin concentration in the forearm control and the AN lesion indicating that this patient’s lesion tissue was very hard to diagnose spectrally [17]. Nonetheless, in Fig. 4, when patient 8’s lesion data was entered as unclassified data into the DFA (shown as golden x symbols in Fig. 4), almost all of the measurements were easily classified as “lesion” compared to the forearm control or the neck control.
Fig. 4 DF plots showing (a) DRS and (b) colorimetry data with patient 8’s lesion data input as unclassified data into the analysis. This patient’s lesion data, which clustered well with itself, was significantly different from the mean of the other patients’ lesion data. Nonetheless, it was easily and reliably classified as “lesion” in both analyses.
The higher diagnostic accuracy of the colorimetry technique as shown in Tables 1 and 2 seems counter-intuitive since the DRS spectral data contains more diagnostic information than the colorimetry data which are calculated from measurements of narrow spectral ranges. The results obtained are due to the inherent differences in the reliability/repeatability of the measurements made by these two instruments, specifically differences resulting from probe design. The DRS probe not only collected light from a much smaller skin area than the colorimeter (a diameter of 2.5 mm compared to 8 mm), but also from a much smaller solid angle compared to the integrating sphere of the colorimeter. This resulted in increased measurement scatter. To prove this, the repeatability of both instruments was tested by making repeated measurements twice daily for 5 days on a standard target consisting of a section of a skin prosthesis. The results of these measurements indicated that the percent deviation of the DRS measurements was inherently higher than the colorimetry measurements due to the smaller light collection area and solid angle of the DRS probe which made measurements made with it more sensitive to inhomogeneities of the skin prosthesis.
Although forearm and neck controls were both used as normal controls in this AN study, the more realistic site for a normal control is the neck normal (which should have similar properties compared to the lesion measurement site due to tanning, aging, etc). This can be observed by the neck control data being closer to the lesion data than the forearm control data in Figs. 3 and 4. Thus, it was vital to know the sensitivity and specificity of the lesion site measurements as compared to the neck control. To investigate this, receiver operating characteristic (ROC) curves were constructed using the DF1 values of the DRS and colorimetry data to act as a “cut point” to discriminate lesion and neck control data (discarding the forearm control data).
In a ROC curve [21], the sensitivity of the technique (as defined above) is plotted against 1-specificity (as defined above). ROC curves are shown in Fig. 5 for DRS (a) and colorimetry (b). The ROC area under the curve (AUC) establishes the usefulness of the test, with an AUC of 1.0 denoting a perfect test, and an AUC of 0.5 denoting a worthless test. The ROC curves of Fig. 5 possess an AUC of more than 0.98, indicating a highly reliable test for this AN investigation.
Fig. 5 ROC curves of a test to differentiate neck control measurements from lesion measurements (as shown in Fig. 3) on the basis of the DF1 score alone for (a) DRS and (b) colorimetry data. The area under the curves for (a) was 0.985 and for (b) was 0.995.
4. Conclusions
Spectroscopic and colorimetric measurements combined with chemometric analysis methods provided sensitive and specific diagnoses of acanthosis nigricans lesions compared to nearby control skin. Analysis of raw DRS absorbance spectra showed clear clustering of normal controls and lesion groups (Fig. 3 (a)) for all patients denoting a commonality that would allow diagnoses of previously unmeasured lesions. Colorimetry data also showed the ability to reliably identify AN lesions (Fig. 3 (b)). Excluding patients one at a time from a DFA model and then testing that patient’s spectral data with the model constructed only from other patient measurements provided a realistic simulation of an acanthosis nigricans screening test. DRS data provided more than 87% sensitivity and 94% specificity and colorimetry data showed more than 95% sensitivity and specificity (Table 2) in this type of test. ROC curves also confirmed that the use of a discriminant function analysis on DRS and colorimetry data can provide a sensitive and specific AN test even when only the DF1 score is used to assess skin condition. Unfortunately, none of the patients showed any improvement in acanthosis nigricans from treatment during the duration of this study which was also confirmed by visual and photographic observations. Thus, the changes in skin DRS and colorimetry data that occur during the healing process of AN could not be established. However, the existing data showed both DRS and colorimetry can be used as a successful diagnostic tool for acanthosis nigricans when combined with chemometric methods such as PCA and DFA.
Acknowledgments
The authors acknowledge Dr. Nikiforos Kollias and Eduardo Ruvolo, Johnson & Johnson Consumer and Personal Products, Skillman, New Jersey, USA for their technical and scientific support with the colorimetry and DRS.
References and links
1. K. M. Flegal, M. D. Carroll, C. L. Ogden, and L. R. Curtin, “Prevalence and trends in obesity among US adults, 1999-2008,” JAMA 303(3), 235–241 (2010). [CrossRef] [PubMed]
2. A. S. Katz, D. C. Goff, and S. R. Feldman, “Acanthosis nigricans in obese patients: presentations and implications for prevention of atherosclerotic vascular disease,” Dermatol. Online J. 6(1), 1 (2000). [PubMed]
3. G. Yosipovitch, A. DeVore, and A. Dawn, “Obesity and the skin: skin physiology and skin manifestations of obesity,” J. Am. Acad. Dermatol. 56(6), 901–916, quiz 917–920 (2007). [CrossRef] [PubMed]
4. A. Ghatak, Optics, 2nd ed. (Tata McGraw-Hill, New Delhi, India, 2002).
5. T. J. Farrell, M. S. Patterson, and B. Wilson, “A diffusion theory model of spatially resolved, steady-state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Med. Phys. 19(4), 879–888 (1992). [CrossRef] [PubMed]
6. G. Zonios, L. T. Perelman, V. Backman, R. Manoharan, M. Fitzmaurice, J. Van Dam, and M. S. Feld, “Diffuse reflectance spectroscopy of human adenomatous colon polyps in vivo,” Appl. Opt. 38(31), 6628–6637 (1999). [CrossRef] [PubMed]
7. L. V. Wang and H.-I. Wu, Biomedical Optics: Principles and Imaging (Wiley-Interscience, 2007).
8. G. Zonios, J. Bykowski, and N. Kollias, “Skin melanin, hemoglobin, and light scattering properties can be quantitatively assessed in vivo using diffuse reflectance spectroscopy,” J. Invest. Dermatol. 117(6), 1452–1457 (2001). [CrossRef] [PubMed]
9. R. A. Schwarz, D. Arifler, S. K. Chang, I. Pavlova, I. A. Hussain, V. Mack, B. Knight, R. Richards-Kortum, and A. M. Gillenwater, “Ball lens coupled fiber-optic probe for depth-resolved spectroscopy of epithelial tissue,” Opt. Lett. 30(10), 1159–1161 (2005). [CrossRef] [PubMed]
10. R. A. Schwarz, W. Gao, D. Daye, M. D. Williams, R. Richards-Kortum, and A. M. Gillenwater, “Autofluorescence and diffuse reflectance spectroscopy of oral epithelial tissue using a depth-sensitive fiber-optic probe,” Appl. Opt. 47(6), 825–834 (2008). [CrossRef] [PubMed]
11. K. M. Katika and L. Pilon, “Steady-state directional diffuse reflectance and fluorescence of human skin,” Appl. Opt. 45(17), 4174–4183 (2006). [CrossRef] [PubMed]
12. S. Prince and S. Malarvizhi, “Analysis of diffuse reflectance spectra of various skin conditions by principal component method,” in International Conference on Biomedical and Pharmaceutical Engineering (ICBPE, India. 2009), pp.1–4.
13. Konica Minolta, “Precise color communication: color control from perception to instrumentation,” 1998, http://www.konicaminolta.com/sensingusa/products/Color-Measurement/spectrophotometer2/cm2600d-2500d/index.html.
14. P. Clarys, K. Alewaeters, R. Lambrecht, and A. O. Barel, “Skin color measurements: comparison between three instruments: the Chromameter(R), the DermaSpectrometer(R) and the Mexameter(R),” Skin Res. Technol. 6(4), 230–238 (2000). [CrossRef] [PubMed]
15. G. N. Stamatas, B. Z. Zmudzka, N. Kollias, and J. Z. Beer, “In vivo measurement of skin erythema and pigmentation: new means of implementation of diffuse reflectance spectroscopy with a commercial instrument,” Br. J. Dermatol. 159(3), 683–690 (2008). [CrossRef] [PubMed]
16. L. Andreassi and L. Flori, “Practical applications of cutaneous colorimetry,” Clin. Dermatol. 13(4), 369–373 (1995). [CrossRef] [PubMed]
17. B. Pattamadilok, S. Devpura, Z. U. Syed, P. Vemulapalli, M. Henderson, S. J. Rehse, B. H. Mahmoud, H. W. Lim, R. Naik, I. Hamzavi, are preparing a manuscript to be called “Quantitative skin color measurements: comparison between colorimetry and diffuse reflectance spectroscopy in acanthosis nigricans patients.”
18. J. A. Delgado Atencio, E. E. Orozco Guillén, S. Vázquez y Montiel, M. Cunill Rodríguez, J. Castro Ramos, J. L. Gutiérrez, and F. Martínez, “Influence of probe pressure on human skin diffuse reflectance spectroscopy measurements,” Opt. Memory Neural Networks 18(1), 6–14 (2009). [CrossRef]
19. W. R. Klecka, Discriminant Analysis, Series: Quantitative Applications in the Social Sciences (Sage Publication, Calif., USA, 1980).
20. G. N. Stamatas, B. Z. Zmudzka, N. Kollias, and J. Z. Beer, “Non-invasive measurements of skin pigmentation in situ,” Pigment Cell Res. 17(6), 618–626 (2004). [CrossRef] [PubMed]
21. S. J. Mason and N. E. Graham, “Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation,” Q. J. R. Meteorol. Soc. 128(584), 2145–2166 (2002). [CrossRef]
