1. Introduction

Globally, 570,000 women are diagnosed, and 311,000 women die from cervical cancer each year [1]. More than 85% of these deaths occur in low- and middle-income countries (LMICs), mainly due to lack of organized screening, diagnosis, and treatment programs [2]. In high-resource settings, the standard of care includes: 1) screening with cytology and/or testing for high-risk human papillomavirus (HPV); 2) diagnosis of screen-positive patients with colposcopy and biopsy; and 3) treatment of histopathologically identified high-grade precancer lesions or cancer (cervical intraepithelial neoplasia grade 2 or more severe [CIN2+]) with loop electrosurgical excision procedure (LEEP) or other excisional procedures [3]. This requires three visits, with a need for both pathology services and skilled clinicians at each step. This approach is not feasible in low-resource settings as the multiple visits result in many women being lost to follow-up and incompletely treated [4].

In 2021, the World Health Organization (WHO) released updated guidelines for the screening and treatment of cervical precancerous lesions in LMICs, and recommended either a Screen & Treat or a Screen, Triage & Treat approach [5]. In a Screen & Treat approach, women with a positive screening test (cytology, HPV testing or visual inspection with acetic acid (VIA)) directly undergo treatment with ablation (cryotherapy or thermal ablation) or excision with a LEEP without the need for a cervical biopsy and histologic diagnosis. This allows for a single-visit approach, decreasing loss to follow-up and obviating the need for gynecology and pathology services which are often lacking in LMICs. A limitation of a Screen & Treat approach is that it results in overtreatment, as the majority (50-70%) of women infected with HPV do not, and will not, have high-grade cervical disease [6,7]. A Screen, Triage & Treat approach is therefore preferred and is the only recommended option for women living with HIV (WLWH). In a Screen, Triage & Treat approach, treatment is based on a positive primary screening test followed by a positive second “triage” test, with or without a histologically confirmed diagnosis. The WHO suggests HPV genotyping, colposcopy, VIA or cytology as possible triage tests, with the choice depending on feasibility, training, program quality assurance and local resources [5]. Unfortunately, these triage tests are not widely available in many LMICs due to cost (HPV genotyping), lack of trained personnel and infrastructure (cytology, colposcopy) or limited reproducibility and accuracy (VIA). In addition, they have limited specificity for detecting high grade cancer [8]. Thus, there is a crucial need for point-of-care methods to triage screen-positive women to enable effective screening, triage, and treatment in a single visit.

A number of low-cost optical imaging techniques have been developed to improve point-of-care detection of cervical precancer and cancer. Most of these techniques can be divided into macroscopic “widefield” imaging modalities (i.e., variations of the colposcope) and microscopic (high-resolution) imaging modalities. For example, low-cost, mobile digital colposcopes have been developed to enable clinicians to perform colposcopy in low-resource settings. Examples include cell-phone based digital colposcopes such as Gynius AB’s Gynocular [9] and MobileODT’s EVA [10], as well as custom built imaging devices such as Calla Health’s Pocket Colposcope [11,12]. Mobile digital colposcopes can increase the accessibility of colposcopy in low-resource settings [13,14]. These devices generally combine high-quality image capture with secure online data management services, allowing providers to visualize the cervix, document exams, and add annotations directly to cervical images for further review. Yet, these digital colposcopes still rely on clinician interpretation for detection of precancer, which has been shown to be often inaccurate and unreliable due to the limited availability of properly trained expert colposcopists in LMICs [15,16]. To reduce the need for an expert provider to interpret images, automated image analysis algorithms have been developed for identification of cervical precancers using various machine learning techniques [17–20]. Initial studies have shown that the performance of automated algorithms is comparable to or possibly more accurate than human interpretation of the same cervical images, but more evidence is needed before their widespread use can be established.

High-resolution imaging provides a non-invasive method to visualize biological tissues in real time, displaying features that would otherwise only be seen following biopsy and histopathologic analysis. Optical coherence tomography (OCT), for example, provides high-resolution images that allow clinicians to characterize epithelial architecture of the cervix [21–23]. Other technologies such as scanning fluorescence microscopy [24], confocal endomicroscopy [25–27], and high resolution microendoscopy [28] can directly image nuclear morphology at the surface of the cervical epithelium. One such device, the high resolution microendoscope (HRME), is a fiber optic fluorescence microscope capable of imaging the cervical epithelium with subcellular resolution after topical application of fluorescent dye [29]. Results from large clinical studies have shown that images acquired by the HRME can be interpreted objectively in real-time using standard morphologic and/or neural network based algorithms with comparable sensitivity and specificity as colposcopy for detection of high grade cervical precancer [4,6,30]. While sensitivity exceeds 94%, specificity is lower and recent work shows that specificity is limited by misclassification of non-neoplastic sites that contain columnar epithelium [30]. Recent advances in this technology have significantly increased the speed at which high resolution images can be obtained [31].

The combination of widefield imaging and high-speed, high-resolution imaging can provide important spatial context that may improve interpretation of a large volume of high-resolution images. For example, widefield imaging can be used to track the spatial location of the high resolution imaging probe in relation to anatomic landmarks such as the squamo-columnar junction (SCJ). In addition, high-speed, high-resolution imaging can facilitate the collection of data from larger areas of tissue, thereby increasing the field of view of the high-resolution system. There may also be opportunities to combine information from both imaging modalities to develop diagnostic algorithms. However, this requires the integration of two imaging systems and it is often challenging to provide precise co-registration of multi-scale image data throughout the entire imaging session. Furthermore, there are challenges to the implementation of real-time processing of the acquired images, which is necessary to leverage the information provided by both imaging modalities at the point of care.

In this manuscript, we describe for the first time the Multimodal Mobile Colposcope (MMC), a multimodal optical imaging system and software architecture that integrates two low-cost imaging devices: 1) a commercially available Pocket Colposcope (Calla Health, Durham, North Carolina); and 2) a high-speed, high-resolution microendoscope (fastHRME) which is being evaluated in a large clinical study in Brazil (ClinicalTrials.gov NCT05078528). In this report, a new method is presented to correlate high-resolution images to the anatomical location from which they were acquired using a combination of fiber tracking and widefield image co-registration. In addition, we describe the first real-time deployment of a deep learning algorithm designed to diagnose CIN2 + from HRME images to provide automated diagnostic results delineating areas of the cervix with greater likelihood of high grade cervical precancer. Data from a representative group of patients illustrate that these advances can be realized in a clinical setting and that results agree well with the gold standard of histopathology. These clinical results support the potential of multiscale imaging to provide a sensitive and specific strategy to support “Screen-Triage-Treat” programs that can be performed in a single visit in low-resource settings.

2. Methods

2.1 Instrumentation

The MMC is shown in Fig. 1; widefield images are obtained using a Pocket Colposcope and high resolution images are acquired using a fastHRME. The system is controlled through an integrated user interface on a laptop computer with a touchscreen display (Surface Book 3; Microsoft, Redmond, Washington, USA). When slightly inserted through a speculum, the Pocket Colposcope provides magnified images of the cervix. It has three illumination settings: low brightness white light, high brightness white light, and green light, which are toggled by pushing a button on the device. The Pocket Colposcope has a 3.5-5.0 cm field of view, a working distance of 3.5-4.5 cm, and a lateral resolution of approximately 20 µm. The fastHRME is a fiber-optic fluorescence microscope designed to be used with the fluorescent dye proflavine, a topical contrast agent that stains epithelial cell nuclei [31]. Excitation illumination at 460 nm is coupled to an optical fiber bundle secured in a plastic holder; returning fluorescence is directed through a longpass filter to a high frame rate camera. The field of view of the fastHRME is 790 µm and the lateral resolution is 4 µm. During imaging, the high-resolution imaging probe is placed in contact with cervical mucosa and microscope images are acquired at 90 frames per second (FPS). Images from the Pocket Colposcope and fastHRME are displayed in real time on the laptop; the position of the high-resolution imaging probe is tracked across the cervix and high-resolution images are classified using a previously described Multi-Task Network (MTN) [30]. Software to acquire video feeds from both the Pocket Colposcope and the FastHRME was developed in NI LabVIEW 2021 (National Instruments, Austin, TX, USA).

 

Fig. 1. a) The Multimodal Mobile Colposcope (MMC) consists of a widefield imaging system (the commercially available Pocket Colposcope) and a fiber optic high-resolution imaging system (the fastHRME). The system is controlled through a single user interface on a laptop computer. b) MMC in the colposcopy clinic at Barretos Cancer Hospital.

Download Full Size | PDF

2.2 Patient imaging

The MMC was used to acquire images from women participating in a large multi-center study in Brazil designed to develop and evaluate multi-modal image analysis algorithms for automated detection of high-grade cervical precancer and cancer. The larger study was reviewed and approved by the Barretos Cancer Hospital (BCH) Ethics Research Committee, the Brazilian National Ethics Research Commission/CONEP (CAAE:38969820.9.1001.5437) and the Institutional Review Boards of Rice University (ID#2020-342) and The University of Texas MD Anderson Cancer Center (ID#2021-0356). Written informed consent was obtained from all participants. The protocol was registered at ClinicalTrials.gov (NCT05078528). Participants were recruited from the colposcopy clinic at BCH in Barretos, Brazil and provided informed consent.

2.3 Imaging workflow

Images were acquired using the MMC by an expert colposcopist and medical assistant in the Cancer Prevention Department at BCH. The user interface guides the clinician through a colposcopy exam and the multi-modal imaging process; the imaging workflow is outlined in Fig. 2. When imaging is initiated with the MMC, live video feed from the Pocket Colposcope is displayed on the user interface. The colposcopist applies 5% acetic acid to the cervix and focuses the Pocket Colposcope; the assistant saves a focused widefield image. The saved frame is displayed on the interface for colposcopist review; image acquisition can be repeated if desired to ensure a high quality image is saved. Lugol’s Iodine is then applied to enhance the image contrast of HRME imaging and the widefield imaging sequence is repeated.

 

Fig. 2. During the Multimodal Mobile Colposcope imaging workflow, reference colposcopy images are acquired using the Pocket Colposcope after application of acetic acid and Lugol’s Iodine. Next, proflavine is applied to the cervix and the imaging probe of the high-speed, high-resolution microendoscope (fastHRME) is used to image the superficial epithelium of the cervix while the Pocket Colposcope tracks its position on the cervix. After multimodal imaging, a biopsy site is selected by the clinician and a biopsy is taken.

Download Full Size | PDF

Second, high resolution images are acquired. The colposcopist applies 0.01% proflavine to the tissue and places the imaging probe on the tissue, targeting colposcopic lesions. The live video feeds of the Pocket Colposcope and the fastHRME are simultaneously displayed and saved at fifteen and ninety frames per second, respectively. The clinician translates the imaging probe across any lesions identified during colposcopy, ensuring that the cervix and the imaging probe tip are visible on the live Pocket Colposcope feed. If no abnormal areas are detected during colposcopy, the imaging probe is used to image a colposcopically normal area. After multimodal image acquisition, an image review display allows the colposcopist to review a video of the entire imaging sequence or manually scroll through the image sequence. Following image acquisition, multi-modal images are co-registered and processed to simultaneously display the widefield image delineating the path of the high-resolution imaging probe and the corresponding diagnostic score at each point.

Sites identified as abnormal by colposcopy are biopsied. If no clinically abnormal sites are identified, a single biopsy is obtained from a clinically normal site where HRME images were taken. Following biopsy, an image of the cervix is acquired with the Pocket Colposcope to document biopsy locations.

Finally, the clinician selects a representative white light colposcopy image and uses a stylus to manually annotate anatomical features including the ectocervical margin, the squamous columnar junction (SCJ), and os. The colposcopist outlines any cervical lesions and designates clinical impression as low grade squamous intraepithelial lesion (LSIL), high grade squamous intraepithelial lesion (HSIL), or suspected cancer. The colposcopist also annotates the locations of any biopsy sites.

2.4 Post-processing and display of MMC images

Post-processing was performed after patient imaging was completed, including: co-registration of Pocket Colposcope frames to account for small shifts in the field of view during imaging; tracking the location of the high-resolution imaging probe on the cervix to determine the anatomic location of each high-resolution image; analyzing each high-resolution image to generate a diagnostic score; and fusing the diagnostic scores of the high-resolution images with their tracked locations. The result of these post-processing steps is an aggregated multimodal imaging data set that includes a series of registered widefield colposcope frames, the pixel locations of the imaging probe in that frame, and the average diagnostic score of the fastHRME images taken at that location. This processed dataset was then used to generate overlays showing the predicted likelihood of high-grade precancer or cancer along the imaging path across the cervix.

An overview of the steps of the co-registration process for multimodal imaging is shown in Fig. S1. Because the Pocket Colposcope is hand-held, the field of view can shift during the imaging procedure. As a result, it is necessary to co-register all widefield images in order to create a common reference coordinate system and correct for movements in the field of view from the clinician moving the Pocket Colposcope or the patient moving during imaging. To do this, the python module “vidstab” was used [32]. Briefly, this algorithm calculates the image translation from frame to frame using optical flow methods and generates a transformation between previous and current frames. The algorithm aggregates the calculated transforms to generate a “trajectory” for each frame. The list of trajectories is smoothed using a sliding window average of frames, and a final transformation is made by applying the smoothed trajectory to all frames. The output of this process is a stabilized video where each frame has been transformed to have a common coordinate system (Fig. S1(a)).

Following co-registration of widefield images through video stabilization, the location of the high-resolution imaging probe on the cervix is determined to co-register high-resolution and widefield images. To facilitate tracking of the probe tip during imaging, a colored dental ligature was attached to the distal tip of the imaging probe. Hue, Saturation, and Value (HSV) based color thresholding was used to segment the colored ligature in each stabilized widefield image. Further morphological filtering was then applied to remove random noise and the location of the probe tip was assigned as the centroid of the largest remaining group detected. These calculated fiber locations are aggregated together and overlaid onto the representative Lugol’s colposcopy image (Fig. S1(b)). Finally, these fiber locations are manually translated from the Lugol’s image to the white light image (Fig. S1(c)). A representative video of this stabilization and fiber tracking process is available in Visualization 1.

Next, the high-resolution images acquired by the fastHRME were scored using a deep learning algorithm to provide automated diagnostic results delineating areas of the cervix with greater likelihood of high grade cervical precancer. Briefly, the Multi-Task Neural Network (MTN) performs nuclear segmentation and scores the probability that the image was obtained from tissue containing high-grade cervical precancer or cancer (i.e. CIN2+). The MTN takes in a single HRME image and outputs a single diagnostic score for that individual image; these scores are correlated to the anatomical area the HRME image was acquired from. The diagnostic scores generated by the MTN are only saved if the number of segmented nuclei exceed a preset threshold so as to avoid scoring frames that contain too few nuclei. The MTN was trained, validated, and tested using data acquired from HRME imaging of cervical tissue in patients from two large studies conducted in rural Brazil [4,6]. In an independent test set, Brenes et al. demonstrated that the MTN achieved sensitivity of 0.93 and specificity of 0.57, which was comparable to that of expert colposcopy [30]. A probability cutoff of 0.17 for the MTN was found to minimize differences in sensitivity between the algorithm and expert colposcopy, and was prospectively applied for this study. To account for differences in magnification, fastHRME images were resampled to the same pixel dimensions as the dataset used to develop the MTN using Lanczos interpolation.

Widefield images were then annotated to delineate the path of the high-resolution imaging probe and the corresponding MTN diagnostic score along the path. Pixel locations of the probe corresponding to each fastHRME frame were overlaid onto the representative white light image and an imaging path encompassing the imaged region was manually selected. Due to the high frame rate of the fastHRME, multiple overlapping images were acquired as the probe was translated across a single anatomic site, resulting in multiple diagnostic scores associated with a single pixel location on the widefield image. Therefore, an average MTN score was generated for each pixel location on the manually designated imaging path by taking the mean of all diagnostic scores associated with that pixel location.

2.5 Real time image analysis

We also developed a method to analyze and score fastHRME frames in real-time (Fig. S2) during the imaging procedure. This real-time workflow uses a two-step process: first, image quality is assessed to determine if the most recently captured image is of sufficient quality; images passing quality control are then analyzed with the MTN to generate a diagnostic score. The quality control step limits the number of images that are sent to the deep learning network for analysis, ensuring that as many frames as possible are scored in real-time.

Using existing high-resolution image libraries from previous and ongoing studies [4,6], we developed an automated quality control metric to identify poor-quality images when the probe is not fully in contact with the tissue (Fig. S3). First, a mask is applied to the raw image acquired from the HRME (Fig. S3(a)) to remove the outer cladding of the fiber bundle and retain only image information (Fig. S3(b)). Next, the masked image is converted to the frequency domain using a Fast Fourier Transform (FFT) (Fig. S3(c)); low frequency components are removed from the image to reduce background noise and out-of-focus features (Fig. S3(d)), and the image is converted back into the spatial domain using an inverse FFT (Fig. S3(e)). The filtered image is then thresholded to generate a binary image where the number of particles approximates the number of nuclei in the field of view (Fig. S3(f)). Processed images containing more than a predetermined number of nuclei pass the quality assurance metric.

The MTN was deployed in real-time using a python executable that runs in the background of the main user interface. Immediately after image acquisition, each frame undergoes the FFT based automated quality check to determine if the image quality is sufficient for diagnostic scoring (i.e. nuclei are visible, in focus, and free of motion blur). Images that pass the quality assurance metric are sent to the MTN for diagnostic scoring. Frames that do not have at least the minimum number of detected particles are also saved to file but are not sent for real-time diagnostic scoring. The python executable detects when new images have been designated for real-time analysis and begins processing these images as they are saved. To ensure adequate sampling throughout the entire imaging procedure, the code prioritizes scoring of images most recently acquired from the fastHRME. As described in the preceding section, the MTN performs a second QC step. Therefore, fastHRME frames that are designated for real-time analysis but fail this second QC step have their scores removed from further analysis. If the fastHRME frames passes the second QC step, the corresponding diagnostic score generated by the MTN is saved to a data table for post-acquisition analysis.

3. Results

In this manuscript, we present MMC images from six representative patients with a range of low- and high-grade lesions. This data includes over 22,800 fastHRME images, over 9,900 Pocket Colposcope images, and 8 lesions annotated by expert colposcopists. On average, multimodal imaging sessions were 75 seconds in duration and did not significantly prolong the time of a standard colposcopy procedure. An average of 79% of the acquired fastHRME frames from each patient passed the post-process or real-time quality check and received a diagnostic score.

Figure 3 shows representative MMC images acquired from a subject with two acetowhite cervical lesions, including a lesion from eleven to twelve o’clock with a colposcopic impression of LSIL and a lesion from two to three o’clock also with a colposcopic impression of LSIL (Fig. 3(a)). After the application of Lugol’s iodine (per standard of care), multimodal images were collected; the black line shows the tracked path of the high-resolution probe as it was translated across both lesions. During the 88 seconds of multimodal imaging, 4,307 high resolution images and 1,203 Pocket Colposcope images were collected (see Visualization 2). Immediately after imaging, both lesions were biopsied at the locations shown in Fig. 3(a). Histopathologic examination of both biopsies showed chronic cervicitis and no evidence of cervical dysplasia. The fastHRME images were prospectively scored by the MTN. Of the 4,307 HRME images collected, 3,568 (82.8%) passed the nuclear segmentation-based quality control metric of the MTN.

 

Fig. 3. MMC results from a patient with two low-grade acetowhite lesions. a) Representative colposcopy image annotated with the high-resolution imaging path and biopsy locations. b) Multimodal image pair acquired at location of Biopsy 1. c) Multimodal image pair acquired at location of Biopsy 2. d) Average MTN diagnostic score from high-resolution images vs. distance along the imaging path; scores above the MTN threshold at 0.17 are classified as CIN2 + . Scale bars on high resolution images represent 200 µm. Microscope images in panel b and c were contrast enhanced to improve visibility.

Download Full Size | PDF

Representative multimodal image pairs are shown in Fig. 3(b-c); each pair includes a widefield Pocket Colposcope image showing the high-resolution imaging probe in contact with the cervix and a corresponding high-resolution image obtained at that location. Figure 3(b) shows an image pair where the imaging probe is located within the lesion at eleven-twelve o’clock; the high-resolution image shows uncrowded, regularly shaped and spaced nuclei characteristic of normal squamous epithelium. The diagnostic score provided by the MTN was below the pre-established threshold for classification as CIN2 + . Figure 3(c) shows an image pair where the imaging probe is located within the two-three o’clock lesion; the nuclear features visible in the high-resolution image are again consistent with normal squamous epithelium and the corresponding diagnostic score was below the pre-established threshold for classification as CIN2 + . Figure 3(d) shows a plot of the MTN diagnostic scores along the high-resolution imaging path; regions corresponding to each biopsy are highlighted. The MTN diagnostic scores for all areas imaged within the lesions were below the threshold, correctly predicting that neither lesion contained CIN2 + .

Figure 4 shows MMC images acquired from a patient with two acetowhite cervical lesions, including a lesion from eleven to one o’clock with a colposcopic impression of LSIL and a lesion from two to three o’clock with a colposcopic impression of HSIL (Fig. 4(a)). The black lines show the tracked locations of the high-resolution probe as it was translated across each lesion; 1,545 HRME images and 2,324 Pocket Colposcope images were collected during the one minute and forty-two second imaging session (see Visualization 3). After multimodal imaging, a biopsy was taken from each lesion as shown in Fig. 4(a). Histopathologic diagnosis of the biopsy at twelve o’clock with a clinical impression of a low-grade lesion was mild chronic cervicitis with mild, focal nuclear atypia; histopathologic diagnosis of the biopsy at three o’clock with the clinical impression of a high-grade lesion was CIN3. Of the 1,545 fastHRME images collect, 949 (61%) of the images passed the nuclear segmentation-based quality control metric of the MTN.

 

Fig. 4. MMC results from a patient with low-grade and high-grade acetowhite lesions. a) Representative colposcopy image annotated with two high-resolution imaging paths and biopsy locations. b) Multimodal image pair at location of Biopsy 1. c) Multimodal image pair from a colposcopically normal area. d) Multimodal image pair at location of Biopsy 2. e) Average MTN diagnostic score from high-resolution images vs. distance along the two imaging paths; scores above the MTN threshold at 0.17 are classified as CIN2 + . Scale bars on high resolution images represent 200 µm. Microscope images in panel b and c were contrast enhanced to improve visibility.

Download Full Size | PDF

Representative multimodal image pairs are shown in Fig. 4(b-d). Figure 4(b) shows an image pair where the high-resolution imaging probe is located in the lesion at twelve o’clock; the high-resolution image shows regularly spaced nuclei and had an MTN diagnostic score below the threshold for high grade precancer. Figure 4(c) shows an image pair where the imaging probe is located in a colposcopically normal area below the lesion at twelve o’clock; the high-resolution image shows sparse, evenly spaced nuclei typical of normal squamous epithelium. The corresponding MTN diagnostic score was again below the pre-established threshold. Figure 4(d) shows an image pair where the high-resolution imaging probe is located at two o’clock in the colposcopically high grade lesion. The high-resolution image shows crowded, enlarged, pleomorphic nuclei, typical of high grade precancer. The corresponding MTN diagnostic score was above the pre-established threshold. The MMC accurately predicted the histologically confirmed CIN3 lesion.

Figure 4(e) shows a line plot of the MTN diagnostic scores along the two high resolution imaging paths, with the location and average scores of both biopsies highlighted on the graph. The MTN diagnostic scores throughout the first lesion all fall below the established threshold, consistent with the biopsy diagnosis of cervicitis. As the probe was translated from surrounding colposcopically normal tissue and into the second lesion, the MTN score increased and remained above the threshold across the entire lesion, consistent with the biopsy diagnosis of CIN 3.

Figure 5 shows MMC images acquired from a patient after real-time QC and diagnostic scoring were integrated into the user interface. A lesion was identified by standard colposcopy at nine to three o’clock; the clinical impression for the lesion was HSIL (Fig. 5(a)). Multimodal images were acquired as the probe was translated across the lesion. A total of one minute and nineteen seconds of multimodal imaging was conducted, comprising 4,986 HRME images and 1,950 Pocket Colposcope images (see Visualization 4). Of the 4,986 fastHRME images collected, 3,522 (71%) of the images passed the FFT based real-time quality control metric and were designated for real-time scoring by the MTN. Of these 3,522 frames, 2,915 (83%) were scored by the python executable during the imaging procedure (see Visualization 4). Immediately after imaging, the lesion was biopsied in two locations as shown on Fig. 5(a); the histologic diagnosis of each biopsy was CIN3.

 

Fig. 5. Real-time MMC results from a patient with a high-grade acetowhite lesion. a) Representative colposcopy image annotated with the high-resolution imaging path and biopsy locations. b) Multimodal image pair at location of Biopsy 1. c) Multimodal image pair at location of Biopsy 2. d) Average MTN diagnostic score from high-resolution images vs. distance along the two imaging paths; scores above the MTN threshold at 0.17 are classified as CIN2 + . Scale bars on high resolution images represent 200 µm.

Download Full Size | PDF

Representative multimodal image pairs are shown in Fig. 5(b-c). Figure 5(b) shows an image pair where the imaging probe is located in the lesion around ten o’clock; the high resolution image shows large crowded nuclei typically associated with high grade precancer. The corresponding MTN diagnostic score was above the threshold. Figure 5(c) shows another image pair where the high-resolution probe was located in the lesion around one o’clock. Again, the nuclei are crowded and enlarged and the MTN diagnostic score was above the threshold. Figure 5(d) shows a plot of the real-time diagnostic scores that were generated in real-time during multimodal acquisition along the imaging path, with the locations of biopsies highlighted. All areas imaged within the lesion had a score above the threshold, and the MTN accurately predicted that the lesion contained CIN3 at the biopsied locations.

 

Fig. 6. MMC results from three representative patients. a) A patient with a low-grade acetowhite lesion. The MTN diagnostic scores for all areas imaged within the biopsy location agreed with the histopathologic designation of cervicitis. b) A patient with a high-grade acetowhite lesion. The MTN diagnostic scores for all areas imaged within the biopsy location agreed with the histopathologic designation of CIN 3. c) A patient with no acetowhite lesion. The MTN diagnostic scores for all areas imaged within the biopsy location were just above the threshold, incorrectly predicting those areas contained CIN2 + .

Download Full Size | PDF

Figure 6 summarizes agreement between MMC images and histopathology in three additional patients. For each patient, the representative image taken after application of acetic acid is shown with the high-resolution imaging path and acquired biopsies overlaid. A plot of the MTN diagnostic scores along the high-resolution imaging path and the results of the histopathological examination of the acquired biopsy are also shown. Figure 6(a) shows results from a patient with a lesion from eleven to one o’clock with a colposcopic impression of LSIL. Histopathologic examination of the biopsy showed cervicitis. The MTN diagnostic scores for all areas imaged within the biopsy location were below the threshold, correctly predicting that area did not contain CIN2 + . Figure 6(b) shows results from a patient with a lesion surrounding the os with a colposcopic impression of HSIL. Histopathologic examination of both biopsies showed CIN 3. The MTN diagnostic scores for all areas imaged within the biopsy locations were above the threshold, correctly predicting those areas contained CIN2 + . Figure 6(c) shows results from a patient without any clinically abnormal sites identified by colposcopy. As per protocol, a single biopsy was obtained from a clinically normal site where HRME images were taken. Histopathologic examination of the biopsy of the clinically normal site showed cervicitis. The MTN diagnostic scores for all areas imaged within the biopsy location were just above the threshold, incorrectly predicting those areas contained CIN2 + .

4. Discussion

This manuscript describes the development and real-time clinical deployment of a novel multi-modal colposcope that integrates widefield and high-resolution imaging; high-resolution images are evaluated in real-time using a previously described convolutional neural network that provides automated quality control and tissue diagnosis. A new method that uses a combination of fiber tracking and widefield image co-registration is presented to correlate high resolution images and their diagnostic scores to the anatomic location from which they were acquired. Imaging results from a representative group of patients referred for colposcopy agree well with the gold standard of histopathology and illustrate the potential clinical role the MMC could play to improve the identification of cervical precancer in screen-positive women.

Multimodal imaging systems have been deployed in a variety of different anatomical sites; typically these systems combine a macroscopic image modality to identify suspicious areas, and a high resolution image modality to probe high-risk areas. For example, widefield fluorescence imaging has been combined with OCT, reflectance confocal microscopy, and nonlinear optical microscopy in the oral cavity [33–36]. Similarly, widefield autofluorescence and narrow-band imaging have been combined with high-resolution endoscopy in the esophagus [37]. Here, we explore a similar approach to improve cervical imaging.

The clinical examples presented here demonstrate the potential of the MMC to support secondary cervical cancer prevention programs. However, to achieve this goal, several improvements are needed. First, a limitation of the current MMC workflow is that it relies on a trained colposcopist to first identify suspicious lesions and then guide placement of the high-resolution imaging probe across those lesions. Colposcopy is subjective, and this approach is particularly challenging in LMICs which suffer shortages of properly trained colposcopists. Instead of relying on expert providers to identify abnormal lesions, ongoing studies are focused on implementing and optimizing artificial intelligence-based image analysis algorithms that can be deployed at the point-of-care for real-time detection of cervical precancerous lesions [18]. Incorporating wide-field algorithms into the MMC workflow could assist less experienced providers with the identification of lesions and placement of the high-resolution imaging probe to further evaluate those lesions.

A second limitation of the current MMC implementation is that the current diagnostic score is based solely on information from high resolution imaging. Although these algorithms were previously validated, they were developed from a single representative high-resolution image for a given anatomic site. Further optimization is needed to leverage the significant increase in field of view provided by the high-frame-rate FastHRME, which can rapidly acquire images across an entire lesion. Numerous high-resolution images can now be correlated to a single biopsy location that is hand annotated by the clinician after each imaging session. In a subset of 30 patients with post-biopsy images, we found that the biopsy annotations were recorded within < 1 mm of the final biopsied location. Furthermore, the MMC does not currently take advantage of the additional information available in widefield images, or the mutual information available from the co-registered dataset. As shown in Fig. 6(c), a pathologically benign region of metaplasia imaged with the HRME results in an MTN score that is just above the threshold for classification as CIN2+; incorporating information about the anatomic location of the imaged region available from the colposcopy images could potentially improve diagnostic performance. Future efforts will focus on multi-modal algorithm development to leverage information from both imaging modalities when generating a diagnostic score.

Finally, to enable prevention strategies in which patients are screened, triaged, and treated in a single-visit, MMC images must be analyzed in real time at the point-of-care. In this manuscript, we demonstrated real-time QC and diagnostic scoring using a previously described convolutional neural network, which allows for a large majority of high resolution images to be scored during the imaging procedure. Yet, the current method of prediction still relies on several post-processing steps that are performed after a patient visit. Even with diagnostic scores correlated to high-resolution images at the end of imaging, it is still necessary to perform video stabilization and fiber tracking to precisely correlate each diagnostic score to its respective anatomical location. Currently, the video stabilization and fiber tracking takes approximately thirty-five seconds to process one thousand frames. Therefore, further work is needed to develop real-time algorithms to stabilize colposcopy videos, to co-register multi-modal images, and to perform tracking of the tip of the HRME probe. A real-time implementation of multi-modal algorithms could potentially allow non-specialists to use the MMC to acquire an image of the entire cervix; automatically identify suspicious regions; acquire co-registered high-resolution images of nuclear morphometry from large fields of view across these areas; and obtain a real-time diagnostic score incorporating information from wide-field and high resolution imagery.

5. Citation diversity

Recent work in several fields of science has identified a bias in citation practices such that papers from women and other minority scholars are under cited relative to the number of papers in the field [38–41]. We recognize this bias and have worked diligently to ensure that we are referencing appropriate papers with fair gender and racial author inclusion.