1. Introduction

Descemet membrane endothelial keratoplasty (DMEK) is the preferred posterior lamellar keratoplasty procedure for treating cases of symptomatic irreversible corneal endothelial cell dysfunction [1,2]. Posterior lamellar surgeries constitute the majority of grafting procedures in the developed world [3]. The thin (∼30µm) and vulnerable DMEK graft – consisting of the Descemet’s membrane and endothelium – is inserted as a roll and unfolded in the anterior chamber of the eye before fixation on the posterior surface of the recipient cornea [4]. A correct orientation of the graft – with the endothelium facing away from the cornea – is imperative. An inadvertently incorrectly positioned graft (i.e., upside-down) will result in severe corneal edema, damage to the graft's endothelial cell layer, and the subsequent need for repeated surgery [5,6].

The assessment of the graft’s orientation can be challenging and several methods have been described to aid the surgeon in determining the orientation. Currently, the Moutsouris sign, ink-stamps, and circular cuts are used to determine intraocular graft orientation [4,7–9]. However, poor visualization of the anterior chamber and graft hinders a proper assessment [10–12]. In addition, the presence of the Moutsouris sign is not always self-evident and both stamps and cuts damage the graft resulting in endothelial cell loss. More recently, intraoperative optical coherence tomography (iOCT) has been used to determine graft orientation, as the iOCT signal is not perturbed by corneal edema [10–14]. Residual stromal fibers in the Descemet’s membrane of the DMEK graft result in a distinctive inward curve of the graft’s ends indicative of a correct orientation, which can be visualized and assessed using iOCT (Fig. 1.) [13–15]. This natural curling behavior of DMEK grafts can be well appreciated on the iOCT image, thereby preventing the need to use manipulation, cutting, or marking to determine the graft orientation, thus preventing endothelial cell loss.

 

Fig. 1. Two cross-sectional intraoperative OCT scans of the cornea. The natural rolling motion of the graft in Descemet membrane endothelial keratoplasty (DMEK) can be used to determine the graft’s orientation. The top image depicts a correctly oriented DMEK graft, indicated by the distinctive upward curve towards the recipient’s cornea. The bottom image depicts an incorrectly oriented graft (i.e., upside-down), indicated by the curling motion away from the recipient’s cornea.

Download Full Size | PDF

Several studies have reported on the use of iOCT during DMEK surgery for determining the orientation of the graft. In all studies the graft orientation could be correctly determined based on the inward rolling of the graft edges visible on the cross-sectional iOCT image [10,12,14,16]. Importantly, the surgeon was able to assess the graft orientation in cases where assessment of the Moutsouris sign or S-stamp was challenging or not possible [10–12]. However, manual assessment of graft orientation on iOCT images can be time consuming and prone to interpretation errors. In particular, when OCT image quality is suboptimal or the graft edges display little inward rolling. We believe an automated tool will aid the surgeon in fast and accurate evaluation of the orientation, thereby improving surgical workflow and reducing the risk of errors.

Several studies have reported on applications of (automated) image analysis of iOCT images showing promising potential for improving clinical decision making and clinical outcomes [17–23]. These studies used different methodologies for segmenting the area of interest from the iOCT image. For example, Weiss et al. used geometric modeling of iOCT images to track the orientation and location of a surgical needle [20]. Using a similar method, Xu et al. developed an automated algorithm to segment the fluid interface gap in Descemet stripping endothelial keratoplasty achieving a high segmentation accuracy [18,19]. In contrast, Roodaki et al. used deep learning to segment different anatomical structures in the anterior segment of the eye in iOCT images, to automatically position the OCT scan area on an anatomy of interest using voice control [21,22]. Furthermore, Keller et al. demonstrated the use of iOCT-guided robotic ophthalmic surgery using volumetric OCT scans and reinforcement learning [23]. To the best of our knowledge no automatic image analysis tool has been developed for analysis of DMEK graft orientation.

Here, we present an automated image analysis method for evaluation of the DMEK graft orientation using iOCT. The method includes a deep learning-based segmentation model to extract the DMEK graft from the iOCT scan. Then the degree of inward rolling by the graft is assessed and related to graft orientation. Our contributions include the development of a method for in-vivo DMEK graft segmentation, a pipeline of post-processing steps to obtain the graft’s curvature, and a method to relate graft curvature to graft orientation.

2. Methods

2.1 Data & preprocessing

All OCT-scans in this study were acquired during DMEK surgery at the ophthalmology department of the University Medical Centre Utrecht between May 2016 and October 2020 using the “No-Touch” technique for DMEK as described by Dapena et al. [4] DMEK grafts were cultured and provided pre-cut by the Euro Cornea Bank (Beverwijk, the Netherlands) and Amnitrans (Rotterdam, the Netherlands). During surgery, iOCT-scans of the anterior segment were made with a commercially available spectral domain microscope integrated OCT system (Zeiss Lumera 700 RESCAN, Carl Zeiss Meditec, Jena, Germany), using the two-line cross-sectional setting. The iOCT system has a wavelength of 850 nm and an axial resolution of 5.5 µm. The system acquires 25 two-line cross-sectional scans per second. This study was performed in accordance with the Declaration of Helsinki and Dutch law regarding research involving human subjects. Ethical approval for this study was waived by the Ethics Review Board of University Medical Center Utrecht (METC no. 18-370) iOCT-scans of the DMEK procedures are embedded in the surgical video feed. The video feed was qualitatively reviewed for scan quality and visibility of the graft during determination of graft orientation (i.e., before adhering the graft). Scans were excluded if the graft was not visible at all or not unfolded. Included iOCT-scans were manually extracted from the video feed using the FFmpeg tool (version 3, 2016, FFmpeg Developers). Each cropped frame contained a single cross-sectional iOCT-scan (iOCT-frame). The ground truth of the graft orientation, either correctly oriented or upside-down, in each iOCT-frame was set by an experienced grader (M.B.M.) who had access to the preceding and follow-up frames and postoperative clinical information. The orientation of each graft was subsequently graded by two corneal surgeons (experts’ opinion; R.W. and A.O.) based on a single iOCT-frame and blinded for the outcome (i.e., without access to the preceding and follow-up frames or postoperative clinical information).

A total of 335 iOCT-frames from 89 DMEK surgeries were obtained; 127 iOCT-frames measuring 550 × 275 (width × height) pixels acquired before 1-1-2019 and 208 iOCT-frames measuring 610 × 275 pixels acquired from 1-1-2019 onwards. The more recently acquired scans were of better image quality due to an improved scan protocol and we selected 100 recent iOCT-frames from 21 patients as a test set for final evaluation of our models. All other iOCT-frames (n = 235) were used for development and optimization of the image analysis methods and will be referred to as the development set. This development set was again divided into a training set (n = 202) and a validation set (n = 33) to determine the optimal model. The data split was done on a patient level to ensure no overlap exists between the train, validation, and test sets. The graft locations were manually annotated in the iOCT frames with marking points (image coordinates) along the graft and converting the resulting contour to a binary mask of an area containing the graft. Zero-padding was used to ensure all iOCT-frames were of width 610 for training the AI segmentation model. As a final preprocessing step, all frames were resized to 576 × 256 pixels for compatibility with the U-Net architecture.

Our image analysis tool consists of three steps (Fig. 2.). First, the area containing the DMEK graft was segmented from the iOCT-frame using a deep learning-based segmentation model. In the subsequent post-processing step, the resulting mask was converted into a one-pixel thick line representation of the graft. Artifacts and gaps in the line were removed and the graft’s endings located. Finally, we build upon the work by Steven et al. to assess the curling behavior of the graft [13] and we relate curvature of the line segment to graft orientation. The predicted graft orientation was then compared to the ground truth and classification by the corneal surgeons.

 

Fig. 2. A schematic representation of the pipeline of the intraoperative OCT DMEK graft orientation model. Shown in section A are the image acquisition process and the automatic segmentation model. The predicted segmentation is the mean of an ensemble of 12 deep learning models. Section B shows the key post-processing steps to obtain a one-pixel line representing the graft. In section C the left top image is a schematic representation of the signed curvature. The top right image shows the polygons fitted to the line representing the graft and the defined curvature parameters. The bottom images of section C show the decision tree for selecting the curvature parameter and determining the orientation.

Download Full Size | PDF

2.2 Segmentation

For segmentation of the DMEK graft from the iOCT frame, we used a deep learning approach [24]. Our model consists of an ensemble of 2D U-Nets [25]. The U-Net architecture incorporates a large contextual region and has resulted in state-of-the-art performance for many biomedical image segmentation tasks [24,25]. Training was done using iOCT-frames and the corresponding manually annotated masks of the trainset (n = 202). Data-augmentation was used to expand the variability in appearance of the training data set. Augmentations included random affine transformations that were applied to the iOCT frames and corresponding mask annotations: translation (< 10 pixels), rotation (< 3°), scaling (< 10%) and vertical reflection. In addition, we applied intensity shift (< 10/256), contrast shift (< 0.1) and addition of white noise (< 10/256) to the iOCT frames. Experiments with different learning rates and loss functions indicated that different models lead to different types of segmentation errors for the validation set (n = 33). We therefore constructed an ensemble of 12 U-Nets: Five models were trained using Dice loss and initial learning rates of 0.0001, 0.0002, 0.0003, 0.004, and 0.0005. Another seven models were trained based on a weighted binary cross-entropy (WBCE) loss, with a weight determining the relative penalty for misclassified foreground pixels (= DMEK graft) in comparison to background pixels. Beta values of 0.5, 1, 2, 4, 8, 12, and 16 were used, and the WBCE models were trained with an initial learning rate of 0.0003. All models were optimized with Adam for 3500 iterations where the initial learning rate was multiplied by 0.3 every 1400 iterations [26]. Each U-Net in the ensemble provides a segmentation prediction and the final segmentation was obtained by taking the mean across the 12 segmentation maps (Fig. 2(A)).

2.3 Post-processing

To ensure the graft is represented as a single smooth line, a post-processing algorithm was developed (Fig. 2(B)) consisting of the following steps: (1) Median filtering (filter size = 2 × 2) to reduce noise; (2) Binarization to assign pixels to either background or graft class; (3) Skeletonization to obtain the topological skeleton (one-pixel thick) of the segmented areas [27]; (4) Removal of small islands (<100 pixels) to get rid of small areas falsely identified as graft; (5) Morphological pruning to remove side-branches from the remaining skeletonized line segments. We implemented the pruning by finding the longest pathway for each segment and removing any pixels not belonging to these paths. The longest pathway was determined by determining the largest number of pixels needed to travel along the skeleton from any branch end to any other branch end; (6) Closing of gaps between endings of line segments with a Euclidian distance less than 100 pixels, by in painting with a one-pixel thick straight line. For the post-processing steps, all design choices and parameter selections were based empirically on results for the validation set.

Next, the largest line segment was identified and the coordinates of every 15th pixel along the line were used to compute a parametric cubic smoothing spline curve. The parametrization was then used to resample 100 points along a smooth line representing the graft.

2.4 Graft orientation

To determine the orientation of the graft, we first assessed the rolling behavior of the graft. The rolling behavior can be measured as the signed curvature κ, similar to the previously described method by Steven et al. [13].

A Python implementation of the Matlab LineCurvature2D package [28] was used to calculate the local curvature at each of the 100 graft points obtained with the post-processing step (Fig. 2(C)). Summing all local curvatures for the length of the graft (L), the total curvature (κ_total) can be calculated, taking into account the distance arc length steps (ds):

2.5 Evaluation and statistical analysis

Performance of the automatic DMEK orientation model was evaluated for the test set iOCT frames (n = 100). The predicted orientation was compared to the ground truth orientation and a receiver operating characteristic (ROC) curve was determined by varying the κ_threshold threshold. Sensitivity was defined as the accurate prediction of correctly oriented grafts while specificity represents true prediction of incorrectly oriented grafts (i.e., upside-down). For comparison of the automatic method with the corneal specialists, an operating point was chosen by setting a single value for κ_threshold, based on an optimal F1-score. The set κ_threshold was used for all prediction methods. All statistical analysis were performed using R statistical software version 4.0.3 (CRAN, Vienna, Austria). The ROC plots were produced using the ROCR package (version 1.0-11).

Quality of the segmentations was evaluated using the Dice score. Additionally, we evaluated a pipeline that uses the manual annotated masks instead of deep learning-based segmentations. The post-processing of these segmentations was similar to the end-to-end pipeline, although steps (4) removal of pixel islands and (6) closing of gaps were skipped. This ‘semi-automatic method’ was evaluated on the test set as well as the recently acquired frames of the development set (n = 108), as these are comparable to the frames in the test set in terms of frame size and resolution.

3. Results

Of the 335 iOCT-frames included in this study, 255 frames contained correctly oriented grafts versus 80 incorrectly oriented grafts (i.e., upside-down). In 195 iOCT-frames the graft was free floating (i.e., no contact with other ocular structures) and in 134 iOCT-frames a mirroring artefact of the cornea was present, which (partially) overlapped with the graft in 65 iOCT-frames. Mean age of the graft donors was 74 years (range: 55-88). The indications were Fuchs endothelial corneal dystrophy (n = 79), Pseudophakic bullous keratopathy (n = 9), and graft failure (n = 1). Segmentation performance on iOCT-frames of the test set was similar across the 12 deep learning models, with Dice scores ranging from 0.72 to 0.74. For the ensemble, where the mean prediction of the 12 models was used, the Dice score was 0.75.

3.1 Performance of the DMEK orientation model

In Fig. 3 the ROC curves are displayed for the DMEK orientation model using the deep learning-based segmentations (automatic method) and manually annotated grafts (semi-automatic method). Additionally, the performance of the corneal specialists is shown for both datasets. The automatic method achieves an AUC of 0.84, which is considered a good to excellent predictive power [29]. The semi-automatic method performs even better than the automatic method, with an AUC of 0.92 for both the development set and test set and is comparable to the performance of the corneal specialists using the same information (i.e., a single iOCT-frame). Causes for the gap in performance between the automatic and semi-automatic methods include segmentation and post-processing errors, which are described in detail in the Qualitative analysis.

 

Fig. 3. Receiver operating characteristic curves of the performance of the DMEK orientation model in the test set (n = 100) and the most recent frames of the development set (n = 108), obtained by varying the curvature threshold. The circles and squares represent the performance by the corneal specialists. The dashed 45-degree line constitutes a model with no discriminative power.

Download Full Size | PDF

In line with the aim of this study – determining graft orientation using iOCT – the optimal trade-off between sensitivity and specificity was selected to determine κ_threshold (Fig. 3). The detailed results of the DMEK orientation model at κ_threshold are shown in Table 1. The automated method was able to correctly identify the grafts’ orientation in the iOCT frames in 78% of the iOCT-frames in the test set and in 86% of the iOCT-frames for both the development and test set using manually segmented grafts. The automatic method achieved a high sensitivity (0.82) and moderate specificity (0.69). Thus, the model was able to correctly classify the majority of the correctly oriented grafts, though had only a moderate predictive power to correctly classify incorrect oriented grafts. Using the manually annotated grafts leads to slightly better sensitivity and markedly higher specificity compared to the automatic method. The outcomes of the semi-automatic methods were comparable to the performance of the corneal specialist (Table 1). If only the segmentation of a single U-Net was used, the AUC varied between 0.78 and 0.86 across the 12 models in the test set (range accuracy: 0.68–0.80).

Table 1. Performance analysis of the orientation model and corneal specialists

View Table

3.2 Qualitative analysis of segmentation and post-processing

All deep learning-based segmentations in the test set were qualitatively evaluated for errors in the predicted segmentation or post-processing. In 54 iOCT-frames a near perfect representation of the graft was achieved after post-processing compared to the manually labeled results (Fig. 4(A)) and in 46 iOCT-frames noticeable segmentation (n = 37) and/or post-processing errors (n = 13) were present after post-processing Fig. 4(B)-(G)). A total of 22 grafts were incorrectly classified using the automatic method of the orientation model, because of segmentation errors in 8 frames, post-processing errors in 2 frames, and a limited differentiative predictive power of the model in 12 frames (i.e., in both the automated and manual method these grafts were incorrectly classified regardless of any errors; Fig. 4(H)). In 29 iOCT-frames containing errors the model still correctly predicted the orientation.

 

Fig. 4. Examples of correct and incorrect segmentation and post-processing: a near perfect segmentation (A), a segmentation error at image boundaries (B), a segmentation error at the graft end (C), segmentation gaps resulting in partial segmentation (D), segmentation gaps too wide to connect in the post-processing (E), segments wrongly connected during post-processing (F & G), correct segmentation resulting in an incorrect prediction (H).

Download Full Size | PDF

The majority of the segmentation errors were considered minor, such as slightly incomplete segmentation of the graft ends or at the image boundary (Fig. 4(B) & Fig. 4(C)). Notwithstanding, despite considered minor these errors may affect the algorithms performance. Partial segmentation of the graft or large gaps between segments were considered large segmentation errors (Fig. 4(D)). Causes for large segmentation errors included: corneal mirror artefacts, background noise, hypo reflectance of the graft, and contact of the graft with the cornea or iris.

All post-processing errors occurred during filtering of the frames and connecting the line segments resulting in partial or wrong segmentation. During filtering smaller segments (<100 pixels) were removed, which resulted in gaps too large to bridge in the subsequent step. Similarly, in cases with large gaps (>100 pixels) the line segments were not connected and the smaller segments were removed after identification of the largest segment (Fig. 4(E)). In some frames the line segments were connected with wrong segments or an image artefact falsely identified as graft (e.g., fluid reflection, the lens capsule) resulting in an incorrect representation of the graft Fig. 4(F) & Fig. 4(G)).

4. Discussion and conclusion

In this exploratory study we developed a method to in-vivo segment a DMEK graft using a deep-learning approach and demonstrated that an image analysis tool that can automatically identify the orientation of a DMEK graft using iOCT. Several studies have pointed out the lack of (integrated) image analysis tools and clinical decision support systems (CDSS) for iOCT that can improve the clinical value [30–33]. Computerized CDSS have the potential to improve outcomes, optimize treatments, and improve workflow efficiency [34–36]. We believe our tool might be of similar value for iOCT by improving and standardizing clinical decision making. Moreover, the tool could help ease the learning curve for starting surgeons and aid experienced surgeons in the transition towards DMEK [13].

Determining graft orientation using iOCT is arguably more reliable and safer compared to other methods in use (i.e., the Moutsouris sign and various stamps/cuts) [10,12,14,16]. However, current manual review of both live and static iOCT-scans for DMEK orientation can be time consuming and disrupt the surgical workflow hindering implementation and sustainable use of iOCT [30,31]. Our proposed automatic image analysis may alleviate these hurdles by aiding the surgeon in determining graft orientation and may reduce interpretation errors.

In recent years corneal OCT image analysis has gained interest. Several studies showed the ability of automatic tools to successful detect a DMEK graft in OCT images, quantifying graft detachment after lamellar corneal transplant surgery [37–40]. Our automatic method has a good to excellent predictive power [29] and when using manually annotated grafts the performance of our model improves considerably and is comparable to the performance of both corneal specialists. The gap in performance between the automatic and manual method is primarily the result of segmentation and post-processing errors, which in turn resulted in wrong predictions as shown with the qualitative analysis of the end-to-end outcome. Automatic segmentation of iOCT imaging is challenging because of the design and dynamic use of iOCT, which may result in higher signal noise, variable image quality, image decentration, and prevents standardized image acquisition [19]. Notwithstanding, we consider our dataset a realistic representation of images acquired in clinical practice for determining the orientation and therefor consider the performance generalizable to other datasets.

The threshold for determining the graft’ orientation used in the results in Table 1 was slightly negative after optimizing the F1-score (i.e., optimal operating point), which corresponds to a slight curve downwards. This makes sense since the cornea itself also curves downward and the floating DMEK typically partly follows the shape of the cornea. In this study the optimal trade-off between sensitivity and specificity was chosen to optimize the predictive power of the model. However, it can be argued that depending on the use case or user expectations, either sensitivity or specificity may be more important.

The difference in AUC between the automatic method and the pipeline with manual annotated grafts indicates that improvements for the automatic method can be achieved by improving the automatic segmentation. In particular, correct segmentation of the graft endings could contribute to a better estimate of the graft curvature. The deep learning-based segmentation can potentially be improved by the addition of more training data, including a wider variety of anatomies and image artifacts. If a large enough training set could be obtained, an end-to-end deep learning method could be considered, where a classification model is trained only on orientation labels. However, even if enough training data would be available, such a method would come at the cost of having a CDDS without explanation for the decision-making, which could hamper acceptation by the end users. Alternatively, future research could investigate a segmentation approach that uses shape constraints [41,42], such as the fact that the graft is a continuous and smooth structure. Such an approach should take into account that not the whole graft necessarily lies in the field of view. We also experimented with the addition of extra frames to the input taken shortly before or after the investigated iOCT image, in which the location and orientation of the DMEK graft slightly differed from the center frame. For example, we added the 5th and 10th frames before and after the center frame as additional channels to the input, similar to Vu et al. [43], hypothesizing that the extra information would help the learning process. However, no benefits were found from this step and it was omitted for the final ensemble.

It should be noted that assessment of graft orientation based on a single frame does not reflect clinical practice. Instead, a corneal specialist would reduce uncertainty by assessing multiple frames or manipulate the graft until orientation is evident. Future work could incorporate such a strategy in the automatic image analysis pipeline, for example by using a recurrent neural network on follow-up frames [44,45]. For clinical implementation, the image analysis pipeline needs to be directly applied to the video-feed. In this research, iOCT frames were qualitatively reviewed for image quality and presence of characteristics on which orientation could be determined. However, the qualitative analysis indicated not every frame contains enough information for evaluation of the orientation. Future research could include an automatic frame-based quality assessment, or an uncertainty estimate and only provide a prediction if the certainty is high. A challenge for real-time image analysis is the speed at which the segmentation and post-processing can be performed. Here an ensemble of 12 U-Nets was used for the segmentation, but this might require more computational power than standardly offered with an iOCT system resulting in a longer inference time required to determine the orientation. However, the benefit of an ensemble compared to a single U-Net seemed marginal and perhaps an ensemble is not required if more annotated training data is used. Another solution could be the use of knowledge distillation techniques, which have recently been proposed to train a single segmentation model that performs similar to an ensemble [46,47]. It should be noted that we only tested our methods for a single OCT-system at a single center and additional research is needed to evaluate the feasibility for other settings. Especially the deep learning-based segmentation is known to often poorly generalize to out of distribution data. To ensure generalization to a wide variety of scanners and scanning parameters, training could be done using data from various OCT systems, or with the use of extensive data-augmentation.

In conclusion, we present an automated image analysis method for iOCT to detect a DMEK graft, quantify the curvature, and determine the graft’s orientation. Our future research efforts will focus on improving automatic segmentation and predictive certainty of our algorithm.