1. Introduction

Optical coherence tomography (OCT) is a rapidly advancing ocular imaging technique that provides non-invasive and high-resolution imaging of the retina. OCT imaging has become an essential tool for diagnosing and managing retinal diseases, including choroidal neovascularization (CNV), central serous chorioretinopathy (CSC), diabetic macular edema (DME), and drusen. Accurate segmentation of lesions in OCT images is critical for evaluating disease progression and treatment outcomes.

Most existing studies have focused on OCT lesion segmentation [1–9]. Traditional OCT lesion segmentation methods [1–3] require manual feature extraction and demonstrate poor generalization performance. In recent years, deep learning-based approaches [4–9] have shown promising results in OCT lesion segmentation. For example, Hu et al. [4] developed a deep neural network with modified atrous spatial pyramid pooling (ASPP) to automatically segment subretinal fluid (SRF) and pigment epithelial detachment (PED) lesions in OCT images. Similarly, Hassan et al. [5] proposed the RFS-Net, a network that integrates ASPP, residual, and inception modules to segment multi-class retinal fluid (MRF), including intraretinal fluid (IRF), SRF, and PED. Liu et al. [6] introduced an improved U-Net segmentation method that automatically locates the fluid region by introducing an attention mechanism and utilizes dense skip connections to combine high-level and low-level features of the image. Xing et al. [7] proposed a network architecture based on FCN to simultaneously segment three types of pathological fluid lesions in OCT images. Wang et al. [8] proposed a novel multi-scale wavelet-enhanced transformation network for segmenting small lesions in OCT images. Parra et al. [9] proposed LOCTSeg for end-to-end automated segmentation of diagnostic markers in OCT B-scans.

Despite recent advances in OCT lesion segmentation, variances in lesion shape, variations in lesion shape, blurred boundaries, and interference from scattered plate noise cause it to remain challenging. Additionally, fully supervised deep learning methods typically rely on a large number of pixel-wise annotations, which are time-consuming and labor-intensive to obtain. In contrast, semi-supervised segmentation-based methods offer a promising alternative by effectively reducing the requirement for pixel-wise annotations while maintaining good performance.

In recent years, numerous studies have attempted to reduce the cost of pixel-wise annotations required for medical image segmentation tasks through semi-supervised learning. These approaches can be broadly categorized into several categories, such as extensions of the Mean-Teacher framework [10], including pixel-wise consistency [11], uncertainty assessment [12,13], and transformation-consistent [14] methods. Co-training [15–17] has also yielded good results in many semi-supervised medical image segmentation tasks. Deep adversarial training [18,19] is another approach that utilizes unlabeled data by employing discriminators to adjust the distribution of labeled and unlabeled data. Additionally, cross-task consistency methods encourage different tasks to achieve similar representations in predefined spaces, such as image reconstruction [20], segmentation and size regression [21], and level set regression [22]. Despite the success of semi-supervised learning in reducing the cost of pixel-wise annotations in medical image segmentation tasks, there are currently only a limited number of semi-supervised methods available for OCT lesion segmentation [23]. As a result, the accuracy of OCT lesion semi-supervised segmentation needs further improvement, especially for lesions with faint regions and various shapes and sizes.

To this end, we propose a novel semi-supervised framework for OCT lesion segmentation, termed Transformation-Consistent with Uncertainty and Self-deep Supervision (TCUS), to tackle the problems above. Firstly, TCUS introduces a transformation-consistent to enhance the regularization effect of the semi-supervised framework. It aims to ensure that the student network learns the transformation-invariant representation as the teacher network. Secondly, an uncertainty-guided transformation-consistent strategy is designed to address the issue of hazy lesion borders. The strategy utilizes uncertainty information from the teacher network to gradually guide the student network to learn from meaningful and reliable targets. Thirdly, self-deep supervision is employed by adding a segmentation head at different resolutions of the student network decoder to generate a set of segmentation results at different scales. It allows the framework to learn about lesions of various sizes and shapes by utilizing multi-scale information and improving the segmentation performance. We evaluate TCUS on the Kermany dataset [24] and private OCT dataset. The results show that TCUS improves significantly compared with the supervised baseline and outperforms existing semi-supervised segmentation methods. This paper’s main contributions are as follows:

2. Method

In this section, we present our proposed TCUS framework for OCT lesion segmentation, as illustrated in Fig. 1. The framework consists of a student network and a teacher network. The student network is trained with loss functions, while the teacher network weights are updated through exponential moving average (EMA) of the student network weights. The labeled data is fed into the student network and the unlabeled data is fed into both the student and teacher network. To enhance the effect of unsupervised regularization, we employ transformation-consistent. To mitigate the impact of unreliable predictions on the student network optimization and improve the reliability of the teacher network’s results, we estimate the uncertainty of the teacher network predictions. Moreover, we design self-deep supervision to capture the multi-scale information of the OCT images. In self-deep supervision, robust pseudo-labels are generated by the reliable teacher network, which combines all models from previous iterations, to improve the accuracy of the segmentation predictions.

 

Fig. 1. Overview of the proposed semi-supervised framework TCUS. $x_i$ represents the original image, and $y_i$ represents the pixel-wise annotation of the labeled image. $\psi _i$ refers to the transformation-consistent regularization, including rotation, flipping, and scaling operations. TCUS leverages the mean-teacher framework, where the student network is trained with a total loss that is a weighted combination of $\mathcal {L}_{sdp}$ (loss on labeled data) and $\mathcal {L}_{rdp}$ (loss on unlabeled data).

Download Full Size | PDF

2.1 Transformation-consistent semi-supervised framework

In the semi-supervised segmentation task, we consider a framework with a training set of $M + N$ input OCT images containing $M$ labeled OCT images and $N$ unlabeled OCT images. The labeled and unlabeled sets are defined as $\mathcal {D}_{L}=\left \{ x_i,y_i\right \}_{i=1}^{M}$, and $\mathcal {D}_{U}=\left \{ x_i\right \}_{i=1}^{N}$, respectively, where $x_i\in \mathbb {R}^{H \times W \times 3}$ represents the input OCT image and $y_i\in \left \{ 0,1\right \}^{H \times W}$ represents the corresponding pixel-wise annotation. Our objective is to minimize the following loss function:

The transformation-consistent [14] is introduced to enhance the regularization effect and make better use of the unlabeled data. TCUS involves three transformation operations, as shown in Fig. 1. For a given input $x_i$, the student network and teacher network are evaluated twice, resulting in two outputs: $o_i$ and $o'_i$, respectively. In the first evaluation, the transformation operation $\psi _i$ is applied to the prediction results of the student network. In the second evaluation, the transformation operation $\psi _i$ is applied to the input image of the teacher network. Additionally, to fully utilize the labeled data, the same transformation operation $\psi _i$ is applied to $y_i$. The overall loss of the framework is a combination of the supervised and unsupervised losses, defined as $\mathcal {L} = \mathcal {L}_s + \omega \mathcal {L}_r$, where $\mathcal {L}_s$ is the supervised loss, and $\mathcal {L}_r$ is the unsupervised regularization loss. The supervised loss $\mathcal {L}_s$ is computed using the labeled data and is typically a cross-entropy loss:

2.2 Uncertainty-guided transformation-consistent

One limitation of the framework discussed in Section 2.1 is that the input $x_i \in \mathcal {D}_U$ to the teacher network is unlabeled, which may result in predictions containing both correct and unreliable pixels, such as lesion boundaries in OCT images. These unreliable predictions can negatively impact the optimization of the student network. To address this issue, we draw inspiration from uncertainty theory [25–27] and estimate the epistemic uncertainty of the teacher network, which allows the framework to guide the student network to learn from the predictions of a more reliable teacher network.

The epistemic uncertainty is estimated by passing the input image through the teacher network $K$ times, with random Gaussian noise added to the input image and dropout applied during evaluation. This process generates different outputs for the same input, which can be used to approximate the uncertainty. The uncertainty $U_i$ for a given input $x_i$ is formulated as follows:

$\mathcal {L}_r$ is masked mean squared error loss and rewritten as Eq. (5):

2.3 Self-deep supervision

Inspired by the role of pyramid networks [28] and deep supervision [29] in fully supervised learning and its application in semi-supervised learning [30], we design self-deep supervision with multi-scale prediction to improve the performance of the framework. The decoder of student network is modified by adding a segmentation head that produces segmentation results at different scales (see Fig. 1), allowing the framework to learn about lesion areas of different sizes and shapes. The labeled data are supervised using pixel-wise annotations. Since there are no annotations available for the unlabeled data and upsampling processing may lead to inaccurate predictions, encouraging consistency between the student and teacher network predictions at each pixel directly may be affected by outliers and lead to a performance drop [12,19,27]. To alleviate this issue, the framework infers robust pseudo-labels by a reliable teacher network, and minimizes the difference between pseudo-labels and segmentation results at different scales using mean squared error loss.

More specifically, for an input image $x_i$, the student network $f_\theta \left ( x_i \right )$ generates a set of segmentation results $o_i^s$, where $o_i^s$ represents the prediction result from the $s$-th scale. Note that smaller $s$ means higher resolution and we use $S$ to represent the total number of scales. $o_i^s$ is upsampled to match the size of the input image and then applied the same transformation operation $\psi _i$, which is denoted as $p_i^s$. For the labeled data, the loss function is defined as Eq. (6):

The unlabeled data utilize the predictions of the teacher network as pseudo-labels to minimize the differences between $p_{i,v}^s$ and $o'_{i,v}$ by mean squared error loss. Directly employing the predictions of the teacher network as pseudo-labels may be problematic because it may contain pixels with unreliable predictions. To solve this problem, we incorporate self-deep supervision with uncertainty estimation:

It is worth noting that when $S=1$, TCUS is not able to leverage the multi-scale information of the image. In our experiments, we set $S = 3$, as discussed in more detail in Section 4.

2.4 Loss function

For the OCT lesion segmentation task, the overall loss function of the proposed TCUS is defined as follows:

3. Experiment

3.1 Experimental setup

3.1.1 Dataset

In this paper, our proposed TCUS is evaluated on two OCT datasets: the Kermany [24] dataset and the private OCT dataset.

The Kermany dataset is an OCT disease classification dataset with four OCT image types: CNV, DME, DRUSEN, and normal. We randomly select 600 DME and 600 DRUSEN images, and an experienced ophthalmologist labeled the lesions of each image. The training set contains 1000 images, and the test set consists of 200 images.

The private OCT dataset contains a total of 760 images of 34 diseased eyes from Changsha Aier Eye Hospital. It includes CNV and CSC, consisting of 458 and 302 images, respectively. All images have a resolution of 1020$\times$960. An experienced ophthalmologist annotates the ground truth of each image. We split the dataset into a training set of 650 images and a test set of 110 images for the purpose of training and evaluation.

3.1.2 Implementation details

We use the 2-D DenseUNet [31] as both the student and teacher network. The TCUS framework is implemented in PyTorch and trained on an NVIDIA GTX 1080Ti GPU. We employ the Adam optimizer with an initial learning rate of 1e-4 and a batch size of 12 during training. The Kermany dataset is trained for 300 epochs, while the private OCT dataset is trained for 500 epochs. Moreover, we follow the scaling, cropping, and flipping operations used in [14] for transformation operations $\psi _i$. To balance the segmentation performance and computational cost, all images are resized to 248$\times$248.

3.1.3 Evaluation metrics

For the proposed TCUS framework, dice coefficient (DI), pixel accuracy (ACC), sensitivity (SE), and specificity (SP) are used to measure the segmentation performance. These evaluation metrics are shown in Eq. (9).

3.2 Performance evaluation and analysis

3.2.1 Experiments on the Kermany dataset

Ablation study. We conduct ablation experiments on the Kermany dataset using 10${\%}$ labeled data to validate the effectiveness of our TCUS framework. Table 1 shows the results, and we observe that: (1) TCUS significantly improves the DI by 3.06${\%}$ compared to the supervised baseline, demonstrating that our method can fully utilize the unlabeled data. (2) The introduction of uncertainty is beneficial for the OCT lesion segmentation task, resulting in a 0.5${\%}$ improvement in DI. Uncertainty ensures the relative reliability of the pseudo-labels obtained from the teacher network. (3) Self-deep supervision is crucial in our proposed framework. Capturing multi-scale information effectively improves segmentation performance.

Table 1. Ablation study on the Kermany dataset with 10${\%}$ labeled data. "TC" denotes Transformation-consistent, "UG" denotes Uncertainty-guided, and "SDP" denotes Self-deep supervision, respectively.

View Table | View all tables in this article

To further illustrate the role of uncertainty and self-deep supervision in the TCUS, we also visualize the segmentation results of the ablation study. Figure 2 shows the visualization results, and two cases are given. When trained with only 100 labeled images, the supervised baseline obtained poor segmentation results with many false positives (yellow regions in Fig. 2). When using unlabeled data as additional training data, the framework incorporating transformation-consistent gets more precise segmentation results than the supervised baseline. By introducing uncertainty into the framework, the isolated FP regions can be effectively removed (orange arrow in Fig. 2). Furthermore, with the integration of self-deep supervision, TCUS obtains more accurate prediction results for lesions of different sizes and shapes (green box in Fig. 2).

 

Fig. 2. Visualization results of ablation studies on the Kermany dataset with 10${\%}$ labeled data. Red regions represent lesions that are segmented correctly, yellow regions represent lesions that are incorrectly, and blue regions represent those that are not segmented, respectively.

Download Full Size | PDF

Results with different amounts of labeled data. Table 2 presents the effect of the TCUS framework on the performance of OCT lesion segmentation with different amounts of labeled data. The results show that regardless of the ratio of labeled data in the training set, TCUS consistently outperforms the supervised baseline (which is trained with labeled data only), indicating that TCUS can effectively leverage the unlabeled data. As the number of labeled data increases, both the supervised baseline and TCUS achieve higher segmentation performance. Notably, when trained with only 5${\%}$ of labeled data (50 labeled images), TCUS achieves the largest performance improvement over the supervised baseline (DI = 3.96${\%}$). However, the performance improvement of TCUS becomes limited as the number of labeled data increases.

Comparison with other semi-segmentation methods. We compare TCUS with some classical and state-of-the-art semi-supervised segmentation methods, including Mean Teacher (MT) [10], Uncertainty Aware Mean Teacher (UAMT) [12], TCSM_v2 [14], Cross Pseudo Supervision (CPS) [32], and Uncertainty Rectified Pyramid Consistency (URPC) [30]. For a fair comparison, 2-D DenseUNet is used for the backbone of all aforementioned methods. Table 3 shows the quantitative results of our TCUS with these semi-supervised segmentation methods. When using 10${\%}$ labeled data for training, all semi-supervised methods improve in DI compared with the supervised baseline. Among the existing methods, TCSM_v2 demonstrates the best segmentation performance. However, TCUS outperforms TCSM_v2 by achieving the best segmentation performance with a 1.77${\%}$ improvement in DI, highlighting the ability of TCUS to effectively utilize information from unlabeled data. Figure 3 shows the visualization results of the supervised baseline and semi-supervised methods using 10${\%}$ labeled data. Note that TCUS outperforms other semi-supervised methods in segmenting lesions in OCT images. Specifically, TCUS achieves higher segmentation accuracy, particularly when the lesion area is blurred (line 2 in Fig. 3). Furthermore, when using 20${\%}$ labeled data for training, TCUS also outperformed the existing semi-supervised segmentation methods, as shown in Table 3.

 

Fig. 3. Lesion segmentation results of different methods on the Kermany dataset. The first and second lines are DME, and the third and fourth lines are DRUSEN. Red regions represent lesions that are segmented correctly, yellow regions represent lesions that are segmented incorrectly, and blue regions represent lesions that are not segmented.

Download Full Size | PDF

Table 2. Segmentation performance of Kermany dataset trained with different amounts of labeled data.

View Table | View all tables in this article

Table 3. Performance comparison on the Kermany dataset. "10${\%}$" and "20${\%}$" denote training with "10${\%}$" and "20${\%}$" labeled data, respectively.

View Table | View all tables in this article

3.2.2 Experiments on the private OCT dataset

We also conduct experiments on the private OCT dataset. Table 4 shows the experimental results. TCUS achieves state-of-the-art performance compared to other semi-supervised segmentation methods with only 10${\%}$ and 20${\%}$ labeled data. Specifically, when trained with 10${\%}$ and 20${\%}$ labeled data (65 and 130 labeled images, respectively), TCUS outperforms the supervised baseline by 6.59${\%}$ and 5.0${\%}$ in DI. This demonstrates the robustness of TCUS in significantly improving the performance of the supervised baseline for different diseases. Additionally, Table 4 shows that TCUS effectively utilizes a large amount of information from unlabeled data to achieve the greatest improvement in segmentation performance when only 10${\%}$ labeled data is available for training.

Table 4. Performance comparison on the private OCT dataset. "10${\%}$" and "20${\%}$" denote training with "10${\%}$" and "20${\%}$" labeled data, respectively.

View Table | View all tables in this article

4. Discussion

We also conduct a discussion on three parameters involved in TCUS to validate its robustness across different parameter settings.

Table 5 shows the effects of using different values of $K$ (Eq. (4)) on segmentation performance. We conduct experiments on the Kermany dataset with 10${\%}$ labeled data. It can be observed that regardless of the value of $K$, TCUS outperforms the supervised baseline in terms of segmentation performance. Additionally, TCUS demonstrates relatively roubles performance with minimal fluctuations across different $K$ values. However, it is worth noting that as K gradually increases, the performance of TCUS also improves. Theoretically, as the value of $K$ increases, the estimated uncertainty becomes closer to the actual value. It aligns with our experimental results. However, when using larger values of $K$, training time is also longer [27]. Thus, we set $K$ to 10 in all experiments.

Table 5. Testing DI under different $K$ values

View Table | View all tables in this article

Table 6 shows the influence of using different $th$ values (Eq. (5)) on the segmentation performance. It can be seen that the best segmentation performance is achieved when $th$ is set to 0.15. When $th$ is too small ($th=0.05$), TCUS selects too few pixels for training, resulting in insufficient training. When $th$ is too large ($th=0.25$), TCUS selects pixels with high uncertainty for training, which adversely affects the optimization of the parameters of the student network. Both of these cases lead to a decrease in lesion segmentation performance. Therefore, it is crucial to select an appropriate value of $th$ for training. In all experiments, we set $th$ to 0.15.

Table 6. Testing DI under different $th$ values

View Table | View all tables in this article

Table 7 illustrates the impact of different values of $S$. It can be observed that increasing $S$ from 1 to 3 gradually improves the lesion segmentation performance. However, when $S$ is set to 4, the segmentation performance is decrease compared to $S=3$. It can be attributed to the lower resolution of the obtained $p_i^4$, which leads to the loss of boundary details. Therefore, in the experiments, we set $S$ to 3.

Table 7. Testing DI under different $S$ values

View Table | View all tables in this article

In summary, TCUS offers several advantages in the field of semi-supervised OCT lesion segmentation. Firstly, it effectively segments OCT image lesions even with limited labeled data. Secondly, it improves segmentation performance by leveraging unlabeled data. Thirdly, it prevents negative impacts on training performance through the implementation of an uncertainty-guided transformation-consistent strategy. Lastly, TCUS provides self-supervision, enabling the learning of lesion information with varying shapes and sizes.

However, it is important to acknowledge certain limitations of TCUS. Firstly, TCUS relies on a set of random transformation operations applied to input images, which may lead to suboptimal transformations and performance. Additionally, uncertainty estimation necessitates multiple forwards passes, resulting in longer training times. Moreover, TCUS has not been tested for multi-class segmentation as binary masks were the only available data. Lastly, TCUS is designed based on the assumption that the data originates from a single domain with a shared data distribution, potentially impacting performance when incorporating data outside the distribution range.

5. Conclusion

In this paper, we introduce TCUS, a semi-supervised segmentation framework designed specifically for OCT lesion segmentation. We propose the uncertainty-guided transformation-consistent strategy in TCUS to address the issue of unreliable predictions made by the teacher network on unlabeled images. By evaluating uncertainty in the teacher network and enforcing transformation-consistent, TCUS enhances the efficacy of unsupervised regularization and guides the student network to learn from meaningful and reliable targets, thereby improving lesion segmentation performance. Additionally, TCUS incorporates self-deep supervision, which involves generating multi-scale predictions on OCT images to capture the characteristics of lesion regions with diverse shapes and sizes, resulting in more accurate segmentation outcomes.

Through extensive evaluation on two OCT datasets, TCUS demonstrates a significant performance improvement over the supervised baseline and outperforms other existing state-of-the-art semi-supervised segmentation methods.