Open Access
Add to BibSonomy
Abstract
Diffuse optical tomography (DOT) is a promising technique that provides functional information related to tumor angiogenesis. However, reconstructing the DOT function map of a breast lesion is an ill-posed and underdetermined inverse process. A co-registered ultrasound (US) system that provides structural information about the breast lesion can improve the localization and accuracy of DOT reconstruction. Additionally, the well-known US characteristics of benign and malignant breast lesions can further improve cancer diagnosis based on DOT alone. Inspired by a fusion model deep learning approach, we combined US features extracted by a modified VGG-11 network with images reconstructed from a DOT deep learning auto-encoder-based model to form a new neural network for breast cancer diagnosis. The combined neural network model was trained with simulation data and fine-tuned with clinical data: it achieved an AUC of 0.931 (95% CI: 0.919-0.943), superior to those achieved using US images alone (0.860) or DOT images alone (0.842).
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Diffuse optical tomography (DOT), a non-invasive functional medical imaging technique, reconstructs the optical properties of biological tissue based on the collection of diffusely reflected light [1]. DOT is increasingly used in several areas, including brain imaging, breast cancer diagnosis, and cancer treatment response monitoring [2–4]. Quantitative estimations of oxygenated, deoxygenated, and total hemoglobin concentrations can be acquired by utilizing multiple wavelengths in the near-infrared range (NIR) [5,6]. However, due to the extensive scattering inside biological tissue, the reconstruction is an ill-posed, underdetermined problem, which leads to low resolution and low accuracy in the reconstructed images. To ease the reconstruction problem, other modalities, such as X-ray CT [7], MRI [8], and ultrasound (US) [9,10], have been introduced to provide prior structural information. The structural information can significantly reduce the number of unknowns in the inverse problem, improving the reconstruction accuracy of DOT.
Our group has developed a safe, portable US-guided DOT system that combines DOT system with a commercial US system [11]. The co-registered US is used to localize a breast lesion, and the DOT reconstruction can be performed by dividing the entire breast imaging volume into a fine-meshed region of interest that contains the lesion and a coarse-meshed background of breast tissue [12]. This dual-mesh scheme improves the reconstruction by significantly reducing the number of imaging voxels to be processed.
In recent years, deep learning-based image reconstruction has grown rapidly and shown promising results in diffuse optical imaging and spectroscopy [13–21]. Yoo used a convolutional neural network to learn the non-linear mapping between measurements and optical anomalies inside tissue [14]. Feng used a fully connected neural network to recover the optical properties of the lesion and showed a significant improvement in image quality [15]. Zhang [16] and Sabir [17] used a convolutional neural network to estimate the average breast tissue optical properties and achieved high accuracy [16]. Yedder used a fast convolutional neural network (FCNN) with a transfer learning approach to obtain better-reconstructed images from their limited-angle DOT and could localize and reconstruct a lesion at same time [18,19]. Deep learning has also been used to localize and classify breast lesions, using DOT measurements [20–22]. Feng trained a neural network model with only simulation data and directly used it for differentiating malignant from benign lesions, with good results [20]. Our group previously developed an auto-encoder machine learning model with physical constraints (ML-PC) to recover the absorption coefficients of breast lesions more accurately [23]. In this current work, we have adapted our auto-encoder model to more general and realistic situations and have re-trained the model with additional simulated data. We have also validated the proposed model on a larger set of clinical cases.
When applied to US imaging, deep learning-based classification has outperformed traditional feature-based classification approaches, and such image classification neural networks have found clinical success as a diagnostic tool [24–26]. For example, Zhang et al. [27] applied transfer learning on the Xception network to classify breast ultrasound images and achieved an accuracy of 92.86% with low-risk breast patients. Zhu et al. used a VGG-16 T model and achieved a 0.829 testing AUC on thyroid ultrasound images [28]. In another study, Zhu et al. used a modified VGG model to differentiate papillary thyroid carcinoma from benign thyroid nodules and achieved 86.43% accuracy in patients [29]. In the machine learning field, the fusion model has been applied to use multiple neural network models to achieve higher accuracy compared to the single neural network model alone [30]. Huang et al. used a multi-model fusion technique to leverage CT imaging and electronic health records and achieved an AUC of 0.947 in pulmonary embolism detection [31].
In this paper, we introduce a fusion model deep learning approach that combines two neural networks. One revised VGG-11 neural network extracts features from US images, and the other ML-PC reconstructs DOT images. We combine those two neural networks to achieve a more accurate classification than a single model approach can provide. The extracted US features and the reconstructed DOT images together achieve a superior AUC to those of either US-only classification or DOT-alone classification. To the best of our knowledge, this is the first time that a fusion model of US and DOT has been used for breast cancer diagnosis.
2. Methods
2.1 US-guided DOT system
Our US-guided frequency-domain DOT system with a hand-held probe is shown in Fig. 1 and has been applied in phantom experiments and clinical studies. The DOT system consists of four laser diodes with wavelengths of 730, 785, 808, and 830 nm, modulated at 140 MHz, and it incorporates 14 parallel photomultiplier (PMT) detectors and a commercial US system. The hand-held probe consists of nine source fibers placed on one side of the ultrasound transducer, which is located in the center of the probe, and the detectors are placed on the other side, with source-detector separations varying from 3 cm to 7.6 cm [11]. The measured reflected light is demodulated to 20kHz and acquired by a data acquisition A/D board (DAQ).
Fig. 1. US-guided DOT system and the hand-held probe. The DOT system uses four laser diodes with wavelengths of 730, 785, 808, and 830 nm, and it incorporates 14 parallel photomultipliers (PMT) detectors and a commercial US system.
2.2 Breast lesion classification by fusion DOT reconstructed images and US features
2.2.1 Breast US image classification
Transfer learning, a promising method for applying neural networks to biomedical imaging [32], uses a model pre-trained on a much larger dataset of various categories to help overcome the limitation of a typical smaller clinical data set. In this study, we used a pre-trained VGG-11 neural network, which has eight convolutional layers for feature extraction and three fully connected (FC) layers for classification [33]. It was pre-trained with the ImageNet database, which contains 1.2 million images. We evaluated the diagnostic performance of different neural network architectures, including VGG-11, VGG-16, VGG-19, Resnet, and Densenet, on the US dataset used in this study. The VGG architectures have the best performance compared with other neural network models. The VGG-11 model is chosen because it has a similar performance to VGG-16 and VGG-19, and it has fewer parameters than VGG-16 or VGG-19, meanwhile preserving the ability to extract features from US images.
We performed transfer learning with both open-source breast US images and images from our US dataset collected over the past years and modified the architecture of the VGG-11 to better suit this task. The detail of the modified VGG-11 network is shown in Fig. 2(a). In the feature extraction layers, we added a zero-padding layer and modified the last convolutional layer so that its output could be stacked with the DOT imaging features extracted from another neural network model. For classification, the number of fully connected layers was reduced to two to reduce the number of parameters needed to be trained within this neural network. Binary cross-entropy loss and the stochastic gradient descent algorithm with an initial learning rate of 0.01 and a momentum of 0.9 were used in the training stage. The learning rate decay was set to be 0.1 for every three epochs, and the number of training epochs was 15.
Fig. 2. (a) Modified VGG-11 to perform feature extraction and classification on US images alone. (b) A convolutional neural network to perform feature extraction and classification on DOT reconstructed images alone. The number of 512 hidden units in the final fully connected layer was empirically chosen as a compromise between fast computation time and good performance. (c) The workflow of the fusion classification model combines both US images and reconstructed DOT images.
To further understand the features extracted from the US images and learn where the neural network pays attention, we applied Grad-CAM [34], a technique for producing ‘visual explanations’ of the decisions that convolutional neural networks make. Grad-CAM computes the backward gradient based on the final prediction and produces a heat map that can be directly overlaid on the original input image, showing in detail both the areas the neural network are interested in, and which areas contribute more to the final prediction.
2.2.2 DOT imaging reconstruction and classification
To achieve higher quality of DOT image reconstruction, our group developed the ML-PC model [23]. To improve the performance and robustness of the ML-PC model on clinical data, which include more complex lesions, more simulation data was generated to represent realistic scenarios more closely, such as heterogeneous breast tissue, irregularly shaped targets, and off-centered targets. After simulations, phantom experimental data were used to re-train the model, the reconstructed images from the ML-PC model were used for classification.
We also compared ML-PC model with a simple total hemoglobin threshold-based classifier to evaluate the improvement of ML-PC on lesion diagnosis. The ML-PC neural network contains three convolutional layers, each convolutional layer has a convolutional kernel of 3 × 3 and followed by the batch normalization layers, and two fully connected layers are applied after the feature extraction to obtain classification. After trying different kernel sizes, we found a 3 × 3 convolutional kernel had the best performance; it is also the most commonly used size. The detail of the DOT reconstruction network is shown in Fig. 2(b). The inputs are the reconstructed images, which are in the form of a 32 × 32 × 7 matrix, and the output is the probability of the lesion being malignant. The DOT only classification model is trained on the simulation data with a learning rate of 0.001 for 60 epochs. To perform the final classification using fusion model, the extracted features from the last convolutional layer of this DOT only neural network are combined with the US features extracted from the modified VGG-11 network.
2.2.3 Fusion classification model
Inspired by the fusion model deep learning approach, which combines two or more deep learning models to enhance performance, we built a neural network to combine the US features extracted by the US classification network and the DOT reconstruction network. The workflow of the fusion classification model is shown in Fig. 2(c). To emphasize that both neural networks contribute equally, the numbers of features extracted from both the modified VGG-11 network and the DOT neural network are the same and in the form of an 8 × 8 × 64 matrix. The features are stacked together to form an 8 × 8 × 128 matrix and input into the final stage of the fusion model.
To further extract features, the combined inputs go through another two convolutional layers with 3 × 3 convolutional kernels. Two fully connected layers learn the relationship between the extracted features and the final classification. The initial weights of feature extraction layers of the DOT reconstruction network and the modified VGG-11 network were applied in the fusion model, and then the fusion model was fine-tuned with patient data with a 0.0001 learning rate for 20 epochs. A learning rate decay was implemented in the fine-tuning stage.
2.3 Data
2.3.1 DOT simulation data
DOT perturbation measurements were generated using the finite element method (FEM) and Monte Carlo simulation. We used FEM to generate data with well-rounded lesions and homogenous backgrounds and used Monte Carlo combined with the VICTRE digital breast phantom to generate data with heterogenous tissue structure [35]. COMSOL software was used to generate the FEM forward measurements. A hemisphere with was used to simulate the breast tissue, and a layer with different optical properties was added beneath the hemisphere to mimic the chest wall [23]. For MC simulation, the breast phantoms consisted of five tissue types – fat, glandular, skin, muscle (chest wall), and blood vessels. The digital phantoms were numerically compressed to 5 cm in the z-direction. The value of the optical properties assigned to each tissue type were calculated based on oxy- and deoxy-hemoglobin concentrations and absorption coefficients, as given in Table 1 in Ref. [36]. A total of 5800 simulated perturbation measurements were generated and reconstructed using the ML-PC model. Among the 5800 sets of simulated input data, 10 percent of the data was put into the testing dataset, and the rest was used in training. Simulated optical parameters are shown in Table 1. After acquiring the different single wavelength DOT simulation data, we used four single wavelength simulation data to linearly compute the total hemoglobin values, using extinction coefficients obtained from Ref. [37].
Table 1. Target lesion and background optical tissue optical properties in training and testing dataset.
2.3.2 DOT patient data
Our US-guided DOT technology was translated into patient studies [9]. The protocol for the current study was approved by the local Institutional Review Boards and was compliant with the Health Insurance Portability and Accountability Act. All patients signed an informed consent form, and all patient data used in this study were deidentified. DOT data were reconstructed from 100 patients (50 benign and 50 malignant).
2.3.3 US data
In this study, three different datasets of breast US images were used: our group’s US dataset collected using commercial linear breast probes of similar specifications over the past years, and two open-source datasets [38] and [39]. Combining the open-source data with our own US images, a total of 1627 breast US images from 256 patients were studied to evaluate the proposed network. Based on biopsy results, 169 patients had benign lesions, and 87 patients had malignant lesions.
To make the US images suitable inputs for the modified VGG-11 network, we first cropped the region of interest from the US images, focusing on the lesion areas. Each US image was resized to 224 × 224, and the images were normalized based on the mean and standard deviation of the ImageNet dataset. Then an augmentation strategy called Auto Augment [40] was applied to enhance the robustness of the dataset. After data augmentation, the dataset was separated into two parts: a testing set including all patient data to be used in the testing set of the fusion model with some randomly selected other cases, and a randomly divided training and validation set for cross-validation. Details of the training and testing dataset for all three models are shown in Table 2.
Table 2. Dataset used for training and fine-tune stages.
3. Results
3.1 Image reconstruction
DOT data collected using our US-guided DOT systems were reconstructed from all 100 patients, including 50 benign and 50 malignant cases. The total hemoglobin concentrations were computed based on the reconstructed images over all four wavelengths.
One benign and one malignant example are presented in Fig. 3. In Fig. 3(a) and (c), the co-registered US images of the benign and malignant lesions show where the lesions (marked by the blue circles) are located. The benign lesion, located around 0.8 cm beneath the tissue surface, is approximately 1.5 cm in diameter in the x-direction. The malignant lesion, located around 1.8 cm deep, is approximately 2 cm in diameter in the x-direction. The benign lesion shows a smoother boundary as compared with the malignant lesion showing an irregular boundary. The ML-PC model reconstructed the benign lesion with a maximum total hemoglobin concentration of 49.90 µM and the malignant lesion with a total concentration 114.41 µM.
Fig. 3. (a) (c) Co-registered US images acquired from a commercial US system, where the lesions are marked by blue circles. (b) Reconstructed total hemoglobin concentration distribution map of a benign lesion from the ML-PC model, where the maximum reconstructed total hemoglobin is 49.90 µM. (d) Reconstructed total hemoglobin concentration distribution map of a malignant lesion from the ML-PC model, where the maximum reconstructed total hemoglobin is 114.41 µM. For each DOT map, 7 slides from left to right and top to bottom represent spatial x-y images of 9 cm by 9 cm each from 0.5 cm depth to 3.5 cm depth with 0.5 cm spacing in depth.
3.2 Ultrasound classification and Grad-CAM
Using the modified VGG-11 network, we achieved an average AUC of 0.860 (95% CI: 0.852-0.869) for the performance of the testing dataset on classifying US images from benign and malignant breast lesions. Grad-CAM was applied to generate a heat map to better understand the focus of the VGG-11 network. Figure 4 shows one malignant and one benign example of the results after applying Grad-CAM. The heat map of the malignant lesion focuses on the shadow area beneath the lesion and the irregular boundary of the malignant lesion, in agreement with the US image characteristics of malignant lesions. The heat map of the benign lesion focuses mainly on the lesion's rounded shape and the lesion itself.
Fig. 4. Heat maps calculated using Grad-CAM for the final convolutional layer of the modified VGG-11 network for (a) a malignant lesion, classified as malignant by VGG-11, and (b) a benign lesion, classified as benign by VGG-11.
3.3 Fusion classification model
The fusion model uses the initial weights of both the network of the reconstructed DOT images and the modified VGG-11 network to extract features from the reconstructed images and the co-registered US images. The fusion model was fine-tuned by the patient data: 30 benign and 30 malignant patients were randomly selected as the training dataset in the fine-tuning stage, and the remaining 20 benign and 20 malignant patients were put into the testing dataset. We ran the code 10 times, with different training-testing splits. The ROC curves are shown in Fig. 5, where the mean training AUC is 0.945 (95% CI: 0.936-0.954) and the mean testing AUC is 0.931 (95% CI: 0.919 - 0.943). For the proposed fusion classification model, we achieved an average accuracy of 0.831, which is the fraction of correct predictions among all predictions. To emphasize sensitivity and minimize false negatives, we lowered the classification threshold to achieve a sensitivity of 0.95 and a specificity of 0.60.
Fig. 5. Ten-times averaged ROC curves in the fine-tuning stage of (a) the validation dataset and (b) the testing dataset.
To make a more explicit comparison between classification combining US images and DOT images and classification using a single modality, the same training and testing patients were used in the DOT reconstruction network and US VGG-11 networks. The same number of epochs and the same learning rate were applied in the DOT reconstruction network and the fusion model. For comparison, Fig. 6 shows the classification results of combining US images with DOT reconstructed images, using US images only, and using DOT reconstructed images only. The US model achieves only 0.860 (95% CI: 0.852-0.869). The ML-PC model of the reconstructed image achieves only 0.842 (95% CI: 0.830-0.854), which outperforms the threshold-based classification based on the total hemoglobin concentration 0.753 (95% CI: 0.728-0.789), indicating that the reconstructed images provided more information than a single total hemoglobin concentration parameter. The combination approach has the highest AUC of 0.931 (95% CI: 0.919 - 0.942) among all the approaches. Both the US-only model and DOT only model have similar performance, however, both models are synergistic when used together to improve the overall diagnostic performance.
Fig. 6. Comparative results for classification based on the fusion model (red), US images only (blue), DOT images only (black), and the hemoglobin threshold only (yellow-green).
4. Discussion and summary
In this paper, we proposed a fusion approach combining DOT and US to classify whether a breast lesion is malignant or benign. A deep learning model uses the DOT images of breast lesions and the features extracted from US images. The deep learning model, trained with simulation data and fine-tuned with clinical data, achieved an AUC of 0.931, outperforming classification using US images only, with an AUC of 0.860, and classification using DOT images only, with an AUC of 0.842. Compared with our previous study, the fusion model provides classification results comparable to those obtained using both DOT features and radiologists’ BIRADs scores (AUC 0.937) [10]. However, no BIRADS scores are needed in the fusion model approach, which is highly desirable in low resource environments. Utilizing the modified VGG-11 neural network, we could extract more information from the co-registered US images, beyond the location and lesion size information used in traditional DOT image reconstruction. The performance of the fusion approach showed that combining multiple imaging modalities in the neural network can yield better results in various tasks. Also, the proposed fusion approach can be generalized to other biomedical imaging areas involving multiple imaging modalities.
Although the proposed deep learning model performed well on the clinical data, our study still has limitations. First, the simulations included in the training set are mostly either simple homogeneous tissue and the chest wall or simple heterogeneous digital breast phantoms with only four tissue types. The proposed approach can be further improved by adding more complex simulations, such as more complex breast tissue and chest wall. Second, since US is a mature modality, it has more open source data for training and the model is expected to be more robust. DOT, a new modality, has less data available in the literature. Hence, we trained the DOT only model using simulation data and fine tuned and tested the fusion model with patient data. We do not believe this strategy degraded the fusion model’s classification performance, but we will continue to evaluate the fusion model as we are acquiring more co-registered US and DOT patient data. Finally, the proposed model needs manual preprocessing for both the perturbation measurements (deleting outliers) and the co-registered US images (cropping the region of interest). The entire processing can be further automated by incorporating an automatic outlier detection algorithm and using an US segmentation algorithm.
Funding
National Cancer Institute (R01CA228047).
Acknowledgement
The authors acknowledge the funding support from U.S. National Cancer Institute (R01CA228047). We thank James Ballard for reviewing and editing the manuscript.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
The self-collected dataset is not publicly available due to privacy reasons but can be made available from the corresponding author upon reasonable request. The public datasets are available in Refs. [38,39]. Associated code for the fusion model is available in Ref. [41].
References
1. D. A. Boas, D. H. Brooks, E. L. Miller, C. A. Dimarzio, M. Kilmer, R. J. Gaudette, and Q. Zhang, “Imaging the body with diffuse optical tomography,” IEEE Signal Process. Mag. 18(6), 57–75 (2001). [CrossRef]
2. A. T. Eggebrecht, S. L. Ferradal, A. Robichaux-Viehoever, M. S. Hassanpour, H. Dehghani, A. Z. Snyder, T. Hershey, and J. P. Culver, “Mapping distributed brain function and networks with diffuse optical tomography,” Nat. Photonics 8(6), 448–454 (2014). [CrossRef]
3. S. P. Poplack, C. A. Young, I. S. Hagemann, J. Luo, C. R. Herman, K. Wiele, S. Li, Y. Zeng, M. F. Covington, and Q. Zhu, “Prospective assessment of adjunctive ultrasound-guided diffuse optical tomography in women undergoing breast biopsy: Impact on BI-RADS assessments,” Eur. J. Radiol. 145, 110029 (2021). [CrossRef]
4. F. Collettini, J. C. Martin, F. Diekmann, E. Fallenberg, F. Engelken, S. Ponder, T. J. Kroencke, B. Hamm, and A. Poellinger, “Diagnostic performance of a near-infrared breast imaging system as adjunct to mammography versus X-ray mammography alone,” Eur. Radiol. 22(2), 350–357 (2012). [CrossRef]
5. B. J. Tromberg, B. W. Pogue, K. D. Paulsen, A. G. Yodh, D. A. Boas, and A. E. Cerussi, “Assessing the future of diffuse optical imaging technologies for breast cancer management,” Med. Phys. 35(6Part1), 2443–2451 (2008). [CrossRef]
6. D. Grosenick, H. Rinneberg, R. Cubeddu, and P. Taroni, “Review of optical breast imaging and spectroscopy,” J. Biomed. Opt. 21(9), 091311 (2016). [CrossRef]
7. Q. Fang, J. Selb, S. A. Carp, G. Boverman, E. L. Miller, D. H. Brooks, R. H. Moore, D. B. Kopans, and D. A. Boas, “Combined optical and x-ray tomosynthesis breast imaging,” Radiology 258(1), 89–97 (2011). [CrossRef]
8. J. Feng, J. Xu, S. Jiang, H. Yin, Y. Zhao, J. Gui, K. Wang, X. Lv, F. Ren, B. W. Pogue, and K. D. Paulsen, “Addition of T2-guided optical tomography improves noncontrast breast magnetic resonance imaging diagnosis,” Breast Cancer Res. 19(1), 117 (2017). [CrossRef]
9. Q. Zhu, A. Ricci, P. Hegde, M. Kane, E. Cronin, A. Merkulov, Y. Xu, B. Tavakoli, and S. Tannenbaum, “Assessment of functional differences in malignant and benign breast lesions and improvement of diagnostic accuracy by using US-guided diffuse optical tomography in conjunction with conventional US,” Radiology 280(2), 387–397 (2016). [CrossRef]
10. K. M. S. Uddin, M. Zhang, M. Anastasio, and Q. Zhu, “Optimal breast cancer diagnostic strategy using combined ultrasound and diffuse optical tomography,” Biomed. Opt. Express 11(5), 2722 (2020). [CrossRef]
11. H. Vavadi, A. Mostafa, F. Zhou, K. M. S. Uddin, M. Althobaiti, C. Xu, R. Bansal, F. Ademuyiwa, S. Poplack, and Q. Zhu, “Compact ultrasound-guided diffuse optical tomography system for breast cancer imaging,” J. Biomed. Opt. 24(2), 021203 (2018). [CrossRef]
12. M. Huang and Q. Zhu, “Dual-mesh optical tomography reconstruction method with a depth correction that uses a prioriultrasound information,” Appl. Opt. 43(8), 1654 (2004). [CrossRef]
13. J. T. Smith, M. Ochoa, D. Faulkner, G. Haskins, and X. Intes, “Deep learning in macroscopic diffuse optical imaging,” J. Biomed. Opt. 27(2), 020901 (2022). [CrossRef]
14. J. Yoo, S. Sabir, D. Heo, K. H. Kim, A. Wahab, Y. Choi, S.-I. Lee, E. Y. Chae, H. H. Kim, Y. M. Bae, Y.-W. Choi, S. Cho, and J. C. Ye, “Deep learning diffuse optical tomography,” IEEE Trans. Med. Imaging 39(4), 877–887 (2020). [CrossRef]
15. J. Feng, Q. Sun, Z. Li, Z. Sun, and K. Jia, “Back-propagation neural network-based reconstruction algorithm for diffuse optical tomography,” J. Biomed. Opt. 24(5), 051407 (2018). [CrossRef]
16. M. Zhang, S. Li, Y. Zou, and Q. Zhu, “Deep learning-based method to accurately estimate breast tissue optical properties in the presence of the chest wall,” J. Biomed. Opt. 26(10), 106004 (2021). [CrossRef]
17. S. Sabir, S. Cho, Y. Kim, R. Pua, D. Heo, K. H. Kim, Y. Choi, and S. Cho, “Convolutional neural network-based approach to estimate bulk optical properties in diffuse optical tomography,” Appl. Opt. 59(5), 1461 (2020). [CrossRef]
18. H. Ben Yedder, M. Shokoufi, B. Cardoen, F. Golnaraghi, and G. Hamarneh, “Limited-Angle diffuse optical tomography image reconstruction using deep learning,” in Proceedings of Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference. (2019), pp. 66–74.
19. H. Ben Yedder, B. Cardoen, M. Shokoufi, F. Golnaraghi, and G. Hamarneh, “Multitask deep learning reconstruction and localization of lesions in limited angle diffuse optical tomography,” IEEE Trans. Med. Imaging 41(3), 515–530 (2022). [CrossRef]
20. J. Feng, W. Zhang, Z. Li, K. Jia, S. Jiang, H. Dehghani, B. W. Pogue, and K. D. Paulsen, “Deep-learning based image reconstruction for MRI-guided near-infrared spectral tomography,” Optica 9(3), 264 (2022). [CrossRef]
21. Q. Xu, X. Wang, and H. Jiang, “Convolutional neural network for breast cancer diagnosis using diffuse optical tomography,” Vis. Comput. Ind. Biomed. Art 2(1), 1 (2019). [CrossRef]
22. N. Murad, M.-C. Pan, and Y.-F. Hsu, “Reconstruction and localization of tumors in breast optical imaging via convolution neural network based on batch normalization layers,” IEEE Access 10, 57850–57864 (2022). [CrossRef]
23. Y. Zou, Y. Zeng, S. Li, and Q. Zhu, “Machine learning model with physical constraints for diffuse optical tomography,” Biomed. Opt. Express 12(9), 5720 (2021). [CrossRef]
24. S. S. Yadav and S. M. Jadhav, “Deep convolutional neural network based medical image classification for disease diagnosis,” J. Big Data 6(1), 113 (2019). [CrossRef]
25. L. Cai, J. Gao, and D. Zhao, “A review of the application of deep learning in medical image classification and segmentation,” Ann. Transl. Med. 8(11), 713 (2020). [CrossRef]
26. Z. Lai and H. Deng, “Medical image classification based on deep features extracted by deep model and statistic feature fusion with multilayer perceptron,” Comput. Intell. Neurosci. 2018, 1–13 (2018). [CrossRef]
27. X. Zhang, H. Li, C. Wang, W. Cheng, Y. Zhu, D. Li, H. Jing, S. Li, J. Hou, J. Li, Y. Li, Y. Zhao, H. Mo, and D. Pang, “Evaluating the accuracy of breast cancer and molecular subtype diagnosis by ultrasound image deep learning model,” Front. Oncol. 11, 623506 (2021). [CrossRef]
28. Y.-C. Zhu, P.-F. Jin, J. Bao, Q. Jiang, and X. Wang, “Thyroid ultrasound image classification using a convolutional neural network,” Ann. Transl. Med. 9(20), 1526 (2021). [CrossRef]
29. Y.-C. Zhu, A. AlZoubi, S. Jassim, Q. Jiang, Y. Zhang, Y.-B. Wang, X.-D. Ye, and H. Du, “A generic deep learning framework to classify thyroid and breast lesions in ultrasound images,” Ultrasonics 110, 106300 (2021). [CrossRef]
30. S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren, “Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines,” NPJ Digit. Med. 3(1), 136 (2020). [CrossRef]
31. S.-C. Huang, A. Pareek, R. Zamanian, I. Banerjee, and M. P. Lungren, “Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection,” Sci. Rep. 10(1), 22147 (2020). [CrossRef]
32. Z. Zhou, J. Shin, L. Zhang, S. Gurudu, M. Gotway, and J. Liang, “Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2017), pp. 4761–4772.
33. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv, arXiv:1409.1556 (2014). [CrossRef]
34. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: visual explanations from deep networks via gradient-based localization,” in 2017 IEEE International Conference on Computer Vision (ICCV) (IEEE, 2017), pp. 618–626.
35. A. Badano, C. G. Graff, A. Badal, D. Sharma, R. Zeng, F. W. Samuelson, S. J. Glick, and K. J. Myers, “Evaluation of Digital Breast Tomosynthesis as Replacement of Full-Field Digital Mammography Using an In Silico Imaging Trial,” JAMA Netw. Open 1(7), e185474 (2018). [CrossRef]
36. S. Li, M. Zhang, and Q. Zhu, “Ultrasound segmentation-guided edge artifact reduction in diffuse optical tomography using connected component analysis,” Biomed. Opt. Express 12(8), 5320 (2021). [CrossRef]
37. M Cope, “The development of a near infrared spectroscopy system and its application for non invasive monitoring of cerebral blood and tissue oxygenation in the newborn infants,” University of London, University College London, 1991.
38. W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, “Dataset of breast ultrasound images,” Data Brief 28, 104863 (2020). [CrossRef]
39. P. S. Rodrigues, “Breast ultrasound image,” Mendeley Data, V1, 2017, https://doi.10.17632/wmy84gzngw.1.
40. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “AutoAugment: Learning Augmentation Policies from Data,” arXiv, arXiv:1805.09501 (2018), [CrossRef] .
41. M. Zhang, M. Xue, S. Li, Y. Zou, and Q. Zhu, “Code for fusion model,” Github, 2023, https://github.com/OpticalUltrasoundImaging/Fusion_model
