MF-AV-Net: an open-source deep learning network with multimodal fusion options for artery-vein segmentation in OCT angiography

This study is to demonstrate the effect of multimodal fusion on the performance of deep learning artery-vein (AV) segmentation in optical coherence tomography (OCT) and OCT angiography (OCTA); and to explore OCT/OCTA characteristics used in the deep learning AV segmentation. We quantitatively evaluated multimodal architectures with early and late OCT-OCTA fusions, compared to the unimodal architectures with OCT-only and OCTA-only inputs. The OCTA-only architecture, early OCT-OCTA fusion architecture, and late OCT-OCTA fusion architecture yielded competitive performances. For the 6 mmx6 mm and 3 mmx3 mm datasets, the late fusion architecture achieved an overall accuracy of 96.02% and 94.00%, slightly better than the OCTA-only architecture which achieved an overall accuracy of 95.76% and 93.79%. 6 mmx6 mm OCTA images show AV information at pre-capillary level structure, while 3 mmx3 mm OCTA images reveal AV information at capillary level detail. In order to interpret the deep learning performance, saliency maps were produced to identify OCT/OCTA image characteristics for AV segmentation. Comparative OCT and OCTA saliency maps support the capillary-free zone as one of the possible features for AV segmentation in OCTA. The deep learning network MF-AV-Net used in this study is available on GitHub for open access.(C) 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement

Automated summary

This study demonstrates the impact of multimodal fusion on deep learning artery-vein segmentation in optical coherence tomography and OCT angiography, and explores the characteristics used in this segmentation. The performance of multimodal architectures with OCT-OCTA fusion is compared to unimodal architectures with only OCT or OCTA inputs. The results show that both early and late fusion architectures perform competitively. Saliency maps are used to identify the characteristics in OCT and OCTA images for artery-vein segmentation.