Modality alignment contrastive learning for severity assessment of COVID-19 from lung ultrasound and clinical information

The outbreak of COVID-19 around the world has caused great pressure to the health care system, and many effort s have been devoted to artificial intelligence (AI)-based analysis of CT and chest X-ray images to help alleviate the shortage of radiologists and improve the diagnosis efficiency. However, only a few works focus on AI-based lung ultrasound (LUS) analysis in spite of its significant role in COVID-19. In this work, we aim to propose a novel method for severity assessment of COVID-19 patients from LUS and clinical information. Great challenges exist regarding the heterogeneous data, multi-modality infor-mation, and highly nonlinear mapping. To overcome these challenges, we first propose a dual-level su-pervised multiple instance learning module (DSA-MIL) to effectively combine the zone-level represen-tations into patient-level representations. Then a novel modality alignment contrastive learning module (MA-CLR) is presented to combine representations of the two modalities, LUS and clinical information, by matching the two spaces while keeping the discriminative features. To train the nonlinear mapping, a staged representation transfer (SRT) strategy is introduced to maximumly leverage the semantic and discriminative information from the training data. We trained the model with LUS data of 233 patients, and validated it with 80 patients. Our method can effectively combine the two modalities and achieve accuracy of 75.0% for 4-level patient severity assessment, and 87.5% for the binary severe/non-severe identification. Besides, our method also provides interpretation of the severity assessment by grading each of the lung zone (with accuracy of 85.28%) and identifying the pathological patterns of each lung zone. Our method has a great potential in real clinical practice for COVID-19 patients, especially for pregnant women and children, in aspects of progress monitoring, prognosis stratification, and patient management. (c) 2021 Elsevier B.V. All rights reserved.

Automated summary

This study aims to propose a novel method for severity assessment of COVID-19 patients from lung ultrasound (LUS) and clinical information, overcoming challenges of heterogeneous data and multi-modality information. By introducing dual-level supervised multiple instance learning module and modality alignment contrastive learning module, the method effectively combines zone-level representations and representations of LUS and clinical information. The model achieves high accuracy in patient severity assessment and provides interpretation of lung zone grading and pathological patterns, showing great potential in clinical practice.