Multi-Label Learning With Visual-Semantic Embedded Knowledge Graph for Diagnosis of Radiology Imaging

A significant task of automatic diagnosis for radiology imaging, especially for chest X-rays, is to identify disease types, which can be viewed as a multi-label learning problem. Prior state-of-the-art approaches adopted the graph convolutional network to model the correlations among disease labels. However, the utilization of medical reports paired with radiology images is neglected in such approaches. Hence, at least two novel improvements are proposed in this paper. First, disease label embeddings are pre-trained on the total radiology reports, and these semantic features along with encoded X-ray features are fused in a transformer encoder for graph initialization. Second, to expand the representation ability of the graph, extra medical terms from radiology reports are mined and added to the graph model as auxiliary nodes without changing the size of the output space. Experiments conducted on two public chest-X-ray datasets demonstrate the outstanding performance over compared models and the advantages of the proposed improvements.

Automated summary

The paper introduces two novel improvements in automatic diagnosis for radiology imaging: pre-training disease label embeddings on total radiology reports and fusing semantic features with X-ray features in a transformer encoder for graph initialization, as well as mining extra medical terms from radiology reports and adding them as auxiliary nodes to expand the graph's representation ability without altering the output space size. Experiments on two public chest-X-ray datasets demonstrate the exceptional performance compared to existing models and the benefits of the proposed enhancements.