Multi-Attention and Incorporating Background Information Model for Chest X-Ray Image Report Generation

Chest X-ray images are widely used in clinical practice such as diagnosis and treatment. The automatic radiology report generation system can effectively reduce the rate of misdiagnosis and missed diagnosis. Previous studies were focused on the long text generation problem of image paragraph, ignoring the characteristics of the image and the auxiliary role of patient background information for diagnosis. In this paper, we propose a new hierarchical model with multi-attention considering the background information. The multi-attention mechanism can focus on the image's channel and spatial information simultaneously, and map it to the sentence topic. The patient's background information will be encoded by the neural network first, then it will be aggregated into a vector representation by a multi-layer perception and added to the pretrained vanilla word embedding, which finally forms a new word embedding after fusion. Our experimental results demonstrated that the model outperforms all baselines, achieving the state-of-the-art performance in terms of accuracy.