Extracting postmarketing adverse events from safety reports in the vaccine adverse event reporting system (VAERS) using deep learning

Objective: Automated analysis of vaccine postmarketing surveillance narrative reports is important to understand the progression of rare but severe vaccine adverse events (AEs). This study implemented and evaluated state-of-the-art deep learning algorithms for named entity recognition to extract nervous system disorder-related events from vaccine safety reports. Materials and Methods: We collected Guillain-Barre syndrome (GBS) related influenza vaccine safety reports from the Vaccine Adverse Event Reporting System (VAERS) from 1990 to 2016. VAERS reports were selected and manually annotated with major entities related to nervous system disorders, including, investigation, nervous AE, other AE, procedure, social circumstance, and temporal expression. A variety of conventional machine learning and deep learning algorithms were then evaluated for the extraction of the above entities. We further pretrained domain-specific BERT (Bidirectional Encoder Representations from Transformers) using VAERS reports (VAERS BERT) and compared its performance with existing models. Results and Conclusions: Ninety-one VAERS reports were annotated, resulting in 2512 entities. The corpus was made publicly available to promote community efforts on vaccine AEs identification. Deep learning-based methods (eg, bi-long short-term memory and BERT models) outperformed conventional machine learning-based methods (ie, conditional random fields with extensive features). The BioBERT large model achieved the highest exact match F-1 scores on nervous AE, procedure, social circumstance, and temporal expression; while VAERS BERT large models achieved the highest exact match F-1 scores on investigation and other AE. An ensemble of these 2 models achieved the highest exact match microaveraged F-1 score at 0.6802 and the second highest lenient match microaveraged F-1 score at 0.8078 among peer models.

Automated summary

This study implemented and evaluated state-of-the-art deep learning algorithms for named entity recognition to extract nervous system disorder-related events from vaccine safety reports. Results showed that deep learning-based methods outperformed conventional machine learning-based methods, with BioBERT and VAERS BERT models achieving the highest exact match F-1 scores for different entities. Additionally, an ensemble of these models achieved the highest exact match microaveraged F-1 score among peer models, demonstrating significant performance advantages.