ReportAGE: Automatically extracting the exact age of Twitter users based on self-reports in tweets

Advancing the utility of social media data for research applications requires methods for automatically detecting demographic information about social media study populations, including users' age. The objective of this study was to develop and evaluate a method that automatically identifies the exact age of users based on self-reports in their tweets. Our end-to-end automatic natural language processing (NLP) pipeline, ReportAGE, includes query patterns to retrieve tweets that potentially mention an age, a classifier to distinguish retrieved tweets that self-report the user's exact age (age tweets) and those that do not (no age tweets), and rule-based extraction to identify the age. To develop and evaluate ReportAGE, we manually annotated 11,000 tweets that matched the query patterns. Based on 1000 tweets that were annotated by all five annotators, inter-annotator agreement (Fleiss' kappa) was 0.80 for distinguishing age and no age tweets, and 0.95 for identifying the exact age among the age tweets on which the annotators agreed. A deep neural network classifier, based on a RoBERTa-Large pretrained transformer model, achieved the highest F-1-score of 0.914 (precision = 0.905, recall = 0.942) for the age class. When the age extraction was evaluated using the classifier's predictions, it achieved an F-1-score of 0.855 (precision = 0.805, recall = 0.914) for the age class. When it was evaluated directly on the held-out test set, it achieved an F-1-score of 0.931 (precision = 0.873, recall = 0.998) for the age class. We deployed ReportAGE on a collection of more than 1.2 billion tweets, posted by 245,927 users, and predicted ages for 132,637 (54%) of them. Scaling the detection of exact age to this large number of users can advance the utility of social media data for research applications that do not align with the predefined age groupings of extant binary or multi-class classification approaches.

Automated summary

This study developed and evaluated a method for automatically identifying the exact age of users based on self-reports in their tweets. They achieved high accuracy in age identification using natural language processing and a deep neural network classifier, and successfully applied the method to a large Twitter dataset.