Task-agnostic representation learning of multimodal twitter data for downstream applications

Twitter is a frequent target for machine learning research and applications. Many problems, such as sentiment analysis, image tagging, and location prediction have been studied on Twitter data. Much of the prior work that addresses these problems within the context of Twitter focuses on a subset of the types of data available, e.g. only text, or text and image. However, a tweet can have several additional components, such as the location and the author, that can also provide useful information for machine learning tasks. In this work, we explore the problem of jointly modeling several tweet components in a common embedding space via task-agnostic representation learning, which can then be used to tackle various machine learning applications. To address this problem, we propose a deep neural network framework that combines text, image, and graph representations to learn joint embeddings for 5 tweet components: body, hashtags, images, user, and location. In our experiments, we use a large dataset of tweets to learn a joint embedding model and use it in multiple tasks to evaluate its performance vs. state-of-the-art baselines specific to each task. Our results show that our proposed generic method has similar or superior performance to specialized application-specific approaches, including accuracy of 52.43% vs. 48.88% for location prediction and recall of up to 15.93% vs. 12.12% for hashtag recommendation.

Automated summary

Twitter is a frequently used subject in machine learning research and applications, with problems like sentiment analysis, image tagging, and location prediction being studied on Twitter data. Most prior work in this area focuses on a subset of the available data, such as text or text and image. However, a tweet can have additional components like location and author, which can provide useful information for machine learning tasks. In this study, the authors propose a deep neural network framework that combines text, image, and graph representations to learn joint embeddings for different tweet components. Experimental results show that this approach has comparable or superior performance compared to specialized application-specific approaches.