Multimodal Topic Modeling by Exploring Characteristics of Short Text Social Media

Millions of people post images and texts to express their feelings and point of views on social media everyday, especially on the short text social media such as Twitter or Weibo. As the images can provide important supplementary information for the text, many multimodal topic models have been developed to mine the topics from the multimodal social media content. We summarize three fundamental characteristics of the short text multimodal social media. The first is that the text of a short social media document generally belong to only one topic. The second is that the attached images can be relevant to multiple topics due to the rich information expressed in the images. The last is that although in most cases, text and images in social media posts are relevant, it should be noted that in a small number of cases, text and pictures are not relevant. However, most of the current multimodal topic models fail to model the these characteristics, and thus may produce low-quality topics. Based on these characteristics, we propose an unsupervised multimodal topic model SMMTM to model the short text multimodal social media documents. In the SMMTM model, only one topic is sampled for the the text while an image can belong to different topics. The correlation of the topics between the text and the images in a document are also formulated in an appropriate way. The experiments on three short text social media datasets with four evaluation metrics show the advantages of our model over the existing models.

Automated summary

Millions of people express their feelings and views on social media through images and texts, especially on short text platforms like Twitter or Weibo. However, existing multimodal topic models often fail to capture the characteristics of short text multimodal social media, leading to low-quality topics. To address this, we propose an unsupervised multimodal topic model (SMMTM) that can effectively model the relationships between text and images in social media posts. Experimental results on three social media datasets demonstrate the superiority of our model over existing ones.