Open-Ended Versus Closed-Ended Responses: A Comparison Study Using Topic Modeling and Factor Analysis

For practical reasons, surveys that aim for a large number of respondents tend to restrict themselves to closed-ended responses. Despite potentially bringing richer insights, the use of open-ended questions poses great challenges in terms of extracting useful information while significantly increasing the analysis time. Nevertheless, automatic text analysis techniques speed up the analysis of open-ended responses. In this research, we explore the potential to use techniques in topic modelling [Latent Dirichlet Allocation (LDA) and Supervised LDA (sLDA)] to extract information from open-ended responses. This is compared to the information obtained from closed-ended responses, accomplished using a questionnaire that measures the intention to use shared autonomous vehicles (SAVs). Two versions of the questionnaire- Ver_OE and Ver_Lk were used, with open-ended and Likert scales measuring the same attitudes in the alternative versions. Factors were extracted for closed-ended questions. For questions common to both versions of the questionnaire, respondents answering Ver_OE had a higher positive attitude towards autonomous vehicles. These attitudinal questions were placed after the open-ended questions. When evaluating the performance of the models that predict the intention to use SAVs, models estimated using Ver_OE performed better. This increased further with the inclusion of the information extracted from the open-ended responses using both, the unsupervised (LDA) and supervised (sLDA) methods. No improvement was observed in the model for Ver_Lk. These indicate the potential for the use of open-ended questions to measure attitudes and topic modelling to extract information from these responses.

Automated summary

The research explores the use of topic modeling techniques to extract information from open-ended responses, compared to closed-ended responses, finding that open-ended responses yield better information extraction results, especially when combined with both unsupervised and supervised methods. Open-ended questions can be used to measure attitudes, while topic modeling can be used to extract information.