Between Clicks and Satisfaction: Study on Multi-Phase User Preferences and Satisfaction for Online News Reading

Click signal has been widely used for designing and evaluating interactive information systems, which is taken as the indicator of user preference. However, click signal does not capture post-click user experience. Very commonly, the user first clicked an item and then found it is not what he wanted after reading its content, which shows there is a gap between user click and user actual preference. Previous studies on web search have incorporated other user behaviors, such as dwell time, to reduce the gap. Unfortunately, for other scenarios such as recommendation and online news reading, there still lacks a thorough understanding of the relationship between click and user preference, and the corresponding reasons which are the focus of this work. Based on an in-depth laboratory user study of online news reading scenario in the mobile environment, we show that click signal does not align with user preference. Besides, we find that user preference changes frequently, hence preferences in three phases are proposed: Before-Read Preference, After-Read Preference and Post-task Preference. In addition, the statistic analysis shows that the changes are highly related to news quality and the context of user interactions. Meanwhile, many other user behaviors, like viewport time, dwell time, and read speed, are found reflecting user preference in different phases. Furthermore, with the help of various kinds of user behaviors, news quality, and interaction context, we build an effective model to predict whether the user actually likes the clicked news. Finally, we replace binary click signals of traditional click-based evaluation metrics, like Click-Through Rate, with the predicted item-level preference, and significant improvements are achieved in estimating the user's list-level satisfaction. Our work sheds light on the understanding of user click behaviors and provides a method for better estimating user interest and satisfaction. The proposed model could also be helpful to various recommendation tasks in mobile scenarios.