A Survey on Evaluation Methods for Chatbots

Nowadays chatbots have been widely adopted in many industries to automatically answer users' questions and requests via chat interfaces. While it has become much easier to develop a chatbot system, the system itself is a complex system in nature. It is a challenge to evaluate and compare various chatbot systems in terms of effectiveness, efficiency, goal achievability, and the ability to satisfy users. This paper presents a survey, starting from literature review, chatbot architecture, evaluation methods/criteria, and comparison of evaluation methods. Focused on the three subprocesses in the chatbot architecture: text processing, semantic understanding, and response generation. Moreover, the survey is conducted with classification of chatbot evaluation methods and their analysis according to chatbot types and three main evaluation schemes; content evaluation, user satisfaction, and chat function.