An Experimental Comparison of the Usability of Rule-based and Natural Language Processing-based Chatbots

Abstract

Service organizations increasingly adopt data-based intelligent engines called chatbots in support of the interaction between customers and the companies. Two different types of chatbots have been suggested and introduced by companies leading the adoption of this emerging technology: rule-based chatbots and natural language processing-based chatbots. While the differences between these two types of technologies look relatively clear, the organizational and practical impacts of the differences have not been systematically explored. This study performed an experiment to compare the use of the two different types of chatbots used in practice by two comparable organizations. These two types of actual chatbots were used by Korean on-line shopping malls with similar business models (mobile shopping), length of history, size and reputation. The comparison was made based on such dimensions as usability, searchability, reliability and attractiveness. Contraty to conventional expectation that the superiority in technology will produce superior usability, the results show mixed superiority. The discussion on the reasons is presented.