A Black-box Approach Evaluation on Conversational Agent using Loebner Prize Competition Datasets

Abstract

Conversational Agents, or more commonly known as chatbots have been well-accepted in most enterprises. Chatbots have been widely used in major enterprises to perform the main purpose of customer service in delivering routine and frequently asked questions on behalf of humans. Bearing the goal to create believable conversational agents in providing accurate, reliable and up-to-date information, researchers have practiced natural language processing, machine learning, and deep learning in chatbot creation. In order to create a chatbot which is indistinguishable from human, many researchers have attempted to create human-like chatbots by neglecting the necessity to evaluate the general knowledge of the chatbot before proceeding into the creation of domain-specific knowledge. This paper presents the black-box approach to evaluate the response quality of a developed conversational agent named Tarie through datasets obtained from Loebner Prize Competition. The black-box approach is therefore deemed necessary to be used as the evaluation approach to evaluate the response quality of the conversational agent. The intelligence of the chatbot denotes the correct answers given to the user’s queries. The results obtained via the black-box approach is further deduced to indicate whether the chatbot is ready for the domain-specific knowledge creation or to further improve its current general knowledge. This paper presents the black-box approach to evaluate the response quality of a proposed conversational agent named Tarie through datasets obtained from Loebner Prize Competition.