What are the limitations of the currently used quantitative metrics? e.g. why are they not 'good'?
The metrics correlate very weakly with human judgements, word-overlap metrics require too many ground-truth reposnses and embedding-based metrics are insufficiently complex for modeling sentence-level compositionality in dialogue
Conversational agents have become ubiquitous, ranging from goal-oriented systems for helping with reservations to chit-chat models found in modern virtual assistants. In this survey paper, we explore this fascinating field. We look at some of the pioneering work that defined the field and gradually move to the current state-of-the-art models. We look at statistical, neural, generative adversarial network based and reinforcement learning based approaches and how they evolved. Along the way we discuss various challenges that the field faces, lack of context in utterances, not having a good quantitative metric to compare models, lack of trust in agents because they do not have a consistent persona etc. We structure this paper in a way that answers these pertinent questions and discusses competing approaches to solve them.
One of the earliest goals of Artificial Intelligence (AI) has been to build machines that can converse with us. Whether in early AI literature or the current popular culture, conversational agents have captured our imagination like no other technology has. In-fact the ultimate test of whether true artificial intelligence has been achieved, the Turing test BIBREF0 proposed by Alan Turing the father of artificial intelligence in 1950, revolves around the concept of a good conversational agent. The test is deemed to have been passed if a conversational agent is able to fool human judges into believing that it is in fact a human being. Starting with pattern matching programs like ELIZA developed at MIT in 1964 to the current commercial conversational agents and personal assistants (Siri, Allo, Alexa, Cortana et al) that all of us carry in our pockets, conversational agents have come a long way. In this paper we look at this incredible journey. We start by looking at early rule-based methods which consisted of hand engineered features, most of which were domain specific. However, in our view, the advent of neural networks that were capable of capturing long term dependencies in text and the creation of the sequence to sequence learning model BIBREF1 that was capable of handling utterances of varying length is what truly revolutionized the field. Since the sequence to sequence model was first used to build a neural conversational agent BIBREF2 in 2016 the field has exploded. With a multitude of new approaches being proposed in the last two years which significantly impact the quality of these conversational agents, we skew our paper towards the post 2016 era. Indeed one of the key features of this paper is that it surveys the exciting new developments in the domain of conversational agents. Dialogue systems, also known as interactive conversational agents, virtual agents and sometimes chatterbots, are used in a wide set of applications ranging from technical support services to language learning tools and entertainment. Dialogue systems can be divided into goal-driven systems, such as technical support services, booking systems, and querying systems. On the other hand we have non-goal-driven systems which are also referred to as chit-chat models. There is no explicit purpose for interacting with these agents other than entertainment. Compared to goal oriented dialog systems where the universe is limited to an application, building open-ended chit-chat models is more challenging. Non-goal oriented agents are a good indication of the state of the art of artificial intelligence according to the Turing test. With no grounding in common sense and no sense of context these agents have to fall back on canned responses and resort to internet searches now. But as we discuss in section SECREF5 , new techniques are emerging to provide this much needed context to these agents. The recent successes in the domain of Reinforcement Learning (RL) has also opened new avenues of applications in the conversational agent setting. We explore some of these approaches in section SECREF6  Another feature that has been traditionally lacking in conversation agents is a personality. O Vinayal et al BIBREF2 hypothesis that not having a consistent personality is one of the main reasons that is stopping us from passing the turing test. Conversational agents also lack emotional consistency in their responses. These features are vital if we want humans to trust conversational agents. In section SECREF7 we discuss state of the art approaches to overcome these problems. Despite such huge advancements in the field, the way these models are evaluated is something that needs to be dramatically altered. Currently there exists no perfect quantitative method to compare two conversational agents. The field has to rely on qualitative measures or measures like BLeU and perplexity borrowed from machine translation. In section SECREF8 we discuss this problem in detail.