How big is the difference in performance between proposed model and baselines?
Metric difference between Aloha and best baseline score:
Hits@1/20: +0.061 (0.3642 vs 0.3032)
MRR: +0.0572(0.5114 vs 0.4542)
F1: -0.0484 (0.3901 vs 0.4385)
BLEU: +0.0474 (0.2867 vs 0.2393)
For conversational AI and virtual assistants to communicate with humans in a realistic way, they must exhibit human characteristics such as expression of emotion and personality. Current attempts toward constructing human-like dialogue agents have presented significant difficulties. We propose Human Level Attributes (HLAs) based on tropes as the basis of a method for learning dialogue agents that can imitate the personalities of fictional characters. Tropes are characteristics of fictional personalities that are observed recurrently and determined by viewers' impressions. By combining detailed HLA data with dialogue data for specific characters, we present a dataset that models character profiles and gives dialogue agents the ability to learn characters' language styles through their HLAs. We then introduce a three-component system, ALOHA (which stands for Artificial Learning On Human Attributes), that combines character space mapping, character community detection, and language style retrieval to build a character (or personality) specific language model. Our preliminary experiments demonstrate that ALOHA, combined with our proposed dataset, can outperform baseline models at identifying correct dialogue responses of any chosen target character, and is stable regardless of the character's identity, genre of the show, and context of the dialogue.
Attempts toward constructing human-like dialogue agents have met significant difficulties, such as maintaining conversation consistency BIBREF0. This is largely due to inabilities of dialogue agents to engage the user emotionally because of an inconsistent personality BIBREF1. Many agents use personality models that attempt to map personality attributes into lower dimensional spaces (e.g. the Big Five BIBREF2). However, these represent human personality at a very high-level and lack depth. They prohibit the ability to link specific and detailed personality traits to characters, and to construct large datasets where dialogue is traceable back to these traits. For this reason, we propose Human Level Attributes (HLAs), which we define as characteristics of fictional characters representative of their profile and identity. We base HLAs on tropes collected from TV Tropes BIBREF3, which are determined by viewers' impressions of the characters. See Figure FIGREF1 for an example. Based on the hypothesis that profile and identity contribute effectively to language style BIBREF4, we propose that modeling conversation with HLAs is a means for constructing a dialogue agent with stable human-like characteristics. By collecting dialogues from a variety of characters along with this HLA information, we present a novel labelling of this dialogue data where it is traceable back to both its context and associated human-like qualities. We also propose a system called ALOHA (Artificial Learning On Human Attributes) as a novel method of incorporating HLAs into dialogue agents. ALOHA maps characters to a latent space based on their HLAs, determines which are most similar in profile and identity, and recovers language styles of specific characters. We test the performance of ALOHA in character language style recovery against four baselines, demonstrating outperformance and system stability. We also run a human evaluation supporting our results. Our major contributions are: (1) We propose HLAs as personality aspects of fictional characters from the audience's perspective based on tropes; (2) We provide a large dialogue dataset traceable back to both its context and associated human-like attributes; (3) We propose a system called ALOHA that is able to recommend responses linked to specific characters. We demonstrate that ALOHA, combined with the proposed dataset, outperforms baselines. ALOHA also shows stable performance regardless of the character's identity, genre of the show, and context of the dialogue. We plan to release all of ALOHA's data and code.