We teach goal-driven agents to interactively act and speak in situated environments by training on generated curriculums. Our agents operate in LIGHT (Urbanek et al., 2019)a large-scale crowd-sourced fantasy text adventure game wherein an agent perceives and interacts with the world through textual natural language. Goals in this environment take the form of character-based quests, consisting of personas and motivations. We augment LIGHT by learning to procedurally generate additional novel textual worlds and quests to create a curriculum of steadily increasing difficulty for training agents to achieve such goals. In particular, we measure curriculum difficulty in terms of the rarity of the quest in the original training distributionan easier environment is one that is more likely to have been found in the unaugmented dataset. An ablation study shows that this method of learning from the tail of a distribution results in significantly higher generalization abilities as measured by zero-shot performance on never-before-seen quests. A key hypothesis in the pursuit towards creating goal-driven natural language-based agents posits that interactivity and environment grounding is critical for effective language learning (Barsalou, 2008; Bisk et al., 2020; Ammanabrolu and Riedl, 2021). Text games provide a platform on which to interactively train agents that can both act and speak in a situated mannerproducing language that is both goal-driven and contextually relevant. Agents in text games operateperceiving, acting in, and speaking to others in a worldentirely using textual natural language. These games are structured generally as sequential decision making problems in the form of puzzles or quests that must be completed to advance in the game. crowdsourced fantasy text-adventure game, consisting of rich textual worldslocations, objects, and characters with personas, and questsmotivations for each character. To complete these quests, an agent must: (1) maintain character via its persona; and (2) reason in a partially observable world about potential actions and utterances based on incomplete descriptions of the locations, objects, and other characters. This requires several human like competencies such as commonsense reasoning, dynamic natural language understanding, and operating in combinatorially sized language-based state-action spaces. Although recent work has provided evidence showing that interactive language learning via reinforcement learning (RL) in text games can be significantly more sample efficient than static supervised learning (Ammanabrolu et al., 8099 2021) when creating goal-driven natural language agents, their ability to robustly generalize to novel scenarios is limited. In sequential decision making problems in particular, this generalization gap is the result of an agent simply memorizing trajectories, e.g. the sequence of actions and dialogues required to finish a game, and thus being unable to react in novel scenariosi.e. the agent learns from the head the training data and simply memorizes the long tail. One way of decreasing this generalization gap is by training agents on procedurally generated environmentswherein the agent learns a family of parametrized tasks with a significantly larger state-action spaces than singular environments, thus effectively making the memorization of trajectories impossible (Justesen et al., 2018; Cobbe et al., 2020). Drawing inspiration from all of these ideas, we create a method that learns to create a training curriculum of increasingly more difficult novel procedurally generated environments. Our contributions are threefold: (1) We present a method of parametrizing and generating a curriculum of environments in text games; (2) We show how to effectively train reinforcement learning agents on this curriculum; and (3) Provide an experimental study showing that our method enables significantly better generalization than those training on singular environments. This section describes our procedural generation pipeline as seen in Figure 2, starting with world and quest generation, followed by aligning both of them. There are two main kinds of models that we use for the different modules in this pipeline: retrieval and generative. The LIGHT Questing Environment. The LIGHT game environment (Urbanek et al., 2019) 1 is a multi-user fantasy text-adventure game consisting of a rich, diverse set of 1775 characters, 663 locations, and 3462 objects. Characters are able to perform templated actions to interact with both objects and characters, and can speak to other characters through free form text dialogues. Actions in text games generally consist of verb phrases (VP) followed optionally by prepositional phrases (VP PP). For example, get OBJ, put OBJ, give OBJ to CHAR , etc.. These actions change the state of the world which is expressed through text descriptions. Quests in LIGHT (Ammanabrolu et al., 2021) take the form of a short motivation and goal action that is required reach the world state required to finish the game. For example, if the short motivation is Your motivation is to acquire a sword , then the corresponding goal state would be for the character to have a sword in their inventory and goal action would be get sword . This environment also contains a set of human expert demonstration of people speaking and acting in character while playing the quests mentioned above. Further details are found in Appendix A.1. World Retrieval. The first step of the pipeline involves choosing an initial character who will perform the quest. For this, we uniformly randomly sample from the set of characters found in the LIGHT-Quest training set. The corresponding character information includes a name and a description of the character's persona. Given this character information, we further retrieve the location where the character is most likely to be found. Retrieval models are trained to return the most highly correlated output for a given input in the dataset. For example, a retrieval model can be asked to return the most likely character that can be found at a particular location. These models compare a human annotated gold standard label with negative candidates drawn from the dataset. The negative candidates provide noise that the model must filter out in order to learn representations that let it best predict the gold label. These models are trained via a ranking loss that maximizes the scores of the gold label while simultaneously minimizing negative candidate score. At test time, the highest ranked candidate based on the score is selected as the model prediction. Specifically, we use a retrieval-based ranker model that checks for similarity of StarSpace (Wu et al., 2018) embeddings. Our choice of model is influenced by Fan et al. (2019) who report state-of-the-art retrieval performance for locations in LIGHT using this model. The overall ranker model first trains a randomly initialized StarSpace embedding model that is designed to correlate characters with the locations they are found in. It learns a single bag-of-words embedding that takes into account all the individual words contained within the inputencoding character and location information as well as the previously mentioned negative 8100 Select Initial Character: Dragon -I am a dragon living in the mountains. retrieval candidates. The rest of the training is similar to other retrieval models described earlier. The retrieved location information consists of a location name as well as a description of the location. Quest Generation. The quest is now generated using the existing character and location information. The generation-based models used in this pipeline are trained to return the most likely output sequence given an input sequence. Given a target sequence Y = { y 1 , ..., y M } and some input context vector via the encoders X . These models use autoregressive decoding techniques that factor the distribution over the target sequence into a chain of conditional probabilities with a causal left to right structure as P ( Y | X ;  ) = (cid:81) M +1 i =1 p ( y i | y 0: i  1 , X ;  ) where  represents the current network parameters. At test time, a special start-of-sequence token is provided to the model which then proceeds to decode the rest of the output sequence using beam search. We train two BART (Lewis et al., 2020) models that encodes input information via a bidirectional transformer encoder and decodes autoregressively: the first takes as input character and location information and produces a short motivation (Sec-tion 2); the second takes as input character, location information, short motivation and produces the sequence of LIGHT game engine executable actions needed to achieve the motivation. This sequence of actions is provided by the human expert demonstrations as mentioned in Section 2. At this stage, the environment contains a motivated main character to perform a quest and a location for them to start in. We now focus on aligning the world with the quest to ensure that the quest is playable and achievable. Intuitively, to ensure that a quest is achievable, the world needs to contain all of the entitieslocations, characters, and objects mentioned within the quest. To this end, the alignment process involves training three BERT-based (Devlin et al., 2018) biencoder retrieval models to retrieve the most likely characters, locations, and objects required flesh the environment out and make the quest achievable. We use the same biencoder architecture proposed by Urbanek et al. (2019) which encodes context using one transformer and candidates with another scoring candidates via inner product between the two encoded vectors. The character retrieval model is conditioned on the initial character, quest, and locationproducing additional characters required to complete the world. We follow the setup in Ammanabrolu et al. (2021) and restrict worlds to only contains 2 characters at maximum but note that this method is extendable to greater numbers of characters. Similarly, the location retrieval model is also conditioned on the same thingsproducing, in this case, 4 neighbors to the initial location (resulting in worlds that are 5 locations large). These locations are connected to the initial location and a character can move between them by using commands such as go west , go up etc.. Once these characters and locations are added to the world, the object retrieval model predicts the set of objects that are required to be distributed for each location given all the character information present in it. The final game environment instance is complete once this object set has been added. Generating Curriculums. We generate curriculums by building off of our procedural LIGHT game instance generation pipeline. We make the observation that the original quests in LIGHT are heavily skewed towards certain quest typeswith the majority involving goals and short motivations that contain objectives related to getting an object, and hitting or hugging another character (Figure 3). We further note that the first verb in the short motivation forms the basis of the quest for that agent. Actions in LIGHT, and more generally in text games, are executed in the game engines on the basis of verbsengine subroutines are linked to verbs with nouns forming argumentsand as such are primarily responsible for changing the state of the world. For example, get sword invokes the get subroutine that places an object, in this case a sword, in the character's surrounding into their inventory. As the quest is generated early in the pipeline, with the world and the rest of the components being conditioned on it, we can say that the first verb in the short motivation is an important dimension along which we can assess the distribution of individual LIGHT game instances. Thus, concretely, the verb counts from the short motivation aggregated over a set of quests represents the primary dimension along which we measure the distribution of quests. Parametrizing Curriculum Difficulty. Given the relative imbalance of this multinomial distribution, as seen in Figure 3, we hypothesize that a LIGHT agent only learns to do well on certain types of objectives and not othersmemorizing trajectories for less seen quest types, i.e. those found in the tail of the distribution. Preliminary evidence for this hypothesis is also seen in Prabhumoye et al. (2020), where they show a positive correlation between the number of instances of a particular type of quest during training and the final test goal-achievement performance. Based on these observations and our initial hypothesis, we use this particular dimension to parametrize curriculum difficulty for training LIGHT agentsquest types that are rarer in the initial training data will be harder for the agent to generalize to in a zero-shot setting. Intuitively, we seek to create curriculums that contain a diverse set of game instances with quest types that are not often found in the initial training data. Our earlier observations let us hypothesize that this will enable the LIGHT agent to more effectively learn from rare instances of quests as op-g e t g i v e hu g h i t f o ll o w t a k e s t e a l p u t g r ee t s h o w d r o p r e m o v e k ill e a t m a k e u s e w a n t b e f r i e n d n ee d s ee 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 N o r m a li z e d V e r b C o un t Short Motivations Distribution in Initial Quests Figure 3: Normalized top-20 verb count distribution of short motivations of the LIGHT-Quests dataset. posed to memorizing the corresponding trajectories. To this end, the generated curriculums each consist of a pool of quests with steadily decreasing quest type imbalance. In our case, this imply that the flatness of the multinomial distribution increases until it tends towards being uniform with respect to the categorical quest type variable. This is done by running the procedural generation pipeline iteratively until the number of instances for the highest count quest type is within n of the lowest count quest type. The total number of additional generated instances is held fixed across curriculums, only the task distribution of quest types within each curriculum changes. Figure 6 shows that decreasing n has the intended effect of decreasing imbalance with respect to verb types. Generating using this pipeline has the added effect of increasing diversity within the pool of each available quest type. One measure of diversity within the pool of a single quest type is the types of nouns contained within the short motivationsthese generally correspond to the characters, locations, and objects mentioned. Figure 6 shows that decreasing imbalance in the verb types for a short motivation also results in decreasing imbalance in noun types, once again corresponding to decreasing n . Short motivation gen-8102 Persona + Motivation Full action/dialoguehistory Setting Encoder Action & Dialogue Policy Networks Update GameEngine Action Utterance LIGHT Environment DM Partner LIGHTAgent Update Reward Reward Figure 5: Architecture and training pipeline for the LIGHT RL Agent (Ammanabrolu et al., 2021). eration is one of the first steps in the pipeline, i.e. the rest of the pipeline is conditioned on it, and as such increasing the flatness of the distribution there has the effects of increasing distribution for downstream components. A2C Curriculum Training. Overall training is done via A2C (Mnih et al., 2016) a policy gradient algorithm that maximizes long-term expected reward by comparing the advantage A ( s t , a  t ) of taking an action a t in a state s t to the average value of taking any valid action as predicted by the critic V ( s t ) . The setup and network architectures used are similar to Ammanabrolu et al. (2021) and are summarized in Figure 5. At every step, the LIGHT agent receives as input the text describing the setting, the character's persona & motivation, and the full dialogue history. This is then encoded using a transformer based encoder and sent to the action and dialogue policy networks which output an ac-tion/dialogue utterance. These are then passed into the LIGHT environment which process them and returns rewards to be used by the agent. Rewards. As seen in Figure 5, all actions, either those of the agent-in-training or the partner agent, are processed by the engine, checking for goal state completionhence known as act goals . For example, if the LIGHT agent had the motivation to acquire a sword, the goal could be completed via a: self act completion : where the agent acquires a sword itself by picking it up, stealing it, convincing the partner to drop theirs so you can pick it up, etc. partner act completion : where the agent uses dialogue utterances to convince their partner to achieve the goal for them (e.g., by persuading the Pipeline Step Model Hits@10 F1 Ppl World Generation Location Biencoder 0.543 0.153 Object Biencoder 0.563 0.154 Character Starspace 0.653 0.289 Quest Generation Short Motive BART -0.488 7.55 Goal Action BART -0.763 3.75 Table 1: Procedural generation evaluation showing metrics for each individual model in the pipeline. partner to give them the sword). The naturalness of the dialogue utterances is further rated by a learned Dungeon Master (DM), a transformer-based ranker model trained on human demonstrations to score how relevant the utterance is given the character's persona and motivation. Further training details are provided in Appendix A.1. We conduct two separate evaluations: the first measures the effectiveness of the various models in the procedural environment generation pipeline as well as the effectiveness of the pipeline as a whole. The second provides zero-shot ablations of the LIGHT RL agents trained on the resulting curriculums and answers the questions (1) how does the relative difficulty of the training quests effect test perfor-mance?; (2) how does the diversity of the environments during training effect test performance?; and (3) how are the results of the previous questions affected by pre-training? All of the models in the pipeline described in Section 2 are trained using only the training set of the original LIGHT and LIGHT-Quests data. LIGHT-Quests inherits characters, locations, and objects from the original LIGHT dataset and adds on motivations and goals in the form of quests. Thus, the character, location, and object retrieval models are evaluated on the LIGHT unseen test set and the motivation and goal generation models are evaluated on the LIGHT-Quests test set. We report the standard array of metrics: hits@10 and F1 ranking prediction score for retrieval models; and F1 (as a harmonic average of BLEU-1 (Papineni et al., 2002) and ROUGE-1 (Lin, 2004)) and perplexity for generative models. Hyperparameters for all models are found in Appendix A.6. (1) character retrieval is easier than retrieving location and objects; and (2) goal action generation is easier than motivation generation. We hypothesize that the first trend is a direct consequence of the fact that generated motivations and goals regularly contain the names of the characters involved but mostly leave implicit information such as the objects requirede.g. the action hit dragon as a knight would require a weapon such as a sword to be equipped first. The second trend stems from the fact that goal actions can often be thought of as condensed version of the short motivationnumber of tokens required to generate goal actions is far less than short motivations. This implies that the goal action model is akin to a summarization model as opposed to the short motivation model which has the more difficult task of generating the motivation with only initial persona and location information. This evaluation tests the LIGHT RL agent's ability to zero-shot generalize to unseen environments. For all experiments in this study, agents were each zero-shot evaluated on 211 human demonstrations from the LIGHT-Quests test set for a single episode per quest across three independent runs. They were measured on the basis of whether or not they were able to achieve their goals in the environments conditioned on their personas: act goals measuring their ability to act consistently, and speech goals reflecting their ability to speak naturally. The study ablates across three dimensions in order to answer the posed research questions relating to: (1) curriculum difficulty, (2) curriculum diversity, and (3) agent pre-training. Curriculum Difficulty. To measure the overall effectiveness of the distribution tuning technique shown in Section 3, we vary the parameter n used to measure curriculum difficultynote that a lower n corresponds to a flatter distribution and so is higher difficulty. As seen in Fig. 6, we generate pools of quests with steadily increasing difficulty with varying n based on the range of the original untuned distributionwith the agents being trained on each pool separately as well as all of them in sequence through a curriculum. Agents received 10 7 total environment interactions per parallel A2C agent in a batch of 16. For the curriculum learning method, the agent received 2 . 5  10 6 interactions per pool of quests starting with the initial pool of untuned quests and then sequentially with n = 64 , 16 , 2 resulting in a total of 10 7 total environment interactions per parallel A2C agent. Curriculum Diversity. The variations in the combinations of quests and worlds themselves seen at training time has potential to effect zero-shot performance (Samvelyan et al., 2021). We introduce two baselines that change the relative diversities of resulting quests in the curriculums, to contrast with our proposed procedural generation pipeline. Generated quest details are found in Appendix A.5. Sampled Curriculums. Inspired by Chawla et al. (2002); Graves et al. (2017), we explore an alternate method of creating curriculums by simply oversampling the same rare quests found in the tails of the distributions. This method does not generate new environments via the pipeline, instead choosing to sample rarer instances of quests with a higher weight when initializing each parallel A2C ac-tor. This means that the distribution of verbs looks similar to what it is in Figure 6 but the quests within a pool are repeated multiple times and so contain no new diversity. Randomly Generated Curriculums. On the other side of the diversity spectrum, we test a method that follows the same steps as the pipeline proposed in Section 2 with the modi-fication that the selection process for each step in the pipeline is random. The characters, objects, location are randomly selected and the generated motivations per character are conditioned on these randomly created worlds. This results in a significantly higher diversity of quests per poolat the expense of the relative coherence of the overall environment. Scratch. No pre-training is done, the encoder is a 3-layer randomly initialized transformer and trained along with the policy networks. Adaptive. Pre-training is done on the tasks introduced in Ammanabrolu et al. (2021) by training a 12 layer transformer with 256 million parameters using a cross-entropy loss as seen in Humeau et al. (2020). These weights are then transferred to the encoder used during RL training then frozen with 3 randomly initialized-layers appended. The encoder is multi-task trained on both pushshift.io Reddit (Baumgartner et al., 2020) and the commonsense dataset ATOMIC-LIGHT (Am-manabrolu et al., 2021), giving the agent general priors on how to act and speak. It is then fine-tuned in LIGHT, giving the agent further domain specific priors. Specific task details are provided in Appendix A.1. Analysis. Table 2 presents the results of this evaluation. We first report that the overall proportion of a pool of procedurally generated environments that contain achievable quests or goals for a single curriculum is 0 . 89 . This metric provides a proxy for measuring the accuracy of the alignment process and the overall error rate of the pipeline. The high achievability rate means that only a small proportion of LIGHT RL A2C agents will waste Expt. environment interactions learning from quests that cannot be completedincreasing this rate even further would likely also improve sample efficiency. Further, we see that just the distribution tuning by itself shows no significant gains in performance over the baselines trained on the original data and in fact loses performance in certain cases. In contrast, learning from the individually tuned quest pools in a sequential curriculum increases performance significantly. This appears to indicate that LIGHTRL agents need to be trained with quests pools of steadily increasing difficultystarting immediately on a set of quests with a high proportion of rare, generated quests can degrade performance. The significantly increased performance of the procedurally generated curriculums over the sampled and randomly generated curriculums indicates the relative importance of diversity within a single quest typebut only up to a certain extent. The sampled quests contain multiple instances of the same quest type but the generated ones have higher 8105 variabilityleading to an increased observation space, ensuring that the agent cannot simply memorize trajectories. On the other hand, randomly generated quests have even higher variability but sacrifice relative coherenceit is more likely that the world contains unlikely scenarios, e.g. a desert and swamp being located right next to each other resulting in significantly decreased performance. We'd finally like to note that the adaptive pre-trained model takes advantage of the generated curriculums and distribution tuning more than the non-pre-trained scratch encoder, showing consis-tenly higher performance across the board. We hypothesize that this is likely a consequence of the adaptive model having greater model capacitythe pre-training enabling it to learn generalizable representations of the generated environments. Overall, trends in performance are independent of pre-trainingboth the scratch and the adaptive pre-trained model benefit significantly from learning from the procedurally generated curriculums. Text-based Game Playing and Generation. Recent text game playing works have focused on tackling three primary challenges: (1) how to represent agent knowledge to effectively operate in partially observable environments (Adhikari et al., 2020; Sautier et al., 2020); (2) scaling RL algorithms to handle combinatorial natural language state-action spaces (Zahavy et al., 2018; Ammanabrolu and Hausknecht, 2020; Jang et al., 2021); and (3) giving agents commonsense priors to better reason about the world (Murugesan et al., 2020, 2021) On the flip side, we have procedural generation of games with works such as Short and Adams (2017); Risi and Togelius (2019); Khalifa et al. (2020) that focus on creating content especially for 2D visual games via search or reinforcement learning based methods. Ammanabrolu et al. (2020b,a) use knowledge graphs to ground language and produce worlds and quests separately for text games from existing corpora such as stories. Fan et al. (2019) leverage LIGHT to learn to generate interactive fiction worlds on the basis of locations, characters, and objectsthis work is closest in spirit to our own World Generation module later on. They all focus on either generating or playing games. Pietquin et al., 2011; Fatemi et al., 2016) and response generation (Li et al., 2016) have used RL to boost performance. As noted by Ammanabrolu et al. (2021), the negotiation tasks of (Yarats and Lewis, 2017; Lewis et al., 2017), where two agents are trying to convince each other to perform certain actions, are related to the tasks in LIGHT-Quests. These works all lack environment grounding. Curriculum Learning. Curriculums in reinforcement learning have traditionally been used to set goals of steadily increasing difficulty for an agent (Bengio et al., 2009; Schmidhuber, 2013). The difficulty of these curriculums are generally measured difficulty via proxy of agent performance (Narvekar et al., 2020)methods either choose to adversarially set goals of steadily increasing difficulty (Sukhbaatar et al., 2018; Racaniere et al., 2019; Dennis et al., 2020; Campero et al., 2021) or to maximize learning performance based on environment instances an agent finds difficult historically (Graves et al., 2017; Portelas et al., 2020). While we were inspired by these works, they all focus on searching for goals for agents which can be difficult to scale to complex tasks such our own natural language motivation-based goals. We'd also like to note that most works using procedural generation to benchmark RL agents such as Cobbe et al. (2020); Kttler et al. (2020); Samvelyan et al. (2021) rely on the underlying richness of the game engines to generate novel environments as opposed to learning to generate. We focus on the problem of improving zero-shot generalization abilities of goal-driven RL agents to act and speak via natural language. An (obviously) key component of achieving this is to train the RL agents on a balanced training dataset that matches the test data in distribution. As this is an unlikely scenario in most real-world applications, we make the observation that we can artificially augment our pool of training environments by generating curriculums to mimic this. In our text game domain, with goal-driven situated natural language agents, we hypothesizeand gather supporting evidence suggestingthat an effective way to parametrize such distributions is by looking at the primary verbs within an agent's motivation and bringing the distribution of verb types as close to uniform as possible. Curriculum training significantly increases an agent's ability to generalize to novel scenarios. As noted by Urbanek et al. (2019) and Ammanabrolu et al. (2021), the ability to speak and act in these textual fantasy worlds has implications for domains beyond text-games. Text games are a platform where agents can interact in a relatively isolated environment and learn to interactively communicate effectively through natural language in a situated manner. Our methods use both large language models and deep reinforcement learning and are prone to the pitfalls that other contemporary methods using these techniques face, especially in the areas of dialogue and text game systems. We mitigate this first pitfall by restricting our current system to a retrieval based dialogue, ensuring that we can filter out non-normative dialogue usages beforehand, though we will note that the system can be extended to generative systems as described in Prabhumoye et al. (2020). Further, the LIGHT dataset is crowdsourced and contains data biases that can be attributed to the crowdworkers tasked with creating the data. Dinan et al. (2020) provides an in depth discussion regarding the inherent dataset biases, such as gender bias in the distribution of characters, in LIGHT and techniques to mitigate themwe follow these methods to reduce their effects on both the environment generation and agent training procedures.
objective abstain abstain objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method other method other other other other other other other objective abstain objective abstain objective objective abstain abstain abstain method abstain abstain abstain