A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interaction. In this work, we explore this direction by designing a simulator and a set of synthetic tasks in the movie domain that allow such interactions between a learner and a teacher. We investigate how a learner can benefit from asking questions in both offline and online reinforcement learning settings, and demonstrate that the learner improves when asking questions. Finally, real experiments with Mechanical Turk validate the approach. Our work represents a first step in developing such end-to-end learned interactive dialogue agents.
A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interaction. In this work, we explore this direction by designing a simulator and a set of synthetic tasks in the movie domain that allow such interactions between a learner and a teacher. We investigate how a learner can benefit from asking questions in both offline and online reinforcement learning settings, and demonstrate that the learner improves when asking questions. Finally, real experiments with Mechanical Turk validate the approach. Our work represents a first step in developing such end-to-end learned interactive dialogue agents.
1 INTRODUCTION
When a student is asked a question by a teacher, but is not confident about the answer, they may ask for clarification or hints. A good conversational agent (a learner/bot/student) should have this ability to interact with a dialogue partner (the teacher/user). However, recent efforts have mostly focused on learning through fixed answers provided in the training set, rather than through interactions. In that case, when a learner encounters a confusing situation such as an unknown surface form (phrase or structure), a semantically complicated sentence or an unknown word, the agent will either make a (usually poor) guess or will redirect the user to other resources (e.g., a search engine, as in Siri). Humans, in contrast, can adapt to many situations by asking questions.
We identify three categories of mistakes a learner can make during dialogue1: (1) the learner has problems understanding the surface form of the text of the dialogue partner, e.g., the phrasing of a question; (2) the learner has a problem with reasoning, e.g. they fail to retrieve and connect the relevant knowledge to the question at hand; (3) the learner lacks the knowledge necessary to answer the question in the first place – that is, the knowledge sources the student has access to do not contain the needed information.
All the situations above can be potentially addressed through interaction with the dialogue partner. Such interactions can be used to learn to perform better in future dialogues. If a human student has problems understanding a teacher’s question, they might ask the teacher to clarify the question. If the student doesn’t know where to start, they might ask the teacher to point out which known facts are most relevant. If the student doesn’t know the information needed at all, they might ask the teacher to tell them the knowledge they’re missing, writing it down for future use.
In this work, we try to bridge the gap between how a human and an end-to-end machine learning dialogue agent deal with these situations: our student has to learn how to learn. We hence design a simulator and a set of synthetic tasks in the movie question answering domain that allow a bot to interact with a teacher to address the issues described above. Using this framework, we explore how a bot can benefit from interaction by asking questions in both offline supervised settings and online reinforcement learning settings, as well as how to choose when to ask questions in the latter setting. In both cases, we find that the learning system improves through interacting with users.
1This list is not exhaustive; for example, we do not address a failure in the dialogue generation stage.
Finally, we validate our approach on real data where the teachers are humans using Amazon Mechanical Turk, and observe similar results.
2 RELATED WORK
Learning language through interaction and feedback can be traced back to the 1950s, when Wittgenstein argued that the meaning of words is best understood from their use within given language games (Wittgenstein, 2010). The direction of interactive language learning through language games has been explored in the early seminal work of Winograd (Winograd, 1972), and in the recent SHRDLURN system (Wang et al., 2016). In a broader context, the usefulness of feedback and interactions has been validated in the setting of multiple language learning, such as second language learning (Bassiri, 2011) and learning by students (Higgins et al., 2002; Latham, 1997; Werts et al., 1995).
In the context of dialogue, with the recent popularity of deep learning models, many neural dialogue systems have been proposed. These include the chit-chat type end-to-end dialogue systems (Vinyals & Le, 2015; Li et al., 2015; Sordoni et al., 2015), which directly generate a response given the previous history of user utterance. It also include a collection of goal-oriented dialogue systems (Wen et al., 2016; Su et al., 2016; Bordes & Weston, 2016), which complete a certain task such as booking a ticket or making a reservation at a restaurant. Another line of research focuses on supervised learning for question answering from dialogues (Dodge et al., 2015; Weston, 2016), using either a given database of knowledge (Bordes et al., 2015; Miller et al., 2016) or short stories (Weston et al., 2015). As far as we know, current dialogue systems mostly focus on learning through fixed supervised signals rather than interacting with users.
Our work is closely related to the recent work of Weston (2016), which explores the problem of learning through conducting conversations, where supervision is given naturally in the response during the conversation. Their work introduced multiple learning schemes from dialogue utterances. In particular the authors discussed Imitation Learning, where the agent tries to learn by imitating the dialogue interactions between a teacher and an expert student; Reward-Based Imitation Learning, which only learns by imitating the dialogue interactions which have have correct answers; and Forward Prediction, which learns by predicting the teacher’s feedback to the student’s response. Despite the fact that Forward Prediction does not uses human-labeled rewards, the authors show that it yields promising results. However, their work did not fully explore the ability of an agent to learn via questioning and interaction. Our work can be viewed as a natural extension of theirs.
3 THE TASKS
In this section we describe the dialogue tasks we designed2. They are tailored for the three different situations described in Section 1 that motivate the bot to ask questions: (1) Question Clarification, in which the bot has problems understanding its dialogue partner’s text; (2) Knowledge Operation, in which the bot needs to ask for help to perform reasoning steps over an existing knowledge base; and (3) Knowledge Acquisition, in which the bot’s knowledge is incomplete and needs to be filled.
For our experiments we adapt the WikiMovies dataset (Weston et al., 2015), which consists of roughly 100k questions over 75k entities based on questions with answers in the open movie dataset (OMDb). The training/dev/test sets respectively contain 181638 / 9702 / 9698 examples. The accuracy metric corresponds to the percentage of times the student gives correct answers to the teacher’s questions.
Each dialogue takes place between a teacher and a bot. In this section we describe how we generate tasks using a simulator. Section 4.2 discusses how we test similar setups with real data using Mechanical Turk.
The bot is first presented with facts from the OMDb KB. This allows us to control the exact knowledge the bot has access to. Then, we include several teacher-bot question-answer pairs unrelated to the question the bot needs to answer, which we call conversation histories3. In order to explore the
2 Code and data are available at https://github.com/facebook/MemNN/tree/master/AskingQuestions. 3 These history QA pairs can be viewed as distractions and are used to test the bot’s ability to separate the
wheat from the chaff. For each dialogue, we incorporate 5 extra QA pairs (10 sentences).
benefits of asking clarification questions during a conversation, for each of the three scenarios, our simulator generated data for two different settings, namely, Question-Answering (denoted by QA), and Asking-Question (denoted by AQ). For both QA and AQ, the bot needs to give an answer to the teacher’s original question at the end. The details of the simulator can be found in the appendix.
3.1 QUESTION CLARIFICATION.
In this setting, the bot does not understand the teacher’s question. We focus on a special situation where the bot does not understand the teacher because of typo/spelling mistakes, as shown in Figure 1. We intentionally misspell some words in the questions such as replacing the word “movie” with “movvie” or “star” with “sttar”.4 To make sure that the bot will have problems understanding the question, we guarantee that the bot has never encountered the misspellings before—the misspellingintroducing mechanisms in the training, dev and test sets are different, so the same word will be misspelled in different ways in different sets. We present two AQ tasks: (i) Question Paraphrase where the student asks the teacher to use a paraphrase that does not contain spelling mistakes to clarify the question by asking “what do you mean?”; and (ii) Question Verification where the student asks the teacher whether the original typo-bearing question corresponds to another question without the spelling mistakes (e.g., “Do you mean which film did Tom Hanks appear in?”). The teacher will give feedback by giving a paraphrase of the original question without spelling mistakes (e.g., “I mean which film did Tom Hanks appear in”) in Question Paraphrase or positive/negative feedback in Question Verification. Next the student will give an answer and the teacher will give positive/negative feedback depending on whether the student’s answer is correct. Positive and negative feedback are variants of “No, that’s incorrect” or “Yes, that’s right”5. In these tasks, the bot has access to all relevant entries in the KB.
3.2 KNOWLEDGE OPERATION
The bot has access to all the relevant knowledge (facts) but lacks the ability to perform necessary reasoning operations over them; see Figure 2. We focus on a special case where the bot will try to understand what are the relevant facts. We explore two settings: Ask For Relevant Knowledge (Task 3) where the bot directly asks the teacher to point out the relevant KB fact and Knowledge Verification (Task 4) where the bot asks whether the teacher’s question is relevant to one particular KB fact. The teacher will point out the relevant KB fact in the Ask For Relevant Knowledge setting or give a positive or negative response in the Knowledge Verification setting. Then the bot will give an answer to the teacher’s original question and the teacher will give feedback on the answer.
3.3 KNOWLEDGE ACQUISITION
For the tasks in this subsection, the bot has an incomplete KB and there are entities important to the dialogue missing from it, see Figure 3. For example, given the question “Which movie did Tom Hanks star in?”, the missing part could either be the entity that the teacher is asking about (question entity for short, which is Tom Hanks in this example), the relation entity (starred actors), the answer to the question (Forrest Gump), or the combination of the three. In all cases, the bot has little chance of giving the correct answer due to the missing knowledge. It needs to ask the teacher the answer to acquire the missing knowledge. The teacher will give the answer and then move on to other questions (captured in the conversational history). They later will come back to reask the question. At this point, the bot needs to give an answer since the entity is not new any more.
Though the correct answer has effectively been included in the earlier part of the dialogue as the answer to the bot’s question, as we will show later, many of the tasks are not as trivial as they look when the teacher reasks the question. This is because the bot’s model needs to memorize the missing entity and then construct the links between the missing entities and known ones. This is akin to the real world case where a student might make the same mistake again and again even though each time the teacher corrects them if their answer is wrong. We now detail each task in turn.
4Many reasons could lead to the bot not understanding the teacher’s question, e.g., the teacher’s question has an unknown phrase structure, rather than unknown words. We choose to use spelling mistakes because of the ease of dataset construction.
5In the datasets we build, there are 6 templates for positive feedback and 6 templates for negative feedback.
Missing Question Entity: The entity that the teacher is asking about is missing from the knowledge base. All KB facts containing the question entity will be hidden from the bot. In the example for Task 5 in Figure 3, since the teacher’s question contains the entity Tom Hanks, the KB facts that contain Tom Hanks are hidden from the bot.
Missing Answer Entity: The answer entity to the question is unknown to the bot. All KB facts that contain the answer entity will be hidden. Hence, in Task 6 of Figure 3, all KB facts containing the answer entity Forrest Gump will be hidden from the bot.
Missing Relation Entity: The relation type is unknown to the bot. In Task 7 of Figure 3, all KB facts that express the relation starred actors are hidden from the bot.
Missing Triples: The triple that expresses the relation between the question entity and the answer entity is hidden from the bot. In Task 8 of Figure 3, the triple “Forrest Gump (question entity) starred actors Tom Hanks (answer entity)” will be hidden.
Missing Everything: The question entity, the relation entity, the answer entity are all missing from the KB. All KB facts in Task 9 of Figure 3 will be removed since they either contain the relation entity (i.e., starred actors), the question entity (i.e., Forrest Gump) or the answer entity Tom Hanks.
4 TRAIN/TEST REGIME
We now discuss in detail the regimes we used to train and test our models, which are divided between evaluation within our simulator and using real data collected via Mechanical Turk.
4.1 SIMULATOR
Using our simulator, our objective was twofold. We first wanted to validate the usefulness of asking questions in all the settings described in Section 3. Second, we wanted to assess the ability of our student bot to learn when to ask questions. In order to accomplish these two objectives we explored training our models with our simulator using two methodologies, namely, Offline Supervised Learning and Online Reinforcement Learning.
4.1.1 OFFLINE SUPERVISED LEARNING
The motivation behind training our student models in an offline supervised setting was primarily to test the usefulness of the ability to ask questions. The dialogues are generated as described in the previous section, and the bot’s role is generated with a fixed policy. We chose a policy where answers to the teacher’s questions are correct answers 50% of the time, and incorrect otherwise, to add a degree of realism. Similarly, in tasks where questions can be irrelevant they are only asked correctly 50% of the time.6
The offline setting explores different combinations of training and testing scenarios, which mimic different situations in the real world. The aim is to understand when and how observing interactions between two agents can help the bot improve its performance for different tasks. As a result we construct training and test sets in three ways across all tasks, resulting in 9 different scenarios per task, each of which correspond to a real world scenario.
The three training sets we generated are referred to as TrainQA, TrainAQ, and TrainMix. TrainQA follows the QA setting discussed in the previous section: the bot never asks questions and only tries to immediately answer. TrainAQ follows the AQ setting: the student, before answering, first always asks a question in response to the teacher’s original question. TrainMix is a combination of the two where 50% of time the student asks a question and 50% does not.
The three test sets we generated are referred to as TestQA, TestAQ, and TestModelAQ. TestQA and TestAQ are generated similarly to TrainQA and TrainAQ, but using a perfect fixed policy (rather than 50% correct) for evaluation purposes. In the TestModelAQ setting the model has to get the form of the question correct as well. In the Question Verification and Knowledge Verification tasks there are many possible ways of forming the question and some of them are correct – the model has to choose the right question to ask. E.g. it should ask “Does it have something to do with the fact that Larry Crowne directed by Tom Hanks?”rather than “Does it have something to do with the fact that Forrest Gump directed by Robert Zemeckis?” when the latter is irrelevant (the candidate list of questions is generated from the known knowledge base entries with respect to that question). The policy is trained using either the TrainAQ or TrainMix set, depending on the training scenario. The teacher will reply to the question, giving positive feedback if the student’s question is correct and no response and negative feedback otherwise. The student will then give the final answer. The difference between TestModelAQ and TestAQ only exists in the Question Verification and Knowledge Verification tasks; in other tasks there is only one way to ask the question and TestModelAQ and TestAQ are identical.
To summarize, for every task listed in Section 3 we train one model for each of the three training sets (TrainQA, TrainAQ, TrainMix) and test each of these models on the three test sets (TestQA, TestAQ, and TestModelAQ), resulting in 9 combinations. For the purpose of notation the train/test combination is denoted by “TrainSetting+TestSetting”. For example, TrainAQ+TestQA denotes a model which is trained using the TrainAQ dataset and tested on TestQA dataset. Each combination has a real world interpretation. For instance, TrainAQ+TestQA would refer to a scenario where a student can ask the teacher questions during learning but cannot to do so while taking an exam. Similarly, TrainQA+TestQA describes a stoic teacher that never answers a student’s question at either learning or examination time. The setting TrainQA+TestAQ corresponds to the case where a lazy
6This only makes sense in tasks like Question or Knowledge Verification. In tasks where the question is static such as ‘What do you mean?” there is no way to ask an irrelevant question, and we do not use this policy.
student never asks question at learning time but gets anxious during the examination and always asks a question.
4.1.2 ONLINE REINFORCEMENT LEARNING (RL)
We also explored scenarios where the student learns the ability to decide when to ask a question. In other words, the student learns how to learn.
Although it is in the interest of the student to ask questions at every step of the conversation, since the response to its question will contain extra information, we don’t want our model to learn this behavior. Each time a human student asks a question, there’s a cost associated with that action. This cost is a reflection of the patience of the teacher, or more generally of the users interacting with the bot in the wild: users won’t find the bot engaging if it always asks clarification questions. The student should thus be judicious about asking questions and learn when and what to ask. For instance, if the student is confident about the answer, there is no need for it to ask. Or, if the teacher’s question is so hard that clarification is unlikely to help enough to get the answer right, then it should also refrain from asking.
We now discuss how we model this problem under the Reinforcement Learning framework. The bot is presented with KB facts (some facts might be missing depending on the task) and a question. It needs to decide whether to ask a question or not at this point. The decision whether to ask is made by a binary policy PRLQuestion. If the student chooses to ask a question, it will be penalized by costAQ. We explored different values of costAQ ranging from [0, 2], which we consider as modeling the patience of the teacher. The goal of this setting is to find the best policy for asking/notasking questions which would lead to the highest cumulative reward. The teacher will appropriately reply if the student asks a question. The student will eventually give an answer to the teacher’s initial question at the end using the policy PRLAnswer, regardless of whether it had asked a question. The student will get a reward of +1 if its final answer is correct and −1 otherwise. Note that the student can ask at most one question and that the type of question is always specified by the task under consideration. The final reward the student gets is the cumulative reward over the current dialogue episode. In particular the reward structure we propose is the following:
For each of the tasks described in Section 3, we consider three different RL scenarios. Good-Student: The student will be presented with all relevant KB facts. There are no misspellings or unknown words in the teacher’s question. This represents a knowledgable student in the real world that knows as much as it needs to know (e.g., a large knowledge base, large vocabulary). This setting is identical across all missing entity tasks (5 - 9). Poor-Student: The KB facts or the questions presented to the student are flawed depending on each task. For example, for the Question Clarification tasks, the student does not understand the question due to spelling mistakes. For the Missing Question Entity task the entity that the teacher asks about is unknown by the student and all facts containing the entity will be hidden from the student. This setting is similar to a student that is underprepared for the tasks. Medium-Student: The combination of the previous two settings where for 50% of the questions, the student has access to the full KB and there are no new words or phrases or entities in the question, and 50% of the time the question and KB are taken from the Poor-Student setting.
4.2 MECHANICAL TURK DATA
Finally, to validate our approach beyond our simulator by using real language, we collected data via Amazon Mechanical Turk. Due to the cost of data collection, we focused on real language versions of Tasks 4 (Knowledge Verification) and 8 (Missing Triple), see Secs. 3.2 and 3.3 for the simulator versions. That is, we collect dialogues and use them in an offline supervised learning setup similar to Section 4.1.1. This setup allows easily reproducible experiments.
For Mechanical Turk Task 4, the bot is asked a question by a human teacher, but before answering can ask the human if the question is related to one of the facts it knows about from its memory.
It is then required to answer the original question, after some additional dialog turns relating to other question/answer pairs (called “conversational history”, as before). For Task 8, the bot is asked a question by a human but lacks the triple in its memory that would be needed to answer it. It is allowed to ask for the missing information, the human responds to the question in free-form language. The bot is then required to answer the original question, again after some “conversational history” has transpired.
We collect around 10,000 episodes (dialogues) for training, 1000 for validation, and 2500 for testing for each of the two tasks. In each case, we give instructions to the Turkers that still follow the original form of the task, but make the tasks contain realistic language written by humans. The instructions given to the Turkers are given in the appendix.
For both tasks, while the human turkers replace the simulator that the bot was previously conversing with, the bot’s dialogue actions (capabilities) are essentially unchanged from before. That is, when answering questions, now the bot is required to answer a human’s questions rather than templated questions from the simulator. When the bot is asking questions, the bot still asks in the same form as before, e.g. questions like “Does it have something to do with X” for Task 4 or “I don’t know. What’s the answer?” for Task 8. However, now its questions are answered by a human. In both cases (asking and answering) the human data is richer with potentially more complex language and lexical variability. Examples of the collected dialogues are given in Figure 5.
5 MODELS
For both offline supervised and online RL settings, we use the End-to-End Memory Network model (MemN2N) (Sukhbaatar et al., 2015) as a backbone. The model takes as input the last utterance of the dialogue history (the question from the teacher) as well as a set of memory contexts including short-term memories (the dialogue history between the bot and the teacher) and long-term memories
(the knowledge base facts that the bot has access to), and outputs a label. We refer readers to the Appendix for more details about MemN2N.
Offline Supervised Settings: The first learning strategy we adopt is the reward-based imitation strategy (denoted vanilla-MemN2N) described in (Weston, 2016), where at training time, the model maximizes the log likelihood probability of the correct answers the student gave (examples with incorrect final answers are discarded). Candidate answers are words that appear in the memories, which means the bot can only predict the entities that it has seen or known before.
We also use a variation of MemN2N called “context MemN2N” (Cont-MemN2N for short) where we replace each word’s embedding with the average of its embedding (random for unseen words) and the embeddings of the other words that appear around it. We use both the preceeding and following words as context and the number of context words is a hyperparameter selected on the dev set.
An issue with both vanilla-MemN2N and Cont-MemN2N is that the model only makes use of the bot’s answers as signals and ignores the teacher’s feedback. We thus propose to use a model that jointly predicts the bot’s answers and the teacher’s feedback (denoted as TrainQA (+FP)). The bot’s answers are predicted using a vanilla-MemN2N and the teacher’s feedback is predicted using the Forward Prediction (FP) model as described in (Weston, 2016). We refer the readers to the Appendix for the FP model details. At training time, the models learn to jointly predict the teacher’s feedback and the answers with positive reward. At test time, the model will only predict the bot’s answer.
For the TestModelAQ setting described in Section 4, the model needs to decide the question to ask. Again, we use vanilla-MemN2N that takes as input the question and contexts, and outputs the question the bot will ask.
Online RL Settings: A binary vanilla-MemN2N (denoted as PRL(Question)) is used to decide whether the bot should or should not ask a question, with the teacher replying if the bot does ask something. A second MemN2N is then used to decide the bot’s answer, denoted as PRL(Answer). PRL(Answer) for QA and AQ are two separate models, which means the bot will use different models for final-answer prediction depending on whether it chooses to ask a question or not.7
We use the REINFORCE algorithm (Williams, 1992) to update PRL(Question) and PRL(Answer). For each dialogue, the bot takes two sequential actions (a1, a2): to ask or not to ask a question (denoted as a1); and guessing the final answer (denoted as a2). Let r(a1, a2) denote the cumulative reward for the dialogue episode, computed using Table 1. The gradient to update the policy is given by:
p(a1, a2) = PRL(Question)(a1) · PRL(answer)(a2) ∇J(θ) ≈ ∇ log p(a1, a2)[r(a1, a2)− b]
(1)
where b is the baseline value, which is estimated using another MemN2N model that takes as input the query x and memory C, and outputs a scalar b denoting the estimation of the future reward. The baseline model is trained by minimizing the mean squared loss between the estimated reward b and actual cumulative reward r, ||r − b||2. We refer the readers to (Ranzato et al., 2015; Zaremba & Sutskever, 2015) for more details. The baseline estimator model is independent from the policy models and the error is not backpropagated back to them.
In practice, we find the following training strategy yields better results: first train only PRL(answer), updating gradients only for the policy that predicts the final answer. After the bot’s final-answer policy is sufficiently learned, train both policies in parallel8. This has a real-world analogy where the bot first learns the basics of the task, and then learns to improve its performance via a question-asking policy tailored to the user’s patience (represented by costAQ) and its own ability to asnwer questions.
7An alternative is to train one single model for final answer prediction in both AQ and QA cases, similar to the TrainMix setting in the supervised learning setting. But we find training AQ and QA separately for the final answer prediction yields a little better result than the single model setting.
8 We implement this by running 16 epochs in total, updating only the model’s policy for final answers in the first 8 epochs while updating both policies during the second 8 epochs. We pick the model that achieves the best reward on the dev set during the final 8 epochs. Due to relatively large variance for RL models, we repeat each task 5 times and keep the best model on each task.
6 EXPERIMENTS
6.1 SIMULATOR
Offline Results: Offline results are presented in Tables 2, 7 and 8 (the latter two are in the appendix). Table 7 presents results for the vanilla-MemN2N and Forward Prediction models. Table 2 presents results for Cont-MemN2N, which is better at handling unknown words. We repeat each experiment 10 times and report the best result. Finally, Table 8 presents results for the test scenario where the bot itself chooses when to ask questions. Observations can be summarized as as follows:
- Asking questions helps at test time, which is intuitive since it provides additional evidence:
• TrainAQ+TestAQ (questions can be asked at both training and test time) performs the best across all the settings.
• TrainQA+TestAQ (questions can be asked at training time but not at test time) performs worse than TrainQA+TestQA (questions can be asked at neither training nor test time) in tasks Question Clarification and Knowledge Operation due to the discrepancy between training and testing.
• TrainQA+TestAQ performs better than TrainQA+TestQA on all Knowledge Acquisition tasks, the only exception being the Cont-MemN2N model on the Missing Triple setting. The explanation is that for most tasks in Knowledge Acquisition, the learner has no chance of giving the correct answer without asking questions. The benefit from asking is thus large enough to compensate for the negative effect introduced by data discrepancy between training and test time.
• TrainMix offers flexibility in bridging the gap between datasets generated using QA and AQ, very slightly underperforming TrainAQ+TestAQ, but gives competitive results on both TestQA and TestAQ in the Question Clarification and Knowledge Operations tasks.
• TrainAQ+TestQA (allowing questions at training time but forbid questions at test time) performs the worst, even worse than TrainQA+TestQA. This has a real-world analogy where a student becomes dependent on the teacher answering their questions, later struggling to answer the test questions without help.
• In the Missing Question Entity task (the student does not know about the question entity), the Missing Answer Entity task (the student does not know about the answer entity), and Missing Everything task, the bot achieves accuracy less than 0.01 if not asking questions at test time (i.e., TestQA).
• The performance of TestModelAQ, where the bot relies on its model to ask questions at test time (and thus can ask irrelevant questions) performs similarly to asking the correct question at test time (TestAQ) and better than not asking questions (TestQA).
- Cont-MemN2N significantly outperforms vanilla-MemN2N. One explanation is that considering context provides significant evidence distinguishing correct answers from candidates in the dialogue history, especially in cases where the model encounters unfamiliar words.
RL Results For the RL settings, we present results for Task 2 (Question Verification) and Task 6 (Missing Answer Entities) in Figure 6. Task 2 represents scenarios where different types of student
have different abilities to correctly answer questions (e.g., a poor student can still sometimes give correct answers even when they do not fully understand the question). Task 6 represents tasks where a poor learner who lacks the knowledge necessary to answer the question can hardly give a correct answer. All types of students including the good student will theoretically benefit from asking questions (asking for the correct answer) in Task 6. We show the percentage of question-asking versus the cost of AQ on the test set and the accuracy of question-answering on the test set vs the cost of AQ. Our main findings were:
• A good student does not need to ask questions in Task 2 (Question Verification), because they already understand the question. The student will raise questions asking for the correct answer when cost is low for Task 6 (Missing Answer Entities).
• A poor student always asks questions when the cost is low. As the cost increases, the frequency of question-asking declines.
• As the AQ cost increases gradually, good students will stop asking questions earlier than the medium and poor students. The explanation is intuitive: poor students benefit more from asking questions than good students, so they continue asking even with higher penalties.
• As the probability of question-asking declines, the accuracy for poor and medium students drops. Good students are more resilient to not asking questions.
6.2 MECHANICAL TURK
Results for the Mechanical Turk Tasks are given in Table 3. We again compare vanilla-MemN2N and Cont-MemN2N, using the same TrainAQ/TrainQA and TestAQ/TestQA combinations as before, for Tasks 4 and 8 as described in Section 4.2. We tune hyperparameters on the validation set and repeat each experiment 10 times and report the best result.
While performance is lower than on the related Task 4 and Task 8 simulator tasks, we still arrive at the same trends and conclusions when real data from humans is used. The performance was expected to be lower because (i) real data has more lexical variety, complexity and noise; and (ii) the training set was smaller due to data collection costs (10k vs. 180k). We perform an analysis of the difference between simulated and real training data (or combining the two) in the appendix, which shows that using real data is indeed important and measurably superior to using simulated data.
More importantly, the same main conclusion is observed as before: TrainAQ+TestAQ (questions can be asked at both training and test time) performs the best across all the settings. That is, we show that a bot asking questions to humans learns to outperform one that only answers them.
7 CONCLUSIONS
In this paper, we explored how an intelligent agent can benefit from interacting with users by asking questions. We developed tasks where interaction via asking questions is desired. We explore both online and offline settings that mimic different real world situations and show that in most cases, teaching a bot to interact with humans facilitates language understanding, and consequently leads to better question answering ability.

This paper introduces a simulator and a set of synthetic tasks for evaluating a dialogue agent's ability to learn from user feedback. For solving these tasks, the paper uses memory networks (Sukhbaatar et al., 2015) learned through previously proposed supervised learning and reinforcement learning methods. In this setup, it is demonstrated that the agent learning from feedback (e.g. through question asking or question clarification) performs better.
The motivation for the paper is excellent; dialogue agents which learn directly from unstructured human feedback (as opposed to reward signals alone) could be very useful in real-world applications. However, the paper falls short on the execution. All the numerous experiments presented are based on the synthetic dialogue simulator, which is highly artificial and different from real-world dialogues. The simulator is based on a simple factoid question-answering framework, which normally is not considered dialogue and which appears to be solvable with a few hand-crafted rules. The framework also assumes that the user's feedback is always correct and is given in one of a handful of forms (e.g. paraphrase of original question without typos) and that the agent can learn from examples of another agent asking questions or making clarifications, which simplifies the task even further.
Because of the artificial setting and limited scope of the experiments, it seems difficult to draw conclusions about how to learn from unstructured user feedback. To test the hypothesis that it is possible to learn from such user feedback, I would strongly recommend the authors to continue working on this project by carrying out experiments with real human users (even in the factoid question answering domain, if necessary). This would provide much stronger evidence that a dialogue agent can learn from such feedback.
Other comments:
- The abstract uses the phrase "interactive dialogue agents". What is meant by "interactive" dialogue agents? All dialogue agents interact with the user, so isn't it redundant to call them interactive?
- A major limitation of the experiments is that the questions the agent can ask are specified a priori. If I understand correctly, in the supervised learning setting the agent is trained to imitate the questions of another rule-based agent. While in the RL setting, the paper states "For each dialogue, the bot takes two sequential actions $(a_1 , a_2)$: to ask or not to ask a question (denoted as a_1 ); and guessing the final answer (denoted as a_2)". This means the agent learns *when* to ask questions but not *what* questions to ask.
- Related to the previous comment, in the sub-section "ONLINE REINFORCEMENT LEARNING (RL)" the paper states "We also explored scenarios where the student learns the ability to decide when to ask a question and what to ask.". Please clarify this by removing the part "what to ask".
- The paper presents an overwhelming amount of results. I understand the benefit of synthetic tasks is precisely the ability to measure many aspects of model performance, but in this case it confuses the reader to present so many results. For example, what was the reason for including the "TrainAQ(+FP)" and "TrainMix" training settings? How do these results help validate the original hypothesis? If they don't, they should be taken out or moved to the appendix.
- Since the contribution of the paper lies in the tasks and evaluation, it might be better to move either the vanilla-MemN2N (Table 2) to the appendix or to move the Cont-MemN2N results (Table 3) to the appendix.
--- UPDATE ---
Following the discussion below and the additional experiments provided by the authors, I have increased my score to 8.


ADVERSARIALLY LEARNED INFERENCE
We introduce the adversarially learned inference (ALI) model, which jointly learns a generation network and an inference network using an adversarial process. The generation network maps samples from stochastic latent variables to the data space while the inference network maps training examples in data space to the space of latent variables. An adversarial game is cast between these two networks and a discriminative network is trained to distinguish between joint latent/data-space samples from the generative network and joint samples from the inference network. We illustrate the ability of the model to learn mutually coherent inference and generation networks through the inspections of model samples and reconstructions and confirm the usefulness of the learned representations by obtaining a performance competitive with state-of-the-art on the semi-supervised SVHN and CIFAR10 tasks.
1 INTRODUCTION
Deep directed generative model has emerged as a powerful framework for modeling complex highdimensional datasets. These models permit fast ancestral sampling, but are often challenging to learn due to the complexities of inference. Recently, three classes of algorithms have emerged as effective for learning deep directed generative models: 1) techniques based on the Variational Autoencoder (VAE) that aim to improve the quality and efficiency of inference by learning an inference machine (Kingma & Welling, 2013; Rezende et al., 2014), 2) techniques based on Generative Adversarial Networks (GANs) that bypass inference altogether (Goodfellow et al., 2014) and 3) autoregressive approaches (van den Oord et al., 2016b;c;a) that forego latent representations and instead model the relationship between input variables directly. While all techniques are provably consistent given infinite capacity and data, in practice they learn very different kinds of generative models on typical datasets.
VAE-based techniques learn an approximate inference mechanism that allows reuse for various auxiliary tasks, such as semi-supervised learning or inpainting. They do however suffer from a wellrecognized issue of the maximum likelihood training paradigm when combined with a conditional independence assumption on the output given the latent variables: they tend to distribute probability mass diffusely over the data space (Theis et al., 2015). The direct consequence of this is that image samples from VAE-trained models tend to be blurry (Goodfellow et al., 2014; Larsen et al., 2015). Autoregressive models produce outstanding samples but do so at the cost of slow sampling speed and foregoing the learning of an abstract representation of the data. GAN-based approaches represent a good compromise: they learn a generative model that produces higher-quality samples than the best VAE techniques (Radford et al., 2015; Larsen et al., 2015) without sacrificing sampling speed and also make use of a latent representation in the generation process. However, GANs lack an efficient inference mechanism, which prevents them from reasoning about data at an abstract level. For instance, GANs don’t allow the sort of neural photo manipulations showcased in (Brock et al., 2016). Recently, efforts have aimed to bridge the gap between VAEs and GANs, to learn generative models with higher-quality samples while learning an efficient inference network (Larsen et al.,
2015; Lamb et al., 2016; Dosovitskiy & Brox, 2016). While this is certainly a promising research direction, VAE-GAN hybrids tend to manifest a compromise of the strengths and weaknesses of both approaches.
In this paper, we propose a novel approach to integrate efficient inference within the GAN framework. Our approach, called Adversarially Learned Inference (ALI), casts the learning of both an inference machine (or encoder) and a deep directed generative model (or decoder) in an GAN-like adversarial framework. A discriminator is trained to discriminate joint samples of the data and the corresponding latent variable from the encoder (or approximate posterior) from joint samples from the decoder while in opposition, the encoder and the decoder are trained together to fool the discriminator. Not only are we asking the discriminator to distinguish synthetic samples from real data, but we are requiring it to distinguish between two joint distributions over the data space and the latent variables.
With experiments on the Street View House Numbers (SVHN) dataset (Netzer et al., 2011), the CIFAR-10 object recognition dataset (Krizhevsky & Hinton, 2009), the CelebA face dataset (Liu et al., 2015) and a downsampled version of the ImageNet dataset (Russakovsky et al., 2015), we show qualitatively that we maintain the high sample fidelity associated with the GAN framework, while gaining the ability to perform efficient inference. We show that the learned representation is useful for auxiliary tasks by achieving results competitive with the state-of-the-art on the semi-supervised SVHN and CIFAR10 tasks.
2 ADVERSARIALLY LEARNED INFERENCE
Consider the two following probability distributions over x and z:
• the encoder joint distribution q(x, z) = q(x)q(z | x), • the decoder joint distribution p(x, z) = p(z)p(x | z).
These two distributions have marginals that are known to us: the encoder marginal q(x) is the empirical data distribution and the decoder marginal p(z) is usually defined to be a simple, factorized distribution, such as the standard Normal distribution p(z) = N (0, I). As such, the generative process between q(x, z) and p(x, z) is reversed.
ALI’s objective is to match the two joint distributions. If this is achieved, then we are ensured that all marginals match and all conditional distributions also match. In particular, we are assured that the conditional q(z | x) matches the posterior p(z | x). In order to match the joint distributions, an adversarial game is played. Joint pairs (x, z) are drawn either from q(x, z) or p(x, z), and a discriminator network learns to discriminate between the two, while the encoder and decoder networks are trained to fool the discriminator.
The value function describing the game is given by:
min G max D
V (D,G) = Eq(x)[log(D(x, Gz(x)))] + Ep(z)[log(1−D(Gx(z), z))]
= ∫∫ q(x)q(z | x) log(D(x, z))dxdz
+ ∫∫ p(z)p(x | z) log(1−D(x, z))dxdz.
(1)
Algorithm 1 The ALI training procedure. θg, θd ← initialize network parameters repeat
x(1), . . . ,x(M) ∼ q(x) . Draw M samples from the dataset and the prior z(1), . . . ,z(M) ∼ p(z) ẑ(i) ∼ q(z | x = x(i)), i = 1, . . . ,M . Sample from the conditionals x̃(j) ∼ p(x | z = z(j)), j = 1, . . . ,M ρ (i) q ← D(x(i), ẑ(i)), i = 1, . . . ,M . Compute discriminator predictions ρ (j) p ← D(x̃(j), z(j)), j = 1, . . . ,M Ld ← − 1M ∑M i=1 log(ρ (i) q )− 1M ∑M j=1 log(1− ρ (j) p ) . Compute discriminator loss
Lg ← − 1M ∑M i=1 log(1− ρ (i) q )− 1M ∑M j=1 log(ρ (j) p ) . Compute generator loss θd ← θd −∇θdLd . Gradient update on discriminator network θg ← θg −∇θgLg . Gradient update on generator networks
until convergence
An attractive property of adversarial approaches is that they do not require that the conditional densities can be computed; they only require that they can be sampled from in a way that allows gradient backpropagation. In the case of ALI, this means that gradients should propagate from the discriminator network to the encoder and decoder networks.
This can be done using the the reparametrization trick (Kingma, 2013; Bengio et al., 2013b;a). Instead of sampling directly from the desired distribution, the random variable is computed as a deterministic transformation of some noise such that its distribution is the desired distribution. For instance, if q(z | x) = N (µ(x), σ2(x)I), one can draw samples by computing
z = µ(x) + σ(x) , ∼ N (0, I). (2)
More generally, one can employ a change of variable of the form
v = f(u, ) (3)
where is some random source of noise.
The discriminator is trained to distinguish between samples from the encoder (x, ẑ) ∼ q(x, z) and samples from the decoder (x̃, z) ∼ p(x, z). The generator is trained to fool the discriminator, i.e., to generate x, z pairs from q(x, z) or p(x, z) that are indistinguishable one from another. See Figure 1 for a diagram of the adversarial game and Algorithm 1 for an algorithmic description of the procedure.
In such a setting, and under the assumption of an optimal discriminator, the generator minimizes the Jensen-Shannon divergence (Lin, 1991) between q(x, z) and p(x, z). This can be shown using the same proof sketch as in the original GAN paper (Goodfellow et al., 2014).
2.1 RELATION TO GAN
ALI bears close resemblance to GAN, but it differs from it in the two following ways:
• The generator has two components: the encoder, Gz(x), which maps data samples x to z-space, and the decoder Gx(z), which maps samples from the prior p(z) (a source of noise) to the input space. • The discriminator is trained to distinguish between joint pairs (x, ẑ = Gx(x)) and (x̃ = Gx(z), z), as opposed to marginal samples x ∼ q(x) and x̃ ∼ p(x).
2.2 ALTERNATIVE APPROACHES TO FEEDFORWARD INFERENCE IN GANS
The ALI training procedure is not the only way one could learn a feedforward inference network in a GAN setting.
In recent work, Chen et al. (2016) introduce a model called InfoGAN which minimizes the mutual information between a subset c of the latent code and x through the use of an auxiliary distribution
Q(c | x). However, this does not correspond to full inference on z, as only the value for c is inferred. Additionally, InfoGAN requires thatQ(c | x) is a tractable approximate posterior that can be sampled from and evaluated. ALI only requires that inference networks can be sampled from, allowing it to represent arbitrarily complex posterior distributions.
One could learn the inverse mapping from GAN samples: this corresponds to learning an encoder to reconstruct z, i.e. finding an encoder such that Ez∼p(z)[‖z −Gz(Gx(z))‖22] ≈ 0. We are not aware of any work that reports results for this approach. This resembles the InfoGAN learning procedure but with a fixed generative model and a factorial Gaussian posterior with a fixed diagonal variance.
Alternatively, one could decompose training into two phases. In the first phase, a GAN is trained normally. In the second phase, the GAN’s decoder is frozen and an encoder is trained following the ALI procedure (i.e., a discriminator taking both x and z as input is introduced). We call this post-hoc learned inference. In this setting, the encoder and the decoder cannot interact together during training and the encoder must work with whatever the decoder has learned during GAN training. Post-hoc learned inference may be suboptimal if this interaction is beneficial to modeling the data distribution.
2.3 GENERATOR VALUE FUNCTION
As with GANs, when ALI’s discriminator gets too far ahead, its generator may have a hard time minimizing the value function in Equation 1. If the discriminator’s output is sigmoidal, then the gradient of the value function with respect to the discriminator’s output vanishes to zero as the output saturates.
As a workaround, the generator is trained to maximize
V ′(D,G) = Eq(x)[log(1−D(x, Gz(x)))] + Ep(z)[log(D(Gx(z), z))] (4)
which has the same fixed points but whose gradient is stronger when the discriminator’s output saturates.
The adversarial game does not require an analytical expression for the joint distributions. This means we can introduce variable changes without having to know the explicit distribution over the new variable. For instance, sampling from p(z) could be done by sampling ∼ N (0, I) and passing it through an arbitrary differentiable function z = f( ).
However, gradient propagation into the encoder and decoder networks relies on the reparametrization trick, which means that ALI is not directly applicable to either applications with discrete data or to models with discrete latent variables.
2.4 DISCRIMINATOR OPTIMALITY
Proposition 1. Given a fixed generator G, the optimal discriminator is given by
D∗(x, z) = q(x, z)
q(x, z) + p(x, z) . (5)
Proof. For a fixed generator G, the complete data value function is
V (D,G) = Ex,z∼q(x,z)[log(D(x, z))] + Ex,z∼p(x,z)[log(1−D(x, z))]. (6)
The result follows by the concavity of the log and the simplified Euler-Lagrange equation first order conditions on (x, z)→ D(x, z).
2.5 RELATIONSHIP WITH THE JENSEN-SHANNON DIVERGENCE
Proposition 2. Under an optimal discriminator D∗, the generator minimizes the Jensen-Shanon divergence which attains its minimum if and only if q(x, z) = p(x, z).
Proof. The proof is a straightforward extension of the proof in Goodfellow et al. (2014).
2.6 INVERTIBILITY
Proposition 3. Assuming optimal discriminator D and generator G. If the encoder Gx is deterministic, then Gx = G−1z and Gz = G −1 x almost everywhere.
Sketch of proof. Consider the event R = {x : ‖x − (Gx ◦ Gz)(x))‖ > } for some positive . This set can be seen as a section of the (x, z) space over the elements z such that z = Gz(x). The generator being optimal, the probabilities of R under p(x, z) and q(x, z) are equal. Now p(x | z) = δx−Gx(z), where δ is the Dirac delta distribution. This is enough to show that there are no x satisfying the event R and thus Gx = G−1z almost everywhere. By symmetry, the same argument can be applied to show that Gz = G−1x . The complete proof is given in (Donahue et al., 2016), in which the authors independently examine the same model structure under the name Bidirectional GAN (BiGAN).
3 RELATED WORK
Other recent papers explore hybrid approaches to generative modeling. One such approach is to relax the probabilistic interpretation of the VAE model by replacing either the KL-divergence term or the reconstruction term with variants that have better properties. The adversarial autoencoder model (Makhzani et al., 2015) replaces the KL-divergence term with a discriminator that is trained to distinguish between approximate posterior and prior samples, which provides a more flexible approach to matching the marginal q(z) and the prior. Other papers explore replacing the reconstruction term with either GANs or auxiliary networks. Larsen et al. (2015) collapse the decoder of a VAE and the generator of a GAN into one network in order to supplement the reconstruction loss with a learned similarity metric. Lamb et al. (2016) use the hidden layers of a pre-trained classifier as auxiliary reconstruction losses to help the VAE focus on higher-level details when reconstructing. Dosovitskiy & Brox (2016) combine both ideas into a unified loss function.
ALI’s approach is also reminiscent of the adversarial autoencoder model, which employs a GAN to distinguish between samples from the approximate posterior distribution q(z | x) and prior samples. However, unlike adversarial autoencoders, no explicit reconstruction loss is being optimized in ALI, and the discriminator receives joint pairs of samples (x, z) rather than marginal z samples.
Independent work by Donahue et al. (2016) proposes the same model under the name Bidirectional GAN (BiGAN), in which the authors emphasize the learned features’ usefulness for auxiliary supervised and semi-supervised tasks. The main difference in terms of experimental setting is that they use a deterministic q(z | x) network, whereas we use a stochastic network. In our experience, this does not make a big difference when x is a deterministic function of z as the stochastic inference networks tend to become determinstic as training progresses. When using stochastic mappings from z to x, the additional flexiblity of stochastic posteriors is critical.
4 EXPERIMENTAL RESULTS
We applied ALI to four different datasets, namely CIFAR10 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), CelebA (Liu et al., 2015) and a center-cropped, 64× 64 version of the ImageNet dataset (Russakovsky et al., 2015).1
Transposed convolutions are used in Gx(z). This operation corresponds to the transpose of the matrix representation of a convolution, i.e., the gradient of the convolution with respect to its inputs. For more details about transposed convolutions and related operations, see Dumoulin & Visin (2016); Shi et al. (2016); Odena et al. (2016).
4.1 SAMPLES AND RECONSTRUCTIONS
For each dataset, samples are presented (Figures 2a, 3a 4a and 5a). They exhibit the same image fidelity as samples from other adversarially-trained models.
1 The code for all experiments can be found at https://github.com/IshmaelBelghazi/ALI. Readers can also consult the accompanying website at https://ishmaelbelghazi.github.io/ALI.
We also qualitatively evaluate the fit between the conditional distribution q(z | x) and the posterior distribution p(z | x) by sampling ẑ ∼ q(z | x) and x̂ ∼ p(x | z = ẑ) (Figures 2b, 3b, 4b and 5b). This corresponds to reconstructing the input in a VAE setting. Note that the ALI training objective does not involve an explicit reconstruction loss.
We observe that reconstructions are not always faithful reproductions of the inputs. They retain the same crispness and quality characteristic to adversarially-trained models, but oftentimes make mistakes in capturing exact object placement, color, style and (in extreme cases) object identity. The extent to which reconstructions deviate from the inputs varies between datasets: on CIFAR10, which arguably constitutes a more complex input distribution, the model exhibits less faithful reconstructions. This leads us to believe that poor reconstructions are a sign of underfitting.
This failure mode represents an interesting departure from the bluriness characteristic to the typical VAE setup. We conjecture that in the underfitting regime, the latent variable representation learned by ALI is potentially more invariant to less interesting factors of variation in the input and do not devote model capacity to capturing these factors.
4.2 LATENT SPACE INTERPOLATIONS
As a sanity check for overfitting, we look at latent space interpolations between validation set examples (Figure 6). We sample pairs of validation set examples x1 and x2 and project them into z1 and z2 by sampling from the encoder. We then linearly interpolate between z1 and z2 and pass the intermediary points through the decoder to plot the input-space interpolations.
We observe smooth transitions between pairs of examples, and intermediary images remain believable. This is an indicator that ALI is not concentrating its probability mass exclusively around training examples, but rather has learned latent features that generalize well.
4.3 SEMI-SUPERVISED LEARNING
We investigate the usefulness of the latent representation learned by ALI through semi-supervised benchmarks on SVHN and CIFAR10.
We first compare with GAN on SVHN by following the procedure outlined in Radford et al. (2015). We train an L2-SVM on the learned representations of a model trained on SVHN. The last three hidden layers of the encoder as well as its output are concatenated to form a 8960-dimensional feature vector. A 10,000 example held-out validation set is taken from the training set and is used for model selection. The SVM is trained on 1000 examples taken at random from the remainder of the training set. The test error rate is measured for 100 different SVMs trained on different random 1000-example training sets, and the average error rate is measured along with its standard deviation.
Using ALI’s inference network as opposed to the discriminator to extract features, we achieve a misclassification rate that is roughly 3.00 ± 0.50% lower than reported in Radford et al. (2015) (Table 1), which suggests that ALI’s inference mechanism is beneficial to the semi-supervised learning task.
We then investigate ALI’s performance when label information is taken into account during training. We adapt the discriminative model proposed in Salimans et al. (2016). The discriminator takes x and z as input and outputs a distribution over K + 1 classes, where K is the number of categories. When label information is available for q(x, z) samples, the discriminator is expected to predict the label. When no label information is available, the discriminator is expected to predict K + 1 for p(x, z) samples and k ∈ {1, . . . ,K} for q(x, z) samples. Interestingly, Salimans et al. (2016) found that they required an alternative training strategy for the generator where it tries to match first-order statistics in the discriminator’s intermediate activations with respect to the data distribution (they refer to this as feature matching). We found that ALI did not require feature matching to obtain comparable results. We achieve results competitive with the state-of-the-art, as shown in Tables 1 and 2. Table 2 shows that ALI offers a modest improvement over Salimans et al. (2016), more specifically for 1000 and 2000 labeled examples.
We are still investigating the differences between ALI and GAN with respect to feature matching, but we conjecture that the latent representation learned by ALI is better untangled with respect to the classification task and that it generalizes better.
4.4 CONDITIONAL GENERATION
We extend ALI to match a conditional distribution. Let y represent a fully observed conditioning variable. In this setting, the value function reads
min G max D V (D,G) = Eq(x) p(y)[log(D(x, Gz(x,y),y))] +Ep(z) p(y)[log(1−D(Gx(z,y), z,y))] (7) We apply the conditional version of ALI to CelebA using the dataset’s 40 binary attributes. The attributes are linearly embedded in the encoder, decoder and discriminator. We observe how a single element of the latent space z changes with respect to variations in the attributes vector y. Conditional samples are shown in Figure 7.
4.5 IMPORTANCE OF LEARNING INFERENCE JOINTLY WITH GENERATION
To highlight the role of the inference network during learning, we performed an experiment on a toy dataset for which q(x) is a 2D gaussian mixture with 25 mixture components laid out on a grid. The covariance matrices and centroids have been chosen such that the distribution exhibits lots of modes separated by large low-probability regions, which makes it a decently hard task despite the 2D nature of the dataset.
We trained ALI and GAN on 100,000 q(x) samples. The decoder and discriminator architectures are identical between ALI and GAN (except for the input of the discriminator, which receives the concatenation of x and z in the ALI case). Each model was trained 10 times using Adam (Kingma & Ba, 2014) with random learning rate and β1 values, and the weights were initialized by drawing from a gaussian distribution with a random standard deviation.
We measured the extent to which the trained models covered all 25 modes by drawing 10,000 samples from their p(x) distribution and assigning each sample to a q(x) mixture component according to the mixture responsibilities. We defined a dropped mode as one that wasn’t assigned to any sample. Using this definition, we found that ALI models covered 13.4± 5.8 modes on average (min: 8, max: 25) while GAN models covered 10.4± 9.2 modes on average (min: 1, max: 22).
We then selected the best-covering ALI and GAN models, and the GAN model was augmented with an encoder using the learned inverse mapping and post-hoc learned inference procedures outlined in subsection 2.2. The encoders learned for GAN inference have the same architecture as ALI’s encoder. We also trained a VAE with the same encoder-decoder architecture as ALI to outline the qualitative differences between ALI and VAE models.
We then compared each model’s inference capabilities by reconstructing 10,000 held-out samples from q(x). Figure 8 summarizes the experiment. We observe the following:
• The ALI encoder models a marginal distribution q(z) that matches p(z) fairly well (row 2, column a). The learned representation does a decent job at clustering and organizing the different mixture components.
• The GAN generator (row 5, columns b-c) has more trouble reaching all the modes than the ALI generator (row 5, column a), even over 10 runs of hyperparameter search.
• Learning an inverse mapping from GAN samples does not work very well: the encoder has trouble covering the prior marginally and the way it clusters mixture components is not very well organized (row 2, column b). As discussed in subsection 2.2, reconstructions suffer from the generator dropping modes.
• Learning inference post-hoc doesn’t work as well as training the encoder and the decoder jointly. As had been hinted at in subsection 2.2, it appears that adversarial training benefits
from learning inference at training time in terms of mode coverage. This also negatively impacts how the latent space is organized (row 2, column c). However, it appears to be better at matching q(z) and p(z) than when inference is learned through inverse mapping from GAN samples.
• Due to the nature of the loss function being optimized, the VAE model covers all modes easily (row 5, column d) and excels at reconstructing data samples (row 3, column d). However, they have a much more pronounced tendency to smear out their probability density (row 5, column d) and leave “holes” in q(z) (row 2, column d). Note however that recent approaches such as Inverse Autoregressive Flow (Kingma et al., 2016) may be used to improve on this, at the cost of a more complex mathematical framework.
In summary, this experiment provides evidence that adversarial training benefits from learning an inference mechanism jointly with the decoder. Furthermore, it shows that our proposed approach for learning inference in an adversarial setting is superior to the other approaches investigated.
5 CONCLUSION
We introduced the adversarially learned inference (ALI) model, which jointly learns a generation network and an inference network using an adversarial process. The model learns mutually coherent inference and generation networks, as exhibited by its reconstructions. The induced latent variable mapping is shown to be useful, achieving results competitive with the state-of-the-art on the semisupervised SVHN and CIFAR10 tasks.
ACKNOWLEDGMENTS
The authors would like to acknowledge the support of the following agencies for research funding and computing support: NSERC, Calcul Québec, Compute Canada. We would also like to thank the developers of Theano (Bergstra et al., 2010; Bastien et al., 2012; Theano Development Team, 2016), Blocks and Fuel (van Merriënboer et al., 2015), which were used extensively for the paper. Finally, we would like to thank Yoshua Bengio, David Warde-Farley, Yaroslav Ganin and Laurent Dinh for their valuable feedback.
A HYPERPARAMETERS
Operation Kernel Strides Feature maps BN? Dropout Nonlinearity
Gz(x) – 3× 32× 32 input Convolution 5× 5 1× 1 32
√ 0.0 Leaky ReLU
Convolution 4× 4 2× 2 64 √
0.0 Leaky ReLU Convolution 4× 4 1× 1 128 √ 0.0 Leaky ReLU Convolution 4× 4 2× 2 256 √
0.0 Leaky ReLU Convolution 4× 4 1× 1 512 √ 0.0 Leaky ReLU Convolution 1× 1 1× 1 512 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 128 × 0.0 Linear
Gx(z) – 64× 1× 1 input Transposed convolution 4× 4 1× 1 256
√ 0.0 Leaky ReLU
Transposed convolution 4× 4 2× 2 128 √
0.0 Leaky ReLU Transposed convolution 4× 4 1× 1 64 √ 0.0 Leaky ReLU Transposed convolution 4× 4 2× 2 32 √
0.0 Leaky ReLU Transposed convolution 5× 5 1× 1 32 √ 0.0 Leaky ReLU
Convolution 1× 1 1× 1 32 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 3 × 0.0 Sigmoid
D(x) – 3× 32× 32 input Convolution 5× 5 1× 1 32 × 0.2 Maxout Convolution 4× 4 2× 2 64 × 0.5 Maxout Convolution 4× 4 1× 1 128 × 0.5 Maxout Convolution 4× 4 2× 2 256 × 0.5 Maxout Convolution 4× 4 1× 1 512 × 0.5 Maxout
D(z) – 64× 1× 1 input Convolution 1× 1 1× 1 512 × 0.2 Maxout Convolution 1× 1 1× 1 512 × 0.5 Maxout
D(x, z) – 1024× 1× 1 input Concatenate D(x) and D(z) along the channel axis
Convolution 1× 1 1× 1 1024 × 0.5 Maxout Convolution 1× 1 1× 1 1024 × 0.5 Maxout Convolution 1× 1 1× 1 1 × 0.5 Sigmoid
Optimizer Adam (α = 10−4, β1 = 0.5, β2 = 10−3) Batch size 100
Epochs 6475 Leaky ReLU slope, maxout pieces 0.1, 2
Weight, bias initialization Isotropic gaussian (µ = 0, σ = 0.01), Constant(0)
Table 3: CIFAR10 model hyperparameters (unsupervised). Maxout layers (Goodfellow et al., 2013) are used in the discriminator.
Operation Kernel Strides Feature maps BN? Dropout Nonlinearity
Gz(x) – 3× 32× 32 input Convolution 5× 5 1× 1 32
√ 0.0 Leaky ReLU
Convolution 4× 4 2× 2 64 √
0.0 Leaky ReLU Convolution 4× 4 1× 1 128 √ 0.0 Leaky ReLU Convolution 4× 4 2× 2 256 √
0.0 Leaky ReLU Convolution 4× 4 1× 1 512 √ 0.0 Leaky ReLU Convolution 1× 1 1× 1 512 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 512 × 0.0 Linear
Gx(z) – 256× 1× 1 input Transposed convolution 4× 4 1× 1 256
√ 0.0 Leaky ReLU
Transposed convolution 4× 4 2× 2 128 √
0.0 Leaky ReLU Transposed convolution 4× 4 1× 1 64 √ 0.0 Leaky ReLU Transposed convolution 4× 4 2× 2 32 √
0.0 Leaky ReLU Transposed convolution 5× 5 1× 1 32 √ 0.0 Leaky ReLU
Convolution 1× 1 1× 1 32 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 3 × 0.0 Sigmoid
D(x) – 3× 32× 32 input Convolution 5× 5 1× 1 32 × 0.2 Leaky ReLU Convolution 4× 4 2× 2 64 √ 0.2 Leaky ReLU
Convolution 4× 4 1× 1 128 √
0.2 Leaky ReLU Convolution 4× 4 2× 2 256 √ 0.2 Leaky ReLU Convolution 4× 4 1× 1 512 √
0.2 Leaky ReLU D(z) – 256× 1× 1 input
Convolution 1× 1 1× 1 512 × 0.2 Leaky ReLU Convolution 1× 1 1× 1 512 × 0.2 Leaky ReLU
D(x, z) – 1024× 1× 1 input Concatenate D(x) and D(z) along the channel axis
Operation Kernel Strides Feature maps BN? Dropout Nonlinearity
Gz(x) – 3× 64× 64 input Convolution 2× 2 1× 1 64
√ 0.0 Leaky ReLU
Convolution 7× 7 2× 2 128 √
0.0 Leaky ReLU Convolution 5× 5 2× 2 256 √ 0.0 Leaky ReLU Convolution 7× 7 2× 2 256 √
0.0 Leaky ReLU Convolution 4× 4 1× 1 512 √ 0.0 Leaky ReLU
Convolution 1× 1 1× 1 512 × 0.0 Linear Gx(z) – 512× 1× 1 input
Transposed convolution 4× 4 1× 1 512 √
0.0 Leaky ReLU Transposed convolution 7× 7 2× 2 256 √ 0.0 Leaky ReLU Transposed convolution 5× 5 2× 2 256 √
0.0 Leaky ReLU Transposed convolution 7× 7 2× 2 128 √ 0.0 Leaky ReLU Transposed convolution 2× 2 1× 1 64 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 3 × 0.0 Sigmoid
D(x) – 3× 64× 64 input Convolution 2× 2 1× 1 64
√ 0.0 Leaky ReLU
Convolution 7× 7 2× 2 128 √
0.0 Leaky ReLU Convolution 5× 5 2× 2 256 √ 0.0 Leaky ReLU Convolution 7× 7 2× 2 256 √
0.0 Leaky ReLU Convolution 4× 4 1× 1 512 √ 0.0 Leaky ReLU
D(z) – 512× 1× 1 input Convolution 1× 1 1× 1 1024 × 0.2 Leaky ReLU Convolution 1× 1 1× 1 1024 × 0.2 Leaky ReLU
D(x, z) – 1024× 1× 1 input Concatenate D(x) and D(z) along the channel axis
Operation Kernel Strides Feature maps BN? Dropout Nonlinearity
Gz(x) – 3× 64× 64 input Convolution 4× 4 2× 2 64
√ 0.0 Leaky ReLU
Convolution 4× 4 1× 1 64 √
0.0 Leaky ReLU Convolution 4× 4 2× 2 128 √ 0.0 Leaky ReLU Convolution 4× 4 1× 1 128 √
0.0 Leaky ReLU Convolution 4× 4 2× 2 256 √ 0.0 Leaky ReLU Convolution 4× 4 1× 1 256 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 2048 √ 0.0 Leaky ReLU Convolution 1× 1 1× 1 2048 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 512 × 0.0 Linear
Gx(z) – 256× 1× 1 input Convolution 1× 1 1× 1 2048
√ 0.0 Leaky ReLU
Convolution 1× 1 1× 1 256 √
0.0 Leaky ReLU Transposed convolution 4× 4 1× 1 256 √ 0.0 Leaky ReLU Transposed convolution 4× 4 2× 2 128 √
0.0 Leaky ReLU Transposed convolution 4× 4 1× 1 128 √ 0.0 Leaky ReLU Transposed convolution 4× 4 2× 2 64 √
0.0 Leaky ReLU Transposed convolution 4× 4 1× 1 64 √ 0.0 Leaky ReLU Transposed convolution 4× 4 2× 2 64 √
0.0 Leaky ReLU Convolution 1× 1 1× 1 3 × 0.0 Sigmoid
D(x) – 3× 64× 64 input Convolution 4× 4 2× 2 64 × 0.2 Leaky ReLU Convolution 4× 4 1× 1 64 √ 0.2 Leaky ReLU
Convolution 4× 4 2× 2 128 √
0.2 Leaky ReLU Convolution 4× 4 1× 1 128 √ 0.2 Leaky ReLU Convolution 4× 4 2× 2 256 √
0.2 Leaky ReLU Convolution 4× 4 1× 1 256 √ 0.2 Leaky ReLU
D(z) – 256× 1× 1 input Convolution 1× 1 1× 1 2048 × 0.2 Leaky ReLU Convolution 1× 1 1× 1 2048 × 0.2 Leaky ReLU
D(x, z) – 2304× 1× 1 input Concatenate D(x) and D(z) along the channel axis
B A GENERATIVE STORY FOR ALI
The Circle of Infinite Painters is a very prolific artistic group. Very little is known about the Circle, but what we do know is that it is composed of two very brilliant artists. It has produced new paintings almost daily for more than twenty years, each one more beautiful than the others. Not only are the paintings exquisite, but their title and description is by itself a literary masterpiece.
However, some scholars believe that things might not be as they appear: certain discrepancies in the Circle’s body of work hints at the Circle being composed of more than one artistic duo. This is what Joseph Discriminator, art critique and world expert on the Circle, believes. He’s recently been working intensively on the subject. Without knowing it, he’s right: the Circle is not one, but two artistic duos.
Xavier and Zach Prior form the creative component of the group. Xavier is a painter and can, in one hour and starting from nothing, produce a painting that would make any great painter jealous. Impossible however for him to explain what he’s done: he works by intuition alone. Zach is an author and his literary talent equals Xavier’s artistic talent. His verb is such that the scenes he describes could just as well be real.
By themselves, the Prior brothers cannot collaborate: Xavier can’t paint anything from a description and Zach is bored to death with the idea of describing anything that does not come out of his head. This is why the Prior brothers depend on the Conditional sisters so much.
Zelda Conditional has an innate descriptive talent: she can examine a painting and describe it so well that the original would seem like an imitation. Xena Conditional has a technical mastery of painting that allows her to recreate everything that’s described to her in the most minute details. However, their creativity is inversely proportional to their talent: by themselves, they cannot produce anything of interest.
As such, the four members of the Circle work in pairs. What Xavier paints, Zelda describes, and what Zach describes, Xena paints. They all work together to fulfill the same vision of a unified Circle of Infinite Painters, a whole greater than the sum of its parts.
This is why Joseph Discriminator’s observations bother them so much. Secretly, the Circle put Mr. Discriminator under surveillance. Whatever new observation he’s made, they know right away and work on attenuating the differences to maintain the illusion of a Circle of Infinite Painters made of a single artistic duo.
Will the Circle reach this ideal, or will it be unmasked by Mr. Discriminator?

After reading the rebuttal, I decided to increase my score. I think ALI somehow stabilizes the GAN training as demonstrated in Fig. 8 and learns a reasonable inference network.
---------------
Initial Review:
This paper proposes a new method for learning an inference network in the GAN framework. ALI's objective is to match the joint distribution of hidden and visible units imposed by an encoder and decoder network. ALI is trained on multiple datasets, and it seems to have a good reconstruction even though it does not have an explicit reconstruction term in the cost function. This shows it is learning a decent inference network for GAN.
There are currently many ways to learn an inference network for GANs: One can learn an inference network after training the GAN by sampling from the GAN and learning a separate network to map X to Z. There is also the infoGAN approach (not cited) which trains the inference network at the same time with the generative path. I think this paper should have an extensive comparison with these other methods and have a discussion for why ALI's inference network is superior to previous works.
Since ALI's inference network is stochastic, it would be great if different reconstructions of a same image is included. I believe the inference network of the BiGAN paper is deterministic which is the main difference with this work. So maybe it is worth highlighting this difference.
The quality of samples is very good, but there is no quantitative experiment to compare ALI's samples with other GAN variants. So I am not sure if learning an inference network has contributed to better generative samples. Maybe including an inception score for comparison can help.
There are two sets of semi-supervised results:
The first one concatenate the hidden layers of the inference network and uses an L2-SVM afterwards. Ideally, concatenating feature maps is not the best way for semi-supervised learning and one would want to train the semi-supervised path at the same time with the generative path. It would have been much more interesting if part of the hidden code was a categorical distribution and another part of it was a continuous distribution like Gaussian, and the inference network on the categorical latent variable was used directly for classification (like semi-supervised VAE). In this case, the inference network would be trained at the same time with the generative path. Also if the authors can show that ALI can disentangle factors of variations with a discrete latent variable like infoGAN, it will significantly improve the quality of the paper.
The second semi-supervised learning results show that ALI can match the state-of-the-art. But my impression is that the significant gain is mainly coming from the adaptation of Salimans et al. (2016) in which the discriminator is used for classification. It is unclear to me why learning an inference network help the discriminator do a better job in classification. How do we know the proposed method is improving the stability of the GAN? My understanding is that one of the main points of learning an inference network is to learn a mapping from the image to the high-level features such as class labels. So it would have been more interesting if the inference path was directly used for semi-supervised learning as I explained above.