Learning to Query, Reason, and Answer Questions On Ambiguous Texts
A key goal of research in conversational systems is to train an interactive agent to help a user with a task. Human conversation, however, is notoriously incomplete, ambiguous, and full of extraneous detail. To operate effectively, the agent must not only understand what was explicitly conveyed but also be able to reason in the presence of missing or unclear information. When unable to resolve ambiguities on its own, the agent must be able to ask the user for the necessary clarifications and incorporate the response in its reasoning. Motivated by this problem we introduce QRAQ (Query, Reason, and Answer Questions), a new synthetic domain, in which a User gives an Agent a short story and asks a challenge question. These problems are designed to test the reasoning and interaction capabilities of a learningbased Agent in a setting that requires multiple conversational turns. A good Agent should ask only non-deducible, relevant questions until it has enough information to correctly answer the User’s question. We use standard and improved reinforcement learning based memory-network architectures to solve QRAQ problems in the difficult setting where the reward signal only tells the Agent if its final answer to the challenge question is correct or not. To provide an upper-bound to the RL results we also train the same architectures using supervised information that tells the Agent during training which variables to query and the answer to the challenge question. We evaluate our architectures on four QRAQ dataset types, and scale the complexity for each along multiple dimensions.
1 Introduction
In recent years, deep neural networks have demonstrated impressive performance on a variety of natural language tasks such as language modeling (Mikolov et al. (2010); Sutskever et al. (2011)), image captioning (Vinyals et al. (2015); Xu et al. (2015)), and machine translation (Sutskever et al. (2014); Cho et al. (2014); Bahdanau et al. (2015)). Encouraged by these results, machine learning researchers are now tackling a variety of even more challenging tasks such as reasoning and dialog. One such recent effort is the so-called “bAbI” problems of Weston et al. (2016). In these problems, the agent is presented with a short story and a challenge question that tests its ability to reason about the events in the story. The stories require the agent to learn unstated constraints, but are otherwise self-contained,
∗The first three authors contributed equally.
requiring no interaction between the agent and the environment. A very recent extension of this work (Weston (2016)) adds interaction by allowing the agent to respond in various ways to a teacher’s questions.
There has also been significant recent interest in learning task-oriented dialog systems such as by Bordes & Weston (2016); Dodge et al. (2016); Williams & Zweig (2016); Henderson et al. (2014); Young et al. (2013). Here the agent is trained to help a user complete a task such as finding a suitable restaurant or movie. These tasks are typically modeled as slot-filling problems in which the agent knows about “slots”, or attributes relevant to the task, and must determine which of the required slot values have been provided, querying the user for the others. The reasoning required to decide on an action in these systems is primarily in determining which slot values the user has provided and which ones are required but still unknown to the agent. Realistic task-oriented dialog, however may require logical reasoning both to minimize irrelevant questions to the user and to focus the inquiry on questions most helpful to solving the user’s task.
In this paper we introduce a new simulator that generates problems in a domain we call QRAQ (Query, Reason, and Answer Questions). In this domain the User provides a story and a challenge question to the agent, but with some of the entities replaced by variables. The introduction of variables, whose value may not be known, means that the agent must now learn additional challenging skills. First it must be able to decide whether it has enough information, in view of existing ambiguities, to answer the question. This requires reasoning about which variables can be deduced from other facts in the problem. Second, if the agent cannot answer the question by reasoning alone, it must learn to query the simulator for a variable value. To do this it must be able to infer which remaining variables are relevant to the question posed. The agent is penalized for asking about irrelevant or deducible variables. Since there may be several rounds of questioning and reasoning, these requirements bring the problem closer to task-oriented dialog and represent a significant increase in the difficulty of the challenge over the original bAbI (“supporting fact”) problems. In another significant departure from previous work on reasoning, including the work on the bAbI problems, we focus on the more realistic and challenging reinforcement learning (RL) setting in which the training agent is only told at the end of the multi-turn interaction whether its answer to the challenge question is correct or not. For an upper bound comparison, we also present the results of supervised training, in which we tell the agent which variable to query at each turn, and what to answer at the end.
In summary, this paper presents two main contributions: (1) a novel domain, inspired by bAbI, but which additionally requires reasoning with incomplete information over multiple turns, and (2) a baseline as well as an improved RL-based memory-network architecture with empirical results on our datasets that explore the robustness of the agent’s reasoning.
2 Related Work
Our work builds on aspects of many different lines of machine learning research for which it is impossible to do full justice in the space available. Most relevant is research on deep neural networks and reinforcement learning for reasoning in natural language domains – in particular, those which make use of synthetic data.
One line of work which inspires our own is the development of novel neural architectures which can achieve deeper “understanding” of text input, thereby enabling more sophisticated reasoning and inference from source materials. In Bordes et al. (2010) for example, the model must integrate world knowledge to learn to label each word in a text with its “concept” which subsumes disambiguation tasks such as pronoun disambiguation. This is similar in spirit to our sub-task of deducing the value of variables but lacks the challenge of answering a question using this information or querying the user for more information. We draw particular inspiration for QRAQ from the bAbI problems of Weston et al. (2016) which are simple, automatically generated natural language stories, along with a variety of questions which can test many aspects of reasoning over the contents of such stories. The dynamic memory networks of Kumar et al. (2016) use the same synthetic domain and include tasks for both part-of-speech classification and question answering, but employ two Gated Recurrent Units (GRUs) to perform inference. Our problems subsume the reasoning required for the
bAbI “supporting fact” tasks and add additional complexity in several ways. First, we allow ambiguous variables which require logical reasoning to deduce. Second, the question is not necessarily answerable with the information supplied and the agent must learn to decide if it has enough information to answer. Third, when the agent does not have the information it needs it must learn to query the user for a relevant fact and integrate the response into its reasoning. Such interactions may span multiple turns, essentially requiring a dialog.
There has been a lot of recent interest on the end-to-end training of dialog systems that are capable of generating a sensible response utterance at each turn, given the context of previous utterances in the dialog (Vinyals & Le (2015); Serban et al. (2016); Lowe et al. (2015); Kadlec et al. (2015); Shang et al. (2015)). Research on this topic tends to focus on large-scale training corpora such as movie subtitles, social media chats, or technical support logs. Because our problems are synthetic, our emphasis is not on the difficulties of understanding realistic language but rather on the mechanisms by which the reasoning and interaction process may be learned. For large corpora it is natural to use supervised training techniques where the Recurrent Neural Networks (RNNs) attempt to replicate the recorded human utterances. However, there are also approaches that envision training via reinforcement learning techniques, given a suitably defined reward function in the dialog (Wen et al. (2016); Su et al. (2015b;a)). We adopt this approach and similarly emphasize RL in this paper.
As described in the previous section, most of the reasoning emphasis in the slot-filling model of task-oriented dialogs is on understanding the user goal and which slot values have been given/which remain unfilled. By contrast, in QRAQ problems the emphasis is on inferring missing information if possible, and reasoning about what is important to ask, which can be much harder than evaluating which of the required slots are still unfilled. In more recent work on end-to-end learning of task-oriented dialog such as Bordes & Weston (2016); Dodge et al. (2016) this paradigm is extended to decompose the main task into smaller tasks each of which must be learned by the agent and composed to accomplish the main task. Williams & Zweig (2016) use an LSTM model that learns to interact with APIs on behalf of the user. Weston (2016) (bAbI-dialog) combines dialog and reasoning to explore how an agent can learn dialog when interacting with a teacher. The dataset is divided into 10 tasks, designed to test the ability of the agent to learn with different supervision regimes. The agent is asked a series of questions and given various forms of feedback according to the chosen scheme. For example: imitating an expert, positive and negative feedback, etc. Our focus is not on comparing supervision techniques, so instead we provide a numeric reward function for QRAQ problems that only assumes knowledge of when the agent’s answer is right. In bAbI-dialog, the agent cannot ask questions, only answer them and receive guidance from the supervisor. The QRAQ problems by contrast allow the agent to query for the values of variables as well as to answer the challenge question. The information received must then be integrated into the reasoning process since the decision about what to query next (or answer) is dependent on the new state. In the bAbI-dialog problems the correct answer to a given question does not depend on previous questions and answers, though previous feedback can improve the agent’s performance.
3 The QRAQ Domain
QRAQ problems have two actors, User and Agent. The User provides a short story, set in a domain similar to the HomeWorld domain of Weston et al. (2016); Narasimhan et al. (2015), and a challenge question. The stories are semantically coherent but may sometimes contain unknown variables, which the Agent may need to resolve to answer the question.1
Consider the simple QRAQ problem in Example 1, where the context for the problem is labeled C1, C2, etc.; the events are labeled E1, E2, etc.; the question is labeled Q; and the ground truth is labeled GT. Here the entities $v and $w are variables whose values are not
1Formally, the solution requires many unstated assumptions which the Agent must learn to correctly solve the problem. These include: domain constraints (e.g. a person can’t be in two places at once), domain closure (the only entities in the world are those referenced explicitly in the story), closed world (the only facts true in the world are those given explicitly in the story), unique names (e.g. Emma != Hannah, kitchen != garden, etc.) and the frame assumption (the only things that change are those explicitly stated to change).
provided to the Agent. In Event E3, for example, $v refers to either Hannah or Emma, but the Agent can’t tell which. In a realistic text this might occur due to spelling or transcription errors, or the use of indefinite pronouns such as “somebody”. Variables such as $u (which aliases Emma) might realistically occur as descriptive references such as “John’s sibling”. The question to be answered “Where is the gift?” is taken to mean “Where is the gift at the end of the story?”, a qualification which the Agent must learn. We call the entity referenced in the question (in this case “the gift”) the protagonist.
C1. Hannah is in the garden.
C2. $u is Emma.
C3. $u is in the garden.
C4. The gift is in the garden.
C5. John is in the kitchen.
C6. The ball is in the kitchen.
C7. The skateboard is in the kitchen
E1. Hannah picks up the gift.
E2. John picks up $x.
E3. $v goes from the garden to the kitchen.
E4. $w walks from the kitchen to the patio.
E5. Having left the garden, $u goes to the patio.
Q. Where is the gift?
GT. $v = Hannah; $w = Hannah; Answer = Patio
Example 1: A QRAQ Problem
C1. Joe is in the kitchen.
C2. Bob is in the kitchen.
C3. Hannah is in the patio.
E1. $v goes from the kitchen to the garden.
E2. $w goes from the garden to the patio.
E3. $x goes from the patio to the basement.
Q. Where is Joe?
GT. $v = Joe; $w = Joe; $x = Hannah; answer = Patio
Example 2: A “deep” QRAQ problem (see Figure 1 for explanation)
The variables occurring in this example can be categorized as follows: A variable whose value can be determined by logical reasoning is called deducible; a variable which is consistently used to refer to an entity throughout the story is called an alias; a variable whose value is potentially required to solve the problem is called relevant (non-relevant variables are called irrelevant). Aliased variables may be defined in the context (e.g. $u is Emma).
In Example 1, $x is irrelevant, since knowing its value doesn’t help solve the problem. Similarly, we can observe that Emma (aliased as $u), leaves the garden in E5, making Hannah the only possible value of $v, so $v is deducible. Only $w remains as a relevant, non-deducible variable. To solve the problem, $w must thus be queried by the Agent.
Example 2 shows a problem that requires two queries to solve. Here all the variables are relevant and none are deducible. Querying, say, $v yields $v = Joe. At this point, $w becomes deducible and we can infer $w = Joe. We now know that Joe is either in the patio or in the basement but not which. A further query for $x reveals that $x = Hannah, so the answer is that Joe is in the patio.
We can visualize the possible solutions to these problems using the “query graphs”
shown in Figure 1. These graphs (technically DAGs) show the query policy required to solve the problem. Each node represents an informational state of the agent, and each edge the outcome of relevant queries. We define the depth of a query graph to be maximum number of variables that must be queried in the worst case2, given the ground truth variable assignments. Examples of such paths in the graphs of Figure 1 are shown in bold. These paths pass only through states consistent with the ground truth. By this definition, Figure 1(a) has depth 1, and figure 1(b) has depth 2. As we discuss in Section 5.2, depth is an important driver of problem complexity.
There are many ways to solve these problems algorithmically. The QRAQ simulator uses a linear-time, graph-based approach to track the possible locations of the protagonist as it processes the events. We will include a detailed description of the simulator and this algorithm when we release the QRAQ datasets to the research community.
4 Learning Agents
We develop and compare two different RL agents: (1) baseRL, which uses the memory network architecture from Sukhbaatar et al. (2015), and (2) impRL, which improves on the memory network architecture of baseRL by using a soft-attention mechanism over memory hops. We emphasize that the only feedback available to both agents, apart from a per-step penalty, is whether their answer to the challenge question is correct or not.
4.1 Control Loop
Both baseRL and impRL use the same control loop shown in Figure 2(a) whose steps are explained below:
(1) Initialization. For each problem, the challenge question is represented as a vector c, where ci is the index of the i-th word in the question in a dictionary. The events and context sentences are encoded similarly and then the event vectors are appended to the context vectors to construct an initial memory matrix S1, where S ij 1 is the index of the j-th word in the i-th sentence in the dictionary. Each word or variable comes from a global dictionary of N words formed by the union of all the words in the training and testing problems.
(2) Action Selection. The agent’s policy at the tth turn in the dialog is a function π(a|St, c) mapping the memory matrix at turn t, St, and question, c, into a distribution over actions. An action, a, could be either a query for a variable or a final answer to the challenge question. The policy network, based on the end-to-end memory network, is shown in section 4.2.
(3) Variable Query and Memory Update. Whenever a query action is selected, the user module provides the true value for the corresponding variable in action at, i.e., the user module provides vt = oracle(at), where vt is the dictionary index of the true word for the variable in action at. Then all occurrences of variable in action at in the memory St are replaced with the true value vt: St+1 = St[at → vt]. The new memory representation St+1 is then used to determine the next action at+1 in the next turn of the dialog.
(4) Final Answer Generation and Termination. If the action is an answer instead of a variable to query, the task terminates and a reward is generated based on whether the answer is correct or not (for the exact values of the rewards, see paragraph on “curriculum learning” in section 5).
4.2 baseRL: End-to-End Memory Network based Policy Learner
The baseRL agent builds on an end-to-end memory network policy as introduced by Sukhbaatar et al. (2015). It maps the memory matrix, S, and the challenge question representation, c, into an action distribution. Specifically, the i-th row of the memory matrix is encoded into a vector mi = ∑ j lj ◦ A[Sij ], where A ∈ Rd×N is an embedding matrix and A[k] returns the k-th column vector, d is the dimension of the embedding, ‘◦’ is an element wise multiplication operator, and lj is a column vector with the structure lkj = (1−j/J)−(k/d)(1−2j/J) with J being the number of words in the sentences. Similarly, the challenge question is converted into a vector q = ∑ j lj ◦A[cj ].
2In the best case the agent may get lucky and query a variable that yields the answer immediately.
The output vector upon addressing and then reading from {mi} in the k-th hop is uk: uk = tanh(H(ok + uk−1)) where ok = ∑ i pkimi, p k i = SoftMax(u ᵀ k−1mi), (1)
Where SoftMax(zi) = e zi/ ∑ j e zj and u0 = q. The policy module is implemented by two separate networks. One is for querying variables and the other is for answering the challenge questions.
Query network output The output of the query network is a distribution over variables in the memory matrix3. Since the problems have at most one variable per sentence, the distribution can be converted into the distribution over sentences without adding additional parameters. Specifically, baseRL implements the following query policy:
πiQ = SoftMax(u ᵀ Mmi) (2)
where uM is the output of the last memory hop and only sentences with variables are considered in the SoftMax computation.
Answer network output The output of the policy module is a distribution over potential actions {a1, ..., aK}. Specifically, baseRL implements the following policy:
πA = SoftMax(WuM + b) (3)
where uM is the output of the last memory hop, W ∈ RK×d and b ∈ RK .
4.3 impRL: Improved End-to-End Memory Network based Policy Learner
In preliminary experiments with the baseRL architecture we tried optimizing the number of memory hops and discovered that the optimal number varied with each dataset. Rather than try to optimize it as a hyperparameter for each dataset, we developed a new architecture shown in Figure 2(b). Unlike the baseRL agent in which the final action-distribution output only conditions on the last hop output from the memory network architecture, the improved RL architecture, impRL, computes the final action-distribution-output over all memory hop outputs (with a fixed number of total hops across all datasets) as follows:
πiQ = SoftMax(u ᵀmi) πA = SoftMax(Wu+ b) u = ∑ j zjuj zj = SoftMax(q ᵀuj) (4)
3A special query action is added to signal a switch from the query network to the answer network.
where W ∈ RK×d and b ∈ RK . Our modification to the memory network architecture of Sukhbaatar et al. (2015) is similar to the adaptive memory mechanism (AM) employed by Weston et al. (2016). The AM mechanism allows a variable number of memory hop computations in memory networks, but it is trained via extra supervision signals to determine when to stop memory hop computation explicitly. Our impRL mechanism departs from the AM mechanism in two ways: (1) it does not require extra supervision signals, and (2) while the AM mechanism uses the last memory hop’s output for prediction, impRL instead uses a weighted average of all memory hops’ outputs. As we show below impRL improves performance over baseRL in our empirical results for the more challenging datasets.
4.4 Policy Gradient & Reward Function
Much of the prior work on reasoning problems has used supervised learning (SL) methods. Here we focus on the far more realistic and challenging setting where the learning agent only has access to a binary reward function that evaluates whether the answer provided by the agent to the challenge question is correct or not. More specifically, we use a reward function that is positive when the action is the correct answer to the challenge question, and negative when the action is a wrong answer to the challenge question or the action is a query to a variable (see section 5 Curriculum Learning for the details). The penalty for a wrong answer is much larger than the penalty for querying a variable. Penalizing queries encourages the agent to query for the value of as few variables as possible. The objective function of the reinforcement learning method is to optimize the expected cumulative reward over (say M)
training problem instances: ∑M m=1 E{ ∑ t r m t }, where rmt is the immediate reward at the time step t of the m-th task. The GPOMDP algorithm with average reward baseline (Weaver & Tao (2001)) is used to calculate the policy gradient which in turn becomes the error signal to train all the parameters in both baseRL and impRL.
5 Data, Training, and Results
We evaluate our methods on four types of datasets described below. Each dataset contains 107,000 QRAQ problems, with 100,000 for training, 2000 for testing, and 5000 for validation.
(Loc). In this dataset, the context sentences describe people in rooms. The event sentences describe people (either by their name or by a variable) moving from room to room. The challenge questions are about the location of a specific person. We created several such datasets, scaling along vocabulary size, the number of sentences per problem (including both the context and event sentences), the number of variables per problem and the depth of the problem. For a definition of “depth” see Section 3. For a detailed configuration of each data set see Table 1.
(+obj). This dataset adds objects to the Loc dataset. The context sentences describe people and objects in rooms. The event sentences describe people moving to and from various rooms, or picking up or dropping objects in rooms. People or objects (but not rooms) may be hidden by variables. The challenge questions are about the location of a person or an object.
(+alias). Adds aliases to the Loc dataset. Some of them are defined in the context.
(+par). This dataset modifies the Loc dataset by substituting sentences with semantically equivalent paraphrases. These paraphrases can change the number of words per sentence, and the ordering of the subject and object.
Pre-training The embedding matrix A is pre-trained via sentence level self-reconstruction. In the memory matrix S, each sentence i is represented as a row vector Si and its corresponding embedding vector is mi = ∑ j lj · A[Sij ]. The self-reconstruction objective is to
maximize ∑ i ∑ j log(pj(S
ij |mi)), where pj = SoftMax(W (j)mi) and W ∈ RN×d. If a word position j has only one word candidate, then such a position is dropped from the objective function.
Time embedding for events If the i-th sentence is an event, a temporal embedding is added to its original embedding so that mi ← mi + T (i), where T (i) ∈ Rd is the i-th column of a special matrix T that is learned from data.
Curriculum Learning We first encourage both reinforcement learning agents to query variables by assigning positive rewards for querying any variable. After convergence under this initial reward function, we switch to the true reward function that assigns a negative reward for querying a variable to reduce the number of unnecessary queries. Specifically, the rewards is +1 for correct final answers, -5 for wrong final answers. We explored five pairs of query reward values for the curriculum: +/-0.01, +/-0.05, +/-0.1, +/-0.5, +/-1, and found that +/-0.05 performed best on a validation set, so that is what we use for our experiments.
Action Mask When choosing an action (query-variable or answer) from the policy network, output units that correspond to choices not available in the set of sentences at that turn4 in the dialog are not considered in the selection process.
Exploration To encourage exploration, which is needed in RL settings, an exploration policy π′ is defined, given the agent’s learned policy π as follows. A random action is chosen with probability and the remaining 1− probability is distributed over actions as π′(a) = (π(a) + δ)/(1 + |A|δ), where δ = /(1− )|A|, and |A| is the number of actions. For our experiments, = 0.1.
We used Adam (Kingma & Ba (2015)) to optimize the parameters of the networks. The number of memory hops is fixed to 4. The embedding dimensionality is fixed to 50.
4When a variable is queried, the simulation engine replaces every occurrence of a variable with its value and returns the updated text to the agent, at the beginning of the next turn.
5.1 Supervised Learning Baselines: baseSL and impSL
We trained the architecture of baseRL and impRL using supervised learning to get an upper-bound on achievable reinforcement learning performance. In the supervised learning setting, the relevant variables and (when appropriate) the correct final answers are provided at each turn, and the cross entropy loss function is used to optimize the parameters.
5.2 Results & Analysis
In our experiments we measure four metrics for each QRAQ datatset: (1) answer-accuracy, the proportion of trajectories (a sequence of variable queries followed by an answer) in which the agent answered correctly, but may have queried irrelevant variables or failed to query relevant ones; (2) trajectory-accuracy, the proportion of trajectories in which the agent queried only relevant variables then answered correctly, but may have left some relevant variables unqueried; (3) trajectory-completeness, the proportion of trajectories in which the agent queried all and only relevant variables before answering correctly; and (4) query accuracy, the proportion of correct queries among all queries made in any trajectory. Note that trajectorycompleteness <= trajectory-accuracy <= answer-accuracy and trajectory-completeness is the most challenging metric by which we measure the success of our experiments.
More precisely, let T be the set of all trajectories produced by the agent on the test problems, TA be the set of trajectories where the agent gave the correct answer, TQ be the set of trajectories completed with only relevant variables queried, TC be the set of trajectories completed with all relevant variables queried, NQ be the number of queried variables, and NrelQ be the number of queried variables that were relevant. Then
answer-accuracy = |TA| |T | ; trajectory-completeness = |TA ∩ TQ ∩ TC | |T | ;
trajectory-accuracy = |TA ∩ TQ| |T | ; query-accuracy = NrelQ NQ .
By comparing the various metrics in the rows of Table 1 we can make a number interesting observations. First, the performance gap between the supervised learning agents and reinforcement learning agents increases as problems become more complex. Figure 3 shows the test trajectory distributions over problem depths. The result shows that impSL performs much better than impRL on deeper problems. This indicates that the RL agent is sensitive to the scaling and hasn’t learned as robust an algorithm as the supervised agent (which, considering the stronger training signal for the supervised agent, was to be expected). Second, for all reinforcement learning agents, the answer accuracy is considerably higher than the trajectory accuracy. This indicates that answering is considerably easier for the agent than solving the whole problem perfectly, with only relevant queries followed by a correct answer.
Third, in the simplest (Loc) datasets (the leftmost 3 to 4 columns) where the number of sentences, number of variables, and depth are all small, both baseRL and impRL do well and furthermore do similarly well. This is also the case for the +obj dataset because it is also similarly simple (in terms of depth range). As expected the answer accuracy, the trajectory accuracy and trajectory completeness get worse as the problems get more complex (in terms of parameters listed in the first 7 rows). This decrease in performance is roughly seen left to right in the 5 columns for the (Loc) datasets. Next, note that the query accuracy results of the leftmost 5 columns closely track the ratio of depth to the number of variables in the problems. That ratio is a rough estimate of the percentage of relevant variables to the total number of variables. An agent who guessed which variables were relevant would be sensitive to this ratio, doing better when it was high and worse when low, lending weight to the hypothesis that the RL agent’s query algorithm is underperforming. Of all the parameters explored, the ‘depth’ and ‘#sentence/prob’ parameters seem the most impactful. Specifically there is a sharp drop-off in performance for both the base and improved architectures in the rightmost (Loc) column where the depth is 4-9 compared with 0-2 in the leftmost four (Loc) columns. Similarly the (+obj) dataset has a low depth and the performance is similarly good. Finally, we note that in the more complex data sets including the rightmost column of the (Loc) set of columns as well as for the (+alias) dataset the performance of both baseRL and impRL is worse than for the simpler datasets but in all these cases impRL improves the answer accuracy significantly.
6 Conclusion
We have introduced the new QRAQ domain that presents an agent with a story consisting of context sentences, temporally ordered event sentences with variables, and a challenge question. Answering the challenge question requires multi-turn interactions in which a good agent should ask only non-deducible and relevant questions at any turn. We presented and evaluated two RL-based memory network architectures, a baseline and an improved architecture in which we added soft-attention over multiple memory hops. Our results show that that both architectures do solve these challenging problems to substantial degrees, even on the quite unforgiving trajectory-completeness measure, and despite being limited to feedback only about whether the the answer is correct. At the same time, as the gap between the supervised and RL approaches shows, there is considerable room for innovation in the reinforcement learning setting for this domain.
Acknowledgments
Xiaoxiao Guo and Satinder Singh’s work on this paper was supported by funding from IBM’s Cognitive Horizons Network. Clemens Rosenbaum’s work was conducted while an intern at IBM Research. The authors would like to thank Iulian Serban for helpful conversations.

This paper investigates a set of tasks that augment the basic bAbI problems. In particular, some of the people and objects in the scenarios are replaced with unknown variables. Some of these variables must be known to solve the question, thus the agent must learn to query for the values of these variables. Interestingly, one can now measure both the performance of the agent in correctly answering the question, and its efficiency in asking for the values of the correct unknown variables (and not variables that are unnecessary to answer the question). This inferring of unknown variables goes beyond what is required for the vanilla version of the bAbI tasks, which are now more or less solved.
The paper is well-written, and the contributions are clear. Due to the very limited vocabulary and structure of the bAbI problems in general, I think these tasks (and variants on them) should be viewed more as basic reasoning tasks than natural language understanding. I’m not convinced by the claim of the paper that this really tests the ‘interaction’ capabilities of agents – while the task is phrased as a kind of interaction, I think it’s more aptly described by simply ‘inferring important unknown variables’, which (while important) is more related to reasoning. I’m not sure whether the connection of this ability to ‘interaction’ is more a superficial one.
That being said, it is certainly true that conversational agents will need basic reasoning abilities to converse meaningfully with humans. I sympathise with the general goal of the bAbI tasks, which is to test these reasoning abilities in synthetic environments, that are just complicated enough (but not more) to drive the construction of interesting models. I am convinced by the authors that their extension to these tasks are interesting and worthy of future investigation, and thus I recommend the acceptance of the paper.


Multi-label learning aims to automatically assign to an instance (e.g., an image or a document) the most relevant subset of labels from a large set of possible labels. The main challenge is to maintain accurate predictions while scaling efficiently on data sets with extremely large label sets and many training data points. We propose a simple but effective neural net approach, the Semantic Embedding Model (SEM), that models the labels for an instance as draws from a multinomial distribution parametrized by nonlinear functions of the instance features. A Gauss-Siedel mini-batch adaptive gradient descent algorithm is used to fit the model. To handle extremely large label sets, we propose and experimentally validate the efficacy of fitting randomly chosen marginal label distributions. Experimental results on eight real-world data sets show that SEM garners significant performance gains over existing methods. In particular, we compare SEM to four recent state-ofthe-art algorithms (NNML, BMLPL, REmbed, and SLEEC) and find that SEM uniformly outperforms these algorithms in several widely used evaluation metrics while requiring significantly less training time.
1 INTRODUCTION
The multi-label learning problem is to learn to predict potentially multiple relevant labels given an instance. Instances that have multiple labels naturally occur in many application domains, including multimedia information retrieval, tag recommendation, semantic scene classification, query categorization, gene function prediction, medical diagnosis, drug discovery, and marketing.
A popular approach to the multi-label learning problem is to embed the labels in a low-dimensional latent space via linear or local non-linear embeddings. The approach of Hsu et al. (2009) projects the label vectors to a random low-dimensional space, fits a regression model in this space, then projects these predictions back to the original label space. Balasubramanian & Lebanon (2012) use a sparsity-regularized least squares reconstruction objective to select a small set of landmark labels that are used to predict the remaining labels. Bi & Kwok (2013) take a similar approach, with a greatly decreased computation cost, by posing the problem of selecting the landmark labels as one of column subset selection and adopting the leverage score sampling approach (Boutsidis et al., 2009). Recently, Yu et al. (2014) and Jing et al. (2015) propose using trace norm regularization to identify a low-dimensional representation of the original large label space. Mineiro & Karampatziakis (2015) use randomized dimensionality reduction to learn a low-dimensional embedding that explicitly captures correlations between the instance features and their labels. These approaches, like other linear embedding methods, assume that the label matrix is low-rank. However, the label matrix in most applications of multi-label learning is a sparse binary matrix, and thus is extremely likely to violate this low-rank assumption (Bhatia et al., 2015).
Rather than working with the original label and feature matrices, some methods work instead with label or feature similarity matrices, and seek to preserve the local structure of the data in the learned low-dimensional latent space. Tai & Lin (2010) use PCA on the label covariance matrix to extract a low-dimensional latent space for labels and Chen & Lin (2012) extend this method to integrate feature information. Lin et al. (2014) apply PCA to a similarity matrix constructed using both label and feature information; this approach is time-consuming as it requires computing a large similarity matrix. Nam et al. (2014) introduce a neural network model to capture non-linear relationships between the input features and the labels. However, this approach is computationally infeasible when the number of possible labels is large. Similarly, Cissé et al. (2016) shows that using a deep learning approach built on top of an informative partitioning of the label space gives good performance; the scalability of this method was not characterized. Prabhu & Varma (2014) propose a method to efficiently train a classification tree by minimizing the Normalized Discounted Cumulative Gain. Rai et al. (2015) assumes that the label vectors are generated by sampling from a weighted combination of label topics, where the mixture coefficients are determined by the instance features.
Bhatia et al. (2015) proposes a multi-phase algorithm (SLEEC) that first clusters the instances into a number of relatively small groups, learns label embeddings for each group via an SVD, and then trains linear regressors from the input features to the latent label factors for each group. SLEEC empirically outperforms previous state-of-the-art multi-label classifiers, but the label embedding in each group is learned from a nearest neighbor graph that is constructed solely from labelling information, ignoring the available feature matrix; the feature matrix has been shown repeatedly to be a source of useful information for label embedding (Chen & Lin, 2012; Lin et al., 2014; Yu et al., 2014; Jing et al., 2015).
The contribution of this paper is a scalable, accurate, and simple neural network approach to multilabel learning. Experiments establish that our method is faster and more accurate than SLEEC, the current state-of-the-art scalable algorithm.
Notation: In the sequel, n is the number of training instances, c is the cardinality of the set of possible labels, d is the dimensionality of the feature vectors, and r is the dimension of the learned latent space. The matrix X ∈ Rn×d contains the instance features, and Y ∈ 0, 1n×c indicates the labels assigned to each instance. We denote the number of observed labels for instance i with `i = ∑c k=1 yik. The notations Ai· and A·j respectively refer to the ith row and jth column of the matrix A. Unless otherwise specified, the notation f(A) denotes the elementwise application of an arbitrary function f to the A, so for example exp(A)ij = exp(aij).
2 THE SEMANTIC EMBEDDING MODEL
Our Semantic Embedding Model (SEM) assumes that the underlying parameters determining the observed labels are low-rank rather than that the observed label matrix is itself low-rank, and it uses a nonlinear model to fit the probability distributions over the labels, conditioned on the instance features.
SEM models the i-th row of Y as the result of `i draws from a multinomial distribution: Yi· ∼ Multinomial(`i;Pi), where P = [
exp(hij)∑c k=1 exp(hik) ] i=1,...,n j=1,...,c . (1)
The parameter matrix H = UVT + 1nbT is the sum of label priors b ∈ Rc and the product of explanatory latent factors associated with the instances (U ∈ Rn×r) and the labels (V ∈ Rc×r). Further, we allow the latent factors associated with each instance to be a nonlinear function of the features associated with that instance, U = f(X,W) for some W to be learned. We note that if f(X,W) = XW, SEM could be viewed as fitting a Bayesian Exponential Family PCA (Mohamed et al., 2009). However, throughout this paper we take f(XW) = σ(XW), where σ(X) = (1 + exp(−X))−1 denotes the elementwise application of the sigmoid function, as we find this gives good results; with this choice, SEM is more naturally viewed as a neural network model.
We fit the SEM parameters by maximizing the likelihood of the observed labels. This is equivalent to minimizing the sum of the KL divergences between the empirical label distributions for each instance and the label distributions predicted by the model (Pawitan, 2001). Accordingly, we define the empirical label distribution matrix G, whose ith row satisfies Gi = Yi/`i, then minimize the
row-wise Kullback-Leibler distance (Yang et al., 2011) between G and P:
JG||P = n∑
i=1 c∑ j=1 Gij log Gij Pij ∼= − n∑ i=1 c∑ j=1 Gij logPij . (2)
Recalling that
Pij = exp(hij)∑c k=1 exp(hik) = exp
( (σ(XW)VT )ij + bj )∑c k=1 exp ((σ(XW)V T )ik + bk) ,
some algebraic manipulations give the final objective
J (W,V,b) = JG||P = − n∑
i=1 c∑ j=1 Gij log exp(σ(XW)i·(V T )·j + bj)∑c k=1 exp(σ(XW)i·(V T )·k + bk)
= − n∑
i=1 c∑ j=1 Gij(σ(XW)i·(V T )·j + bj) + n∑ i=1 log ( c∑ k=1 exp(σ(XW)i·(V T )·k + bk) ) = −Tr(G(σ(XW)VT + 1nbT )T ) + 1Tn log ( exp(σ(XW)VT + 1nb T )1c ) .
(3)
Thus the SEM parameters are learned by solving the optimization problem min
W,V,b J (W,V,b). (4)
Here V ∈ Rc×r are the representations of the labels in a latent semantic space, W ∈ Rd×r controls the nonlinear mapping from the instance features to the same semantic space, and the offsets b ∈ Rc allow for label-specific offsets in the mapping from the semantic space to the log probabilities.
3 MODEL FITTING
The optimization problem (4) is non-convex. To solve it efficiently, we use a Gauss-Siedel approach combined with mini-batching.
Namely, we cyclically update each of W,V, and b using AdaGrad (Duchi et al., 2011) while keeping the other two variable fixed. We compute the gradients using mini-batches. To state the expressions for the gradients with respect to the model parameters, we introduce some helpful notation: A B denotes the entry-wise product of two matrices, M = σ(XW) (1− σ(XW)), and D = Diag ( exp (σ(XW)VT + 1nb T )1c ) . The gradients are readily computed from (2):
G(W) = XT ( M [( D−1 exp ( σ(XW)VT + 1nb T ) −G ) V ])
(5) G(V) = ( exp ( σ(XW)VT + 1nb T )T D−1 −GT ) σ(XW) (6)
G(b) = ( exp ( σ(XW)VT + 1nb T )T D−1 −GT ) 1n. (7)
Using AdaGrad, the update rule for W(τ) is
W(τ) = W(τ−1) − α(τ)W G(W (τ−1)) (8)
where τ is the timestep and αW is a matrix of step sizes computed via( α (τ) W ) iq = ρ√∑τ−1
m=1
( G(W(m)) G(W(m)) ) iq + ε , (9)
where ε and the learning rate ρ determine how much an entry Wij is updated during the first timestep.
V(τ) and b(τ) are computed according to similar updating rules obtained from (8) and (9) by substituting G(W) with G(V) (or G(b)), W with V (or b), and αW with αV (or αb). A listing of the proposed algorithm is given in Algorithm 1. Its computational complexity is O(Tnr(d + c)), where T is the number of epochs. We note that the gradient calculations in lines 7–9 of Algorithm 1 are amenable to parallelization.
Algorithm 1 Mini-Batched Gauss-Siedel Adaptive Gradient Descent for learning SEM parameters Input: Instance feature matrix X ∈ Rn×d, observed label matrix Y ∈ Rn×c, dimensionality of the
latent space r, learning rate ρ and > 0, mini-batch size m < n, and number of epochs T . 1: Initialize W(0), V(0), and b(0) 2: for t = 1, 2, . . . , T do 3: Randomly choose n/m mini-batchs Iz ⊂ {1, ..., n} of size m 4: for b = 1, 2, . . . n/m do 5: Set τ = (t− 1)(n/m) + b 6: Select the data instances in the z-th mini-batch by working with XIz,· in lieu of X 7: Update W(τ) via (8) while fixing V = V(τ−1) and b = b(τ−1) 8: Update V(τ) via the analog of (8) for V while fixing W = W(τ) and b = b(τ−1)
9: Update b(τ) via the analog of (8) for b while fixing W = W(τ) and V = V(τ) 10: end for 11: end for Output: W(τ), V(τ) and b(τ)
3.1 INCREASED EFFICIENCY BY FITTING MARGINALS
Although Algorithm 1 runs in time linear in the dimensions of the model parameters and the input datasets, it can be computationally expensive when there are more than a few thousand labels. To further reduce the running time of our algorithm, we note that in practice, each instance is often associated with `i c labels. To speed up the training, at each timestep τ , rather than attempting to minimize the divergence between the entire empirical and predicted label distributions of each instance, we sample a set of labels L(τ)i for each instance and attempt to minimize the empirical and predicted marginal label distributions over that set of labels L(τ)i . Let PLi denote the set of labels assigned to the ith instance, and ALi denote the set of labels not assigned to that instance. We sample from ALi to form a set NLi, and use L (τ) i = PLi ∪ NL (τ) i . This leads to the modified objective
JMarginal(W,V,b)(τ) = − n∑ i=1 ∑ j∈L(τ)i Gij log exp(σ(XW)i·(V T )·j + bj)∑ k∈L(τ)i exp(σ(XW)i·(VT )·k + bk) . (10)
Note that J (τ) is a random function that changes at each timestep. Minimizing this stochastic objective effectively seeks SEM parameters which fit all the randomly sampled marginals encountered during training. Thus it is important to sample the sets NLi so that the selected marginals capture non-trivial information about the label distributions. One can imagine that uniformly sampling from ALi will not provide very informative marginals. As an improvement on this naïve scheme, we sample labels from ALi with probability proportional to their frequency of occurrence in the training data set. The number of negative labels is set to be β times the number of positive labels i.e., |NLi| = β|PLi| = β`i. Further, when m > 1, to faciliate efficient BLAS operations while mini-batching, we use the same marginals for each instance in the same minibatch, i.e., we fit marginals over L(τ) := ⋃ i∈Iz L (τ) i , where Iz denotes the set of instances in the current minibatch.
In the experiments presented in Section 4, we found that β around 10 suffices when c is relatively small, and β around 100 suffices when c is on the order of tens of thousands.
3.2 LABEL PREDICTION
We present two methods for predicting the labels for a new instance x ∈ Rd given the fitted SEM parameters.
The first uses the generative model behind SEM: form h = σ(xTW)VT+bT and note the probability that the jth label is assigned to that instance is given by
P(yj = 1) = exp(hj)/ c∑
k=1
exp(hk). (11)
Accordingly we assign the most probable labels to x. We call this prediction scheme the direct SEM method; it simply requires choosing the labels corresponding to the largest entries of h.
The second method builds a kernel classifier in the semantic space obtained from the SEM factorization. Following Mineiro & Karampatziakis (2015), a classifier is trained on these semantic representations by solving the optimization problem
min Z∈Rc×s n∑ i=1 `(Yi·,Zψ(xiW)) + λ‖Z‖2F , (12)
where ` is the log-loss penalty and ψ : Rr → Rs is an s-dimensional Random Fourier Feature (RFF) map (Rahimi & Recht, 2007):
ψ(x) = cos (Φx + θ) , (13)
where Φ ∈ Rs×r is a matrix of i.i.d. standard Gaussians and θ ∈ [0, 2π)s is a vector of i.i.d uniform samples from [0, 2π).
At test time, the predicted label probabilities for an instance x are given by Zψ(xW), so we assign the most probable labels according to this model. We refer to this scheme as the kernelized SEM method.
4 EXPERIMENTS
In the sequel we refer to the direct SEM scheme as simply SEM, and the kernelized SEM scheme as SEM-K. We compare SEM and SEM-K with several alternative multi-label learning algorithms: NNML (Nam et al., 2014), REmbed (Mineiro & Karampatziakis, 2015), SLEEC (Bhatia et al., 2015), and BMLPL (Rai et al., 2015). We do not compare to the models proposed in (Tai & Lin, 2010; Chen & Lin, 2012; Bi & Kwok, 2013; Yu et al., 2014; Prabhu & Varma, 2014) because earlier works (Yu et al., 2014; Bhatia et al., 2015) have shown that they are inferior to SLEEC.
4.1 DATASETS
Table 1 summarizes the eight datasets used in our experiments. Here ntrain and ntest are the numbers of training and testing instances, d is the number of features, c is the number of labels/classes, and the avg(`i) column reports the average number of labels per instance. In these datasets, the number of labels varies from 23 to 30938, the average label cardinality varies from 2.508 to 19.020, and the number of instances in different classes varies over a large range. Thus predicting the labels assignments correctly over this collection of datasets is a challenging task.
4.2 METHODOLOGY
The codes of the methods we compare to are provided by the authors, in particular, we note that the computationally intensive portions of REmbed, SLEEC and NNML are implemented in C; by way of comparison, our algorithms are entirely implemented in Matlab. Due to there being several parameters for each method, we hand-tuned the parameters for each dataset as suggested by the authors. All methods were run in MATLAB on a Windows server with 4GB memory and four 2.3GHz CPUs with eight cores.
The prediction performance for each algorithm is evaluated according to widely-used metrics in the field of multi-label classification, viz., label-based Macro-F1 (MaF1) and Micro-F1 (MiF1) and instance-based Precision-at-k (P@k, esp. P@1 and P@3) (Zhang & Zhou, 2014). MaF1 and MiF1 require predefining a threshold to determine the number of labels to be assigned to the testing data. In our experiments, the number of labels assigned to each testing instance was set according to its ground truth.
4.3 PERFORMANCE ON DATASETS WITH SMALL LABEL SETS
First we compare the performance on six multi-label learning problems with c < 1000. To fit both SEM models, we take the number of epochs be 30 and the mini-batch size be 200—i.e., T = 30 and m = 200 in Algorithm 1—and because c is small, we fit the full label distributions. The classification performances of our SEM algorithms and the baseline methods are shown in Table 2. SEM or SEM-K outperform the alternative algorithms in most cases.
Table 3 compares the running times of the algorithms as the size of the dataset is increased, using MediaMill. We see that SEM is the fastest model, followed by REMBED, then closely by SEM-K; the remaining three models are significantly more costly. It is clear that NNML, the previous neural
network approach to multi-label learning costs the most. In the other five algorithms, the latent space dimensionality (r) is set to be 50. SLEEC is expensive because it constructs the nearest neighbor graph among training data and computes the top r eigenvectors of the corresponding similarity matrix, which costs O(n2r + d2r). REmbed is efficient because its main cost is to find the singular vectors of a c× (r + q) matrix (here c is the number of labels and q is a small integer), but its performance is inferior to SEM-K. The BMLPL code provided by the author applies SVD to the training data to initialize than model parameters and then uses conjugate gradient to update the parameters, thus it costs much more than REmbed and our proposed methods.
4.4 PERFORMANCE ON DATASETS WITH LARGE LABEL SETS
We proposed using SEM to fit marginals rather than the entire label distribution when c is large, for computational efficiency. To judge the effectiveness of this proposal, we compare the accuracy and running times of the SEM and SEM-K models with baselines on EurLex-des and Wiki10K, two datasets with c > 1000. As baselines, we use REmbed and SLEEC in accordance with the above discussion which showed that these two methods are efficient and/or have good performance.
The hyperparameters in SLEEC were set according to the original authors’ code: r for EurLex-des and Wiki10K is 100 and 75 respectively, and 3 clusters are used for Eurlex-des and 5 are used for Wiki10K. To fit the SEM models, we used the same value of r as SLEEC on these two datasets and used 10 training epochs. For REmbed, the latent space size r was tuned via cross-validation; r = 300 for Eurlex-des and r = 150 for Wiki10K. The number of Random Fourier Features is 2000 for both REmbed and SEM-K. The latent space size r in SEM is same with SLEEC. The mini-batch sizes and number of epochs are set to be 200 and 10 respectively when fitting the SEM models. The number of threads is set to be 8 for all methods.
Table 4 compares the classification performances of the methods on these two datasets. It is clear that SEM-K with a small set of negative labels obtains better performance than both REmbed and SLEEC. Table 5 shows that, additionally, the SEM-K models are fit much faster than than the other models.
4.5 IMPACT OF THE SIZE OF THE MARGINALS
Figure 1 illustrates the impact of the choice of β on the prediction performance (in terms of P@1) of SEM and SEM-K. The performances of SLEEC and REmbed are included for comparison. The hyperparameters of SLEEC, REmbed and SEM were set as in Section 4.4.
It is evident that the performance of SEM increases significantly in a monotonic fashion with β. However, SEM-K is insensitive to β once it passes a dataset-dependent threshold (e.g., β = 60
for Eurlex-des and β = 100 for Wiki10K). Note that on Wiki10K, even the simpler direct SEM outperforms REmbed when there are sufficient negative labels.
Figure 2 illustrates the effect of β on the running times of SEM and SEM-K. Note that the additional time to fit the classifier in the semantic space required by SEM-K is negligible compared to the time it takes to first fit the direct SEM model.
4.6 ADDITIONAL CONSIDERATIONS
There are other important ways in which the proposed SEM methods can be compared to the baseline multi-label learning methods, including their performance as a function of the latent space dimensionality and as a function of the amount of training. Due to space constraints, a discussion of these two concerns and the convergence behavior of Algorithm 1 is provided in the Supplementary material.
5 CONCLUSION
We proposed a new semantic embedding model (SEM) for handling the multi-label learning task. A framework based on Gauss-Siedel mini-batched adaptive gradient descent was proposed for efficiently solving the non-convex optimization problem required to learn the SEM parameters. For large label sets, we proposed fitting the SEM to marginal distributions rather than the full label distribution. A series of experiments on eight real-world datasets empirically demonstrated that the proposed method is superior to state-of-the-art methods in terms of prediction performance and running time.
A. Effect of Latent Space Dimensionality
It can be seen that the latent space dimensionality r plays an important role to learn latent factors V and a feature mapping matrix W in our proposed methods, as it does in the three baselines BMLPL, REmbed and SLEEC. In order to investigate this dependence, we conducted a series of experiments on the training data sets using 5-fold cross-validation, comparing BMLPL, REmbed, SLEEC and our proposed SEM and SEM-K.
In this experiment, we take Delicious dataset as an example. The training data is separated into five folds where four folds are used as training and one fold as validating, and the averaged results in terms of P@1 and MiF1 are given by Figure 3. It can be seen that their performances usually improve with increasing r until they reach an optimum value. However, once r becomes too large, their performances degrade. This is reasonable: when r is too small, the learned parameters cannot fully characterize the hidden semantic structure in the classification problem, while when r is too large, the benefits of dimensionality reduction are lost, as the model begins to over-fit to the idiosyncrasies of the training data rather than capturing the semantic structure common to both the training and validation data. Usually, these methods could obtain good performance at small r, say 45 for Delicious dataset.
B. Effect of training data size
Meanwhile, we studied the label prediction performance as a function of the amount of labeled training data. In this experiment, we fixed the testing data size, and randomly selected training data from the training set so that the training data size varies from 1% to 70% of the combined training and testing data. In order to avoid the presence of empty categories and instances with no labels, at least one instance is kept for each label and at least one label is kept for each instance during this sampling process. For each fixed size of the training set, the desired amount of data is randomly
sampled ten times, and the resulting average P@1 and MiF1 on the testing data are recorded. During training, the latent dimensionality parameter r is selected via 5-fold cross-validation.
Figure 4 shows these results for the Mediamill dataset which contains the largest number of instances. As expected, the performance of all the methods is positive correlated with the size of the training data set, and we also see that the proposed SEM-K uniformly outperforms the other methods regardless of the training data size. As it is often expensive to obtain large labeled data sets in real applications, this observation suggests that SEM-K is a better choice for these situations.
C. Convergence
In order to demonstrated the convergence of the proposed method, we show the value of objective function (4) (at r = 45) via Figure 5(a) and the prediction result (P@1) via Figure 5(b) along with the number of passes to the dataset (i.e., τ in Algorithm 1). It can be seen that SEM could be convergent and the prediction performance becomes stable in less than 50 epochs, which will leverage SEM dealing with large-scale data.

This paper proposes SEM, a simple large-size multilabel learning algorithm which models the probability of each label as softmax(sigmoid(W^T X) + b), so a one-layer hidden network. This in and of itself is not novel, nor is the idea of optimizing this by adagrad. Though it's weird that the paper explicitly derives the gradient and suggests doing alternating adagrad steps instead of the more standard adagrad steps; it's unclear whether this matters at all for performance. The main trick responsible for increasing the efficiency of this model is the candidate label sampling, which is done in a relatively standard way by sampling labels proportionally to their frequency in the dataset.
Given that neither the model nor the training strategy is novel, it's surprising that the results are better than the state-of-the-art in quality and efficiency (though non-asymptotic efficiency claims are always questionable since implementation effort trades off fairly well against performance). I feel like this paper doesn't quite meet the bar.


SENTATION USING ROBOTIC PRIORS
Our understanding of the world depends highly on how we represent it. Using background knowledge about its complex underlying physical rules, our brain can produce intuitive and simplified representations which it can easily use to solve problems. The approach of this paper aims to reproduce this simplification process using a neural network to produce a simple low dimensional state representation of the world from images acquired by a robot. As proposed in (Jonschkowski & Brock, 2015), we train the neural network in an unsupervised way, using the a priori knowledge we have about the world as loss functions called ”robotic priors” that we implemented through a siamese network. This approach has been used to learn a one dimension representation of a Baxter head position from raw images. The experiment resulted in a 97,7% correlation between the learned representation and the ground truth, and show that relevant visual features form the environment are learned.
Our understanding of the world depends highly on how we represent it. Using background knowledge about its complex underlying physical rules, our brain can produce intuitive and simplified representations which it can easily use to solve problems. The approach of this paper aims to reproduce this simplification process using a neural network to produce a simple low dimensional state representation of the world from images acquired by a robot. As proposed in (Jonschkowski & Brock, 2015), we train the neural network in an unsupervised way, using the a priori knowledge we have about the world as loss functions called ”robotic priors” that we implemented through a siamese network. This approach has been used to learn a one dimension representation of a Baxter head position from raw images. The experiment resulted in a 97,7% correlation between the learned representation and the ground truth, and show that relevant visual features form the environment are learned.
1 INTRODUCTION
The environment we live in is ruled by complex physical laws. However humans are likely to interact with it without any detailed knowledge of these laws. The human brain constructs simple models of the world in order to come up with an easy, though approximate, understanding of it.
This paper aims to reproduce this behavior for robots. We want to build a simple representation of the world that retains enough information to make a machine able to use it to interact afterwards i.e. to perform an assigned task. Finding such a minimal representation (e.g., the position of an object extracted from an image) is the standard way to implement behaviors in robots. However, this is most of the time done in a task specific and supervised way. In this paper, we want to learn such representation without supervision, based on generic learning objectives.
This representation is learned using deep learning. The deep neural network is trained by using images of robot experiences in a given environment and has to estimates for each image a state which is the representation we want to learn. Instead of using a ground truth for supervised training, we make use of an approach that ensures consistency between the states representation. For this purpose, the states are constrained by “robotics priors” (Jonschkowski & Brock, 2015) which are an expression of the knowledge we have about physics.
The main contribution of this paper is the use of the robotics prior approach in a siamese network to train a deep convolutional neural network. The network is trained with images of the robot environment, information on the actions performed by the robot and rewards defining a task. The neural network learns a state representation usable for the robot to perform this task. The resulting neural network also displays useful feature detectors for environment analysis that could be a basis for transfer learning to similar tasks.
2 RELATED WORK
• Robotic Priors The term of prior in Bayesian statistics refers to the prior probability distribution but like in the article from (Bengio et al., 2013), (Jonschkowski & Brock, 2014) and (Jonschkowski & Brock, 2015) we use this word as a reference to an a priori knowledge we have and not to a probability distribution. This knowledge comes from various domains which define several kind of priors : Task-Specific, Generic, and Robotic Priors. (Bengio et al., 2013) state that the key to successful state representation learning is the use of “many general priors about world around us” (cited by (Jonschkowski & Brock, 2015)). As noticed by (Jonschkowski & Brock, 2015), (Bengio et al., 2013) “proposed a list of generic priors for artificial intelligence and argue that refining this list and incorporating it into a method for representation learning will bring us closer to artificial intelligence ” For these authors however those generic priors are too weak to be used in the robotics fields and stronger ones have to be defined to achieve an efficient learning : the robotics priors used in this paper. Those priors, in a way similar to the approach of (Scholz et al., 2014), are physically grounded, which means that they aim at building a representation of the world consistent with physics.
• state representation The goal of state representation learning is to find a mapping from a set of observations to a set of states that makes it possible to describe an environment at a given time with enough information to fulfill a given task. In our approach we impose a dimension of the state and use the priors to guide the neural network in learning task specific states representation in this given dimension. This is an alternative approach to selecting a state representation from a set (Seijen et al., 2014) and (Konidaris & Barto, 2009) or creating a auto-encoder to compress the information into a low dimension state (Lange et al.) ,(Watter et al., 2015), (Finn et al., 2016) and (van Hoof et al., 2016).
• Unsupervised learning Using priors with neural networks is an unsupervised way for training a neural network. This approach is a different, but similar from energy based methods (Lecun et al., 2006), auto-encoder or denoising auto-encoder (Vincent et al., 2010) to train a deep neural network. The training process does not use directly energy functions but more specific functions in order to have targeted representation of task relevant parameters. Using unsupervised learning for training deep neural network may, according to (Bengio, 2009), be efficient because when the neural network does not have information about what to learn precisely and what future learning tasks are “it would appear very rational to collect and integrate as much information as possible about this world so as to learn what makes it tick”. On the other hand this way of training reduces overfitting risks.
• Model Architecture The method used for this paper involves a convolutional network whose architecture is inspired by (Krizhevsky et al., 2012). It makes it possible to create easy to train deep networks. Furthermore convnets are easier to use for training state visualization than GoogleNet (Szegedy et al., 2014) or ResNet (He et al., 2015) architecture. This architecture is coupled with Siamese networks like in (Chopra et al., 2005) or (Xing et al., 2003). Siamese network are used in this paper to impose constraints on learned representation in the implementation of robotic priors.
3 STATE REPRESENTATION LEARNING
The first challenge of state representation is to define which parameters are sufficient to characterize the state of the entire environment. For example, in a visually rich environment where only one object moves through time, the environment description depends only on the object position, which happens to be the only relevant parameter. The challenge is thus to learn what are the various relevant parameters. A second challenge is to find which of those parameters are truly interesting. For this we exploit a reward function. This function will give rewards for given states of the environment according to a defined task. This function gives the learning process a way to know which parameters
are relevant to the assigned task and which are not. The neural network has finally to map the relevant parameters into a state representation of a given dimension. To evaluate this representation there are two main possibilities:
• evaluating if the representation is compatible with a ground truth • determining if a reinforcement learning algorithm can use this learned representation to
learn to perform the assigned task (Jonschkowski & Brock, 2015)
While the second approach is more objective, the first one is simpler and we use it in this paper as a proof of concept using the correlation (Eq : 1) between learned representation and a ground truth.
Corr(s, ŝ) = E[(s−E[s])(ŝ−E[ŝ])]
σsσŝ , (1)
4 METHOD
4.1 ROBOTIC PRIORS
Robotic priors are used to provide the model to train with basic knowledge about the environment physical features. They add constraints to make the learned representation altogether consistent with simple physical and task specific rules. Each prior is formalized by a cost function implemented through a siamese network. By minimizing them, the model is trained according to the prior and can learn task-specific representation. The four priors we used are the one presented in (Jonschkowski & Brock, 2015). We will use the following notations:
• I(t) is the image perceived at time t • s(t) is the state at time t and ŝ(t) is its estimation.
• φ is a function which to an image I(t) returns a state s(t). φ̂ is its estimation • r(t) is the reward at time t • a(t) is the action done a time t • D is the input data (images, actions, rewards) • ∆s(t) = s(t+ 1)− s(t)
The definitions of loss functions associated to priors and the attached assumption are as follows:
Temporal coherence Prior: Two states close to each other in time are also close to each other in the state representation space.
LTemp(D, φ̂) = E[‖ ∆ŝt ‖2] , (2)
Proportionality Prior: Two identical actions should give two proportional state variations.
LProp(D, φ̂) = E[(‖ ∆ŝt2 ‖ − ‖ ∆ŝt1 ‖)2|at1 = at2 ] , (3)
Repeatability Prior: Two identical actions should give similar state variations.
LRep(D, φ̂) = E[e −‖ŝt2−ŝt1‖
2
‖ ∆ŝt2 −∆ŝt1 ‖2| at1 = at2 ] , (4)
Causality Prior: Two states on which the same action gives two different rewards should not be close to each other in the state representation space.
LCaus(D, φ̂) = E[e −‖ŝt2−ŝt1‖
2
| at1 = at2 , rt1+1 6= rt2+1] , (5)
This last prior is the only one giving information about the task and helps discovering the underlying factors which give a reward.
4.2 SIAMESE NETWORKS
The training using the priors needs the simultaneous estimation of several states to perform the optimization process. At this end our approach uses siamese networks. They are neural networks which share all their parameters. With this method the cost functions can be applied on several states computed at the same time. Those cost functions requires to choose the right images as input for siamese networks. For example for applying temporal prior, two input images are chosen following each other in time. Two siamese networks are used to compute the state estimation for each image and applying the temporal cost function. The backward propagation can then be done by computing gradient based on the loss function. Another example would be to use proportionality prior in which two state variations are estimated with the state of two images. Therefore four images and four siamese networks are needed to compute the proportionality cost function. The figure 1 shows the global network architecture. An image is input on each network and the application of the cost function is used on all the outputs. The layers are shared among all the siamese networks.
4.3 MODEL ARCHITECTURE
The architecture of the network uses four stacked convolution layers with 3*3 filters (inspired by (Simonyan & Zisserman, 2014)) with for each convolution layer, batch normalization, ReLu activation and 2*2 maxpooling (in that order). The convolution layers have 32-64-128-256 filters/Layer respectively. On the top of the network, there are two stacked fully connected layer (500 neurons and then one neurons for one dimension output). Between the last convolution layer with 3*3 filters and the fully connected layers, We insert a convolution layer with one 1*1 filters. This architecture helps the network to choose which feature map it wants to use to build its representation. This leads to a reduction of parameters in the fully connected layer by a factor of 256.
We use batch Normalization (Ioffe & Szegedy, 2015) for training, which helps to keep reasonable internal values and to make the training possible. In the experiment we make without batch normalization, the network is unable to learn the representation. Relu is used for fast learning and increasing the sparsity of neuron activation.
4.4 DATA
The data we used for training is a set of RGB images 200 pixels * 200 pixels which come from simulation. Those images are taken by the head camera of a Baxter robot. Images thus represent the front view of the robot and what it is able to see with its camera. The inputs images are normalized with 0 mean and 1 standard deviation before use, but for training we also add data augmentation to the images in order to improve robustness to noise and luminosity variations. To make the training
resistant to those perturbation, we add a random color filter to training sample like in (Krizhevsky et al., 2012) weighted with a Gaussian with random parameters. This transformation aims to make the representation invariant to the luminosity variation. We also add noise with the same mean and standard deviation than images. Those two data augmentation are added online during the training process. This method force the neural network to learn a lot of feature detector to make its representation robust.
4.5 TRAINING
The training has been done with Adagrad Duchi et al. (2011) with the following hyper-parameters: - Batch Size : 12 - Learning rate : 0.001 - weight Decay : 0 - Epoch :200 - iterations/Epoch : 10 The training is very fast in comparison with imageNet (Deng et al., 2009) classification training for example. Our understanding of this behavior is that the network does not need as many high level feature detectors as in classification because it is specialized in only one environment. A training without data augmentation is done for bootstrapping the neural network before the training with data augmentation.
5 EXPERIMENT
5.1 TASK AND ENVIRONMENT DESCRIPTION
This environment is produced by a simulation of a Baxter robot developed with the gazebo software. The robot is in front of a table (figure 2). The objective is to produce a representation that will make it possible to control the head joint position. The images are taken when the robot moves its head from right to left or left to right, a unit reward is obtained when the head is at maximum left or maximum right. In this context actions are defined by movement of the head between t and t + 1. The representation constructed by the neural network is in one dimension and should be consistent with the actual head position which is used as ground truth. The dataset based on the image generated is a set of 27 chronological series of images. Each series is composed of approximately 75 images. For each series the robot arms has a different position but only the state of the head joint change trough time. We know for each image at what time it has been taken and if the actual state gives a reward. We also know which action has been made between image at time t and image at time t+1. The actions are “move left” and “move right” with a certain angle. With these actions we can find pairs of images with same angle variation therefore compatible with proportionality and repeatability priors. Furthermore, with reward information we can find which images generate a reward with which action. Those images are then gathered to constitute an image set compatible with the causality cost function.
During the training process, the sets of images compatible with a certain prior are randomly sampled for training inside 26 of the 27 series. The last series is used for validation.
5.2 LEARNING PROCESS
The training process is done by minimizing each of the cost functions but those functions are in conflict to impose their constraint. The result of the training is an equilibrium between cost functions minimization. The result of the sum of the priors costs is presented on 3 which shows the decreasing of the global cost. The value of each cost functions separately is on figure 4. It is not surprising that all the cost functions are not minimized the same way. For example, the temporal function aims at minimizing the distance between representations when causality aims at maximizing it.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
5.3 RESULT
The resulting state representation of the head position for the validation data, after training with the deep model presented above and the results after training with 1 fully connected layer model similar to the one used in (Jonschkowski & Brock, 2015) are on figure 5. The correlation computed between state representation learned for both models and ground truth are in table 1. Those results show that both models are able to learn a good state representation of the Baxter head position in the case where no noise is present. However, table 1 shows that the deep neural network is much more robust to noise and luminosity perturbation than the one-layer-network. The evolution at each epoch of the correlation during bootstrapping and training are in figure 6.
Beside the performance gain, our deep model makes it possible to learn relevant visual features that could be interesting for other tasks in a transfer learning scenario. For example, the left image of figure 7 shows that a button of the environment has been learned to be a good feature for the current task, but could obviously be used in other scenarios.
Regarding these features, using data augmentation makes it possible to train the neural network to use a larger part of the image. To illustrate this assumption we train the network with and without data augmentation to compare the activation on the last convolution layer. Those activation are shown on figure 7. We can see that the training without data augmentation makes the neural network use only the position of the blue button of the image when with data augmentation the neural network use much more pieces of information such as the table top border. Furthermore, Table 2 shows that training with data augmentation slightly increases the performance of the deep neural network.
6 DISCUSSION
This approach makes it possible to train a deep neural network to learn specialized feature detectors used to build state representation in an unsupervised way. Those trained feature detectors could be used or transferred for learning another state representation in this environment.
In the reported preliminary experiments, the simulation environment is not as rich as the real world, therefore the variability of input image is low. However, this approach should work with real images in order to make the neural network learn more specialized feature detectors. It will be tested in further experiments.
A limitation of our approach is the assessment of the the training quality. In the case presented in this paper, correlation between state representation and ground truth is a possible measurement of the training quality. On the other hand, had we tried to learn a representation in higher dimension, the correlation could have been irrelevant. Furthermore if the process is applied to a situation where ground truth is unavailable, the correlation cannot be measured. A possible method would be to use a reinforcement learning algorithm to measure if the learned representation is suitable to the task like in (Jonschkowski & Brock, 2015).
7 CONCLUSION
This approach provides us with evidences that a deep network trained by the method of robotics priors can learn state representations. This technique makes it possible to learn a one dimension representation and furthermore to train a network to be robust to both noise and luminosity perturbations. The next step to be done will be to learn more complex representations like objects positions in three dimensions, and to use those representations within a reinforcement learning process to check if a robot can use the learned representations to perform various tasks. The use of real image for training is also one of our goals.
ACKNOWLEDGMENTS
The authors would like to thanks Clement Masson for fruitful discussion and his help in generating the data used in this paper. This work is supported by the DREAM project1 through the European Union Horizon 2020 research and innovation program under grant agreement No 640891.

This paper implements the method of Jonschkowski & Brock to learn a low-dimensional state representation represented as the last layer of a neural network. The experiments apply the method for learning a one-dimensional state representation of a simulated robot’s head position from synthetic images.
Learning state representations is an active and useful area of research for learning representations in interactive domains such as robotics. However, there seems to be no novelty in the method, over Jonschkowki & Brock. The primary contribution is the experimental evaluation performed on one task, where the paper evaluates the correlation between the learned state representation and the ideal state representation for the task (which is the robot’s head position).
As acknowledged by the authors, the experiments are very preliminary, only showing one simple task with a one-dimensional learned representation and a two-dimensional discrete action space. To make the experiments compelling, there need to be comparisons to prior methods such as Lange et al. ’12, Watter et al. NIPS ’15, and Finn et al. ICRA ’16 which also learn state representations from raw images. PCA on the images would also be a useful comparison, especially for simple tasks. Without these comparisons, it is impossible to evaluate the effectiveness of the method.
Lastly, as mentioned in the pre-review questions, the related work should include a discussion of other state representation learning methods such as Watter et al. NIPS ’15, Finn et al. ICRA ’16, and van Hoof et al. IROS ’16.
In summary, this paper lacks novelty and significance, as the paper implements an existing method and demonstrates results on only one simple task. Without comparisons, the results are impossible to interpret. More challenging tasks and experimental comparisons would significantly improve the paper. Additionally, this paper does not introduce any novel contributions to state representation learning for solving challenges in this domain. One pro is that the paper is generally written clearly.