We explore learning web-based tasks from a human teacher through natural language explanations and a single demonstration. Our approach investigates a new direction for semantic parsing that models explaining a demonstration in a context, rather than mapping explanations to demonstrations. By leveraging the idea of inverse semantics from program synthesis to reason backwards from observed demonstrations, we ensure that all considered interpretations are consistent with executable actions in any context, thus simplifying the problem of search over logical forms. We present a dataset of explanations paired with demonstrations for web-based tasks. Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average), and are competitive with simple exploration-and-demonstration based methods, while requiring no exploration of the environment. In learning to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interesting example of pragmatic language acquisition without any linguistic annotation. People routinely perform repetitive web-based tasks, involving sequences of clicking and typing actions. These include activities such as forwarding emails, booking flight tickets, ordering pizza, etc. These activities largely consist of small sequences of actions in an environment with restricted semantics, and are potentially amenable to automation. In this work, we explore whether an AI agent can be taught such tasks through natural language explanations and a single demonstration by a user (as one might teach such a task to a human assistant). Figure 1 : AI assistants that can be taught web-based procedures by their users can have diverse practical applications. Here, we explore learning very simple tasks from the Mini World-of-Bits framework using natural language explanations and a single demonstration of the task From the perspective of language understanding, this involves challenges such as converting instructional language to actions, resolving ambiguities through pragmatics, and learning script-like behavior. The web domain is rich in textual, structural and spatial features, allowing for exploration of multiple types of grounding behavior including spatial and visual language understanding, as well as reasoning over semi-structured data. Also, despite its richness, the tasks involved usually do not require much background knowledge. From a practical perspective, teachable AI assistants can change the way people interact with computers. Today's conversational assistants such as Alexa or Cortana act on a small number of preprogrammed language commands (e.g., What is the weather going to be like?). However, they cannot be taught new functionalities important to a user (as in Figure 1). Enabling users to teach computers personalized procedures through explained demonstrations can make conversational AI sys-tems fundamentally more useful. In Section 2, we situate our work in the broader body of work on grounded semantic parsing and learning from language. Section 3 summarizes our framework and dataset. In Section 4, we describe our approach in detail. Here, we investigate a new paradigm for interpreting language in grounded contexts. Instead of mapping statements to logical forms that then execute in a context as in traditional semantic parsing, the method considers the set of possible typing and clicking actions in a context, identifies features of corresponding web elements and their relationships with other elements on the webpage, and aligns these to natural language explanations through a generative model. Section 5 describes the empirical evaluation. Our contributions are:  An approach towards learning web-based tasks from a single explained demonstration. A dataset of explanations and demonstrations for tasks from the MiniWoB framework. Empirical results showing that explained demonstrations can be an effective mode of supervision for learning such tasks. Language can significantly reduce the number of samples needed compared to learning from demonstrations alone. Semantic Parsing: Supervised models for converting statements to logical forms have long been studied in a wide range of settings (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Wong and Mooney, 2007; Kwiatkowksi et al., 2010; Yin and Neubig, 2017). More recent approaches focused on using weaker forms of supervision such as denotations or observations of world state (Berant et al., 2013; Clarke et al., 2010; Krishnamurthy and Mitchell, 2012) and semi-supervised methods aimed at efficient prototyping (Pasupat and Liang, 2015; Wang et al., 2015). These methods require more readily available supervision, such as ques-tion/answer pairs for model training, rather than annotations of logical forms. (Artzi and Zettlemoyer, 2013) learn to follow instructions in the context of robot navigation by conditioning parsing on environmental context. Artzi and Zettlemoyer (2011) use conversational feedback as a signal to induce logical forms for individual utterances from transcripts of conversations in a dialog-based setup. Some other recent approaches (Long et al., 2016; Guu et al., 2017) explore learning language from sequences of utterances and interactions in simple environments, which is conceptually similar to our work. Muhlgay et al. (2019) and Guu et al. (2017) explore better strategies to search the space of logical forms. While all of these methods are related to multiple facets of work, our method diverges from them in that the space of candidate logical forms is driven by the constraints of possible actions in an environment rather than the natural language utterance. This guarantees that all of the considered logical forms during search are consistent with executable actions in any novel context. Finally, some recent methods (Andreas et al., 2016) marginalize over latent interpretations of language in context of downstream tasks. We use a similar Bayesian approach, where actions are chosen by marginal-izing logical forms (rather than choosing a single interpretation of an explanation). Interactive Learning from Language: Several frameworks have leveraged natural language supervision to learn new tasks, starting with early work on the SHRLDU system (Winograd, 1972) and Interactive Task Learning (Laird et al., 2017). In particular, several reinforcement learning approaches have been explored in text-based environments for learning strategies, following instruction manuals, game playing, etc. (Branavan et al., 2009; Goldwasser and Roth, 2014; Misra et al., 2018; Narasimhan et al., 2015). These approaches leverage the ability to explore and interact with the environment to learning policies that lead to favourable outcomes. This is different from our goal here, where the agent needs to learn from a single explained demonstration of a task, and no interactivity with the environment is assumed. Some recent approaches have shown language explanations to be effective for learning realistic tasks including relation extraction, concept learning and question answering (Hancock et al., 2018; Srivastava et al., 2017, 2018; Andreas et al., 2018). In terms of the goal and problem formulation, our approach extends multiples lines of previous work. Quirk et al. (2015)'s work is similar to ours in motivation in learning user-specified recipes, but has no aspects of grounding or demonstra-tions.(Wang et al., 2016) explore interactive parser training through language games in context of block-world environments. Pasupat et al. (2018) explore mapping natural language to specific elements on complex and realistic web-pages, although not in context of learning from demonstrations. Our framework directly extends previous work on learning web-based tasks from the Mini Figure 2 : Crowd-worker interface used for collecting natural language explanations and demonstrations Word-of-Bits framework using multiple demonstrations and exploration of the environment (Shi et al., 2017; Liu et al., 2018). In particular, our DSL extends the constraint language defined in Liu et al. (2018) to explore learning from explained demonstrations instead. We build on the Mini World-of-bits (MiniWoB) framework (Shi et al., 2017), a collection of web-based tasks initially proposed as a testbed for reinforcement learning agents. The tasks vary in dif-ficulty in terms of the number of actions required, variability between instances of the task, and types of reasoning involved (including clicking specified buttons, forwarding emails and playing tic-tac-toe). See the top half of Figure 2 for an example of a task. Each task consists of a task description (yel-low box), and an interactive web interface. While previous methods have focused on learning sequential decision making to complete these tasks through a mixture of exploration (the framework provides simulators, where correctly completing a task yields a reward) and behavior cloning (by observing multiple demonstrations from human users); our focus is on learning to complete these tasks in a one-shot sense (without any exploration). This is because the one-shot case is a much more realistic scenario for learning web-based procedures from a teacher. In practical situations (where there are no simulators), it would not be feasible for an AI agent to learn to book flights by booking multiple incorrect tickets, or manage a user's email by sending multiple incorrect emails. On the other hand, a paradigm where the agent attempts to generalize from a single demonstration and explanations can be feasible for many more of such scenarios. Figure 3 : Examples of collected explanations We created a dataset of natural language explanations paired with demonstrations by human users for tasks from the MiniWoB framework. For this, crowdsourced workers on Amazon Mechanical Turk were asked to demonstrate how to complete these tasks and provide stepwise explanations to an AI assistant on how to complete the task. Since users would be unfamiliar with most of the tasks, for each task they were allowed to experiment with the interface as many times as they liked, and only the final demonstration was logged. In all, we collect 520 demonstrations (each consisting of a sequence of click/type actions in the context of a MiniWoB task) paired with stepwise explanation sequences. Figure 3 shows samples of collected explanations. On average, each explanation sequence contains 3.3 explanations. The dataset contains 1719 explanations in total (indi-vidual steps), averaging 8.4 words per explanation. The size of the vocabulary of the explanations is 995. In general, workers found the teaching process to be engaging, with an average rating of 8.3 on a 1-10 scale on how they enjoyed the HIT in a post-completion survey. The dataset is available at https://aka.ms/Web-D-E . Data characteristics: From a manual analysis of 100 randomly selected explanation sequences and task demonstrations, we find that in almost all cases (97%), the sequence of actions described in the explanations corresponds to the sequence of actions in the demonstration. More than 85% of explanations mention a clicking or typing action, while around 10% identify an entity/string on the webpage that is used in an action in the next step (e.g., the first explanation for the second task in Figure 3). Around 3% of the explanations correspond to conditionals and hypotheticals, which go beyond the scope of our approach. Roughly 15% of the explanations mention multiple entities on the webpage  usually specifying one element in relation to the other (e.g., the radio button to the right of the text-box ). Table 1 : Major operators in DSL for learning of web-based procedures 3.2 DSL for semantic parsing We define a domain specific language (DSL) for describing web-based procedures in terms of DOM elements by expanding on the constraint language in Liu et al. (2018). The DSL operators correspond to actions on DOM elements, element features and relations between them. The DSL defines the vocabulary of logical forms for parsing of user explanations, and grounds sensors and effectors in the web environment. Table 1 summarizes the DSL. There are three types of operations: (1) click and type actions on specified web elements (with a specified string, in case of a type action), (2) operations that filter elements on a page that satisfy a criterion, and (3) operations that filter strings based on a criterion. We include a special operator FindMatchingContext to accommodate cases in which the users provide explanations for an instance of a task with specific arguments mentioned in the task description (e.g., see the last row in Table 1). In this case, the operator can pick out the corresponding argument for the new instance by looking at the surrounding context in the new task description. The evaluation of logical forms in the DSL in the context of a webpage consists of set operations over all DOM elements on the webpage (and text-spans of up to two tokens for string operators). For example, the logical form HasTag(type=button) will evaluate to the set of elements on a page that have a HTML tag type with value button . Our approach for learning web-based tasks, which we call LED  for Learning from Explained Demon- Figure 4 : Modeling principle for Learning from Explained Demonstrations (LED). We prefer logical forms ( l ) that are both consistent with the user demonstration ( d ) in the context ( c ), and relevant to the user's explanations ( x ). strations , models the process of explaining a demonstration of a task in a grounded context. We assume that the reasoning behind each action in a demonstration can be described by a logical form, l , in the DSL. 1 LED 's essential idea is that preferred logical forms are both (1) consistent with the user demonstration, d , in the observed context, c , and (2) relevant to the user's language explanations, x . Figure 4 illustrates this for a toy-example, where the context consists of a web-page with three elements, the demonstration consists of a single action, and a corresponding explanation is provided. Based on the observed demonstration (that elem3 was clicked), it is hard to infer the reason behind clicking it. Multiple logical forms in the DSL can be consistent with clicking elem3 in this context. e.g., it is at the top of the page, its color is blue, etc. However, these interpretations would not justify the provided explanation as those logical forms are not relevant to the explanation. Modeling relevance between logical forms and explanations can help identify the reasoning behind user demonstrations. cal forms l (e.g., database queries), which are then are executed against a context c (e.g., a knowledge base) to get a denotation (corresponds here with a demonstration) d . i.e., d = (cid:74) l ( x ) (cid:75) c . In this model-theoretic view of semantics, parsed logical forms are not informed by the environmental context until execution. In comparison, LED roots logical forms in the observed context, and thus pragmatic consistency is ensured by design. 2 We maximize the log-likelihood of observing the explanations given the demonstration in a grounded context: L (  ) = log p ( x | d, c ) = log (cid:80) l p ( x | l ) (cid:124) (cid:123)(cid:122) (cid:125) relevance p ( l | d, c ) (cid:124) (cid:123)(cid:122) (cid:125) consistency (1) Here, the first term corresponds to scoring relevance between logical forms and explanations (modeled using a semantic parsing model). 4.1 Grounded Logical forms as latent variables Eqn 1 marginalizes over latent logical forms. The second term enforces consistency between candidate logical forms and the demonstration in the context, and can be deterministically evaluated. As we see in Section 4.2, consistency is enforced by temperature-based annealing during training. To make this tractable, we represent a logical form in a grounded context as an assignment of a tuple of discrete variables, l := ( e 0 , f 0 , r, e 1 , f 1 , a, t, f t ) . These variables indicate things such as which DOM element is acted upon ( e 0 ), if its relation ( r ) with another element on the page ( e 1 ) is relevant, and so on. These are defined below. e 0  domElements ( c ) denotes the DOM-element on which an action is performed. (e.g., e 0 = elem 3 in Fig 4) This is observed from the demonstration, thus p ( e 0 ) = I e 0 = e observed . 3  f 0 = ( f 01 . . . f 0 n F ) is a set of selector variables, where f 0 i denotes if feature i of element e 0 is relevant for choosing it. Its domain is {   F i } , where F i is the range of values feature i can take. f 0 i =  denotes that the feature was not relevant for choosing e 0 (e.g., f 0 color =  in Fig 4). If f 0 i (cid:54) =  , it can only take the observed value of the feature for e 0 in the context (e.g., f 0 tag = square in Fig 4). In Table 1, these correspond to operators that return web-elements and have names with prefix Has . 2 For example, in Figure 4, click(tag=triangle & rightOf(square)) won't be considered for the provided utterance, as it is inconsistent with the context. 3 I condition denotes an indicator function for condition . r denotes if relation r between e 0 and another element on the webpage is relevant for choosing it. Its domain is {   R} , where R is the set of (binary) relations between elements in the DSL. In Table 1, these are operators that have names with prefix Reln . r =  denotes that the no relation was relevant for choosing e 0 . If r (cid:54) =  , it can only take the value of a relation that exists between e 0 and another element. (e.g., in Fig 4, r can't take the value LeftOf , since elem 3 is the rightmost element in the context). Our choice of having a single variable for r disallows logical forms with multiple or nested relations. This was guided by an analysis of our dataset, where none of the collected explanations show such behavior. e 1 denotes that relation r between elements e 0 and e 1 is relevant for choosing e 0 . Its domain is {   domElements ( c ) } . e 1 =  if and only if r =  , i.e. if no relation is relevant for choosing e 0 . If r = reln , e 1 can only take values of elements such that reln ( e 0 , e 1 ) is true in the context. f 1 = ( f 11 . . . f 1 n F ) is a set of selector variables, where f 1 i denotes if feature i of element e 1 is relevant. e.g., for click the checkbox next to the button that says submit' , the HasText feature of the button is relevant). f 1 i =  denotes that feature i was not relevant. If f 1 i (cid:54) =  , it can only take the observed value of the feature for e 1 . a denotes the action performed on e 0 (click or type). This is observed from the demonstration. t denotes the string to type, if a = type . This is observed from the demonstration (and is a substring of the task description text). f t = ( f t 1 . . . f tn T ) is a set of selector variables, where f tj denotes if the text feature j of t is relevant for choosing it (In Table 1, operators with a string return type correspond to text features). Inverse Semantics: Assignments of values to these variables represents a search in the DSL space, since given any context, there is a mapping a from logical forms to an assignment of these variables. A key idea here is that, borrowing from program synthesis, we can leverage the inverse semantics of operators in the DSL (Polozov and Gulwani, 2015) to guarantee consistency of logical forms with the grounded context. i.e., at any step, the space of candidate logical forms we consider is consistent with the observed demonstration. This is possible because in our case, computing the inverse semantics for all operators in the DSL is feasible. 4 4 Since there is only a relatively small number of candidates As just described, our approach will use the context of the webpage leverage DSL inverse semantics to maintain an implicit set of candidate logical forms that are consistent with the observed demonstration. We will use variational inference to infer the logical forms that are most relevant to the seen explanations, and choose the action to take based on the inferred distribution over logical forms. In Eqn 1, the second term corresponds to a prior probability overs logical forms given a demonstration and context (webpage). Our representation of logical forms as latent variable assignments (from Section 4.1) enables us to decompose this probability into local factor distributions. We choose these local priors to correspond to distributions that are uniform over assignments that are consistent, and has zero support otherwise, similar to previous work on pragmatic reasoning (Frank and Goodman, 2012; Monroe et al., 2017). In other words, these distributions are proportional to indicator function over valid assignments of variables in each factor. As seen below, these define a prior over l that is also proportional to a simple indicator function over values of l that are consistent with the observed demonstration and context. p ( l | d, c ) = p ( e 0 , f 0 , r, e 1 , f 1 , a, t, f t | d, c ) = p ( e 0 | d ) p ( f 0 | e 0 , c ) p ( e 1 , r | e 0 , c )  p ( f 1 | e 1 , c ) p ( a, t | d ) p ( f t | t, c )  IV alid ( e 0 ,d ) IV alid ( f 0 ,e 0 ,c ) IV alid ( e 1 ,r,e 0 ,c )  IV alid ( f 1 ,e 1 ,c ) IV alid ( a,t,d ) IV alid ( f t ,t,c ) = IV alid ( l,d,c ) (2) Substituting this in Eqn 1 and using Jensen's inequality, any distribution q over logical forms provides a lower-bound on the log-likelihood: L (  )  (cid:88) l q ( l ) log p ( x | l ) IV alid ( l ) q ( l ) = (cid:88) l q ( l ) (cid:0) log p ( x | l ) + log IV alid ( l ) (cid:1) + H q (3) where H q is the entropy for distribution q . In Sec 4.1, we represent l as a tuple of variables. Next, we make a mean field approximation by assuming the distribution q ( l ) decomposes as:  DOM elements or strings on the webpage  to search over. Compare this with an operation in arithmetic, e.g., add(int, int) , which might require a search over infinite co-domains. Parsing model: We assume that the probability of an explanation decomposes into the probability of individual words as log p  ( x | l ) = (cid:80) w  x log p ( w | f 0 , r, f 1 , f t , a ) . Further, we assume that individual words are generated from features, relations and actions in the logical form as: log p ( w | f 0 , r, f 1 , f t , a ) = log 1 C { f 0 ,r,f 1 , f t ,a } (cid:88) k p ( w | k ) z kw p ( z kw )  { f 0 ,r,f 1 , f t ,a } (cid:88) k b kw (cid:0) log p ( w | k, z kw ) + log p ( z kw ) (cid:1) + H b kw (5) Here, k is an index over values of f 0 , r , f 1 , f t and a . z kw denotes an alignment between a particular value of a feature, relation or action ( k ) and word w in the explanation, in which case the word is generated from the distribution p ( w | k ) . The presence of a summation inside of a logarithm makes maximizing this objective hard. We again use Jensen's inequality to get a bound by introducing variational distributions b kw over alignments z kw . b kw can be thought of as representing the proportions of an explanation word contributed by specific feature values, relations or actions k in the logical form. Each p ( w | k ) is parameterized as a multinomial distribution,  kw , over the vocabulary. Training and Inference: Our model training follows a variational EM approach, where in the E-step, we perform inference for the latent logical form variables and alignment proportions, keeping the model parameters as fixed. In the M-step, we update the parameters,  kw , taking the variational distributions and alignments as fixed. Combining Eqn 2, Eqn 3 and Eqn 5, we get: L (  )  (cid:88) l q f 0 q e 1 q r q f 1 q f t (cid:16)(cid:0) (cid:88) w (cid:88) k b kw [log  z kw kw + log p ( z kw )] + H b kw (cid:1) + log IV alid ( l,d,c ) (cid:17) + H f 0 + H e 1 + H r + H f 1 + H f t (6) Maximizing this objective w.r.t. the variational 5 Using q f 0 as shorthand notation for the product of variational distributions (cid:81) i q f 0 i , and so on. distributions yields the following E-step updates: 6 q f 0 i ( v f 0 i )  exp (cid:16) (cid:88) w b v f 0 i w log(  v f 0 i w ) + log IV alid ( v f 0 i , e 0 ,c ) (cid:17) q e 1 ( v e 1 )  exp (cid:16)(cid:0) (cid:88) f 1 i (cid:88) w b f 1 i w (cid:88) v f 1 i q f 1 i ( v f 1 i ) log  v f 1 iw + log IV alid ( v f 1 i ,e 1 ,c ) (cid:1) + (cid:0) (cid:88) v r q r ( v r ) (cid:88) w b v r w log  v r w + log IV alid ( e 1 ,v r ,e 0 ,c ) (cid:1)(cid:17) q r ( v r )  exp (cid:16) (cid:88) v e 1 q e 1 ( v e 1 ) (cid:0) (cid:88) w b v r w log  v r w (cid:1) + log IV alid ( e 1 ,v r ,e 0 ,c ) (cid:17) q f 1 i ( v f 1 i )  exp (cid:16) (cid:88) v e 1 q e 1 ( v e 1 ) (cid:0) (cid:88) w b v f 1 iw log  v f 1 iw (cid:1) + log IV alid ( v f 1 i ,e 1 ,c ) (cid:17) q f tj ( v f tj )  exp (cid:16) (cid:88) w b v ftj w log(  v ftj w ) + log IV alid ( v ftj ,t,c ) (cid:17) (7) Similarly, the updates for the alignment proportions (taking p ( z kw ) in Eqn 6 to be uniform) are: b kw  exp (cid:0) (cid:88) k q k ( k ) log  kw (cid:1) (8) LED(+syntax) : The above approach allows for arbitrary alignments between words and features, relations or actions in the grounded logical form ( k ), essentially representing x as a bag-of-words. 7 We also explore a variant that models x as a sequence of tokens by introducing a prior over joint alignments z kx = z k 1 w 1 . . . z k T w T in a sentence x := w 1 . . . w T (in Eqn 5). This is done by simply modeling p ( z kx ) with pairwise transition probabilities as p ( z kx ) := (cid:81) n p ( z k t | z k t  1 ) = (cid:81) n T k t ,k t  1 . In this case, updates for alignment proportions (Eqn. 8) correspond with emission probabilities in a HMM (which we omit here for brevity). Since the updates in Eqn 7 and Eqn 8 are cyclic, in each E-step, we make 20 iterations of updates to the variational distributions and alignment proportions in a round-robin schedule. We note that consistency is enforced during training by the log-of-indicator-variable terms in Eqn 7. This is because any inconsistent assignments get a score of log(0) , which tends to negative infinity. However, to ensure smooth training (and alleviate modeling issues from our mean field approximation), we leverage an annealing based strategy, where we incremen-6 The optimal value for the concave problem (cid:80) j x j log y j x j s.t. (cid:80) j x j = 1 is achieved when x  j  y j . 7 E.g., this won't differentiate between  click the URL below the button  and  click the button below the URL . tally increase the penalty for log(0) terms during training as  N/ 2 for the N 'th EM iteration (for large N , this also is a prohibitive penalty). In our experiments, this was seen to improve training. In the M-step, we maximize the objective w.r.t.  k :  k ( w )  exp (cid:16) (cid:88) n (cid:88) w  x n b nkw q k (cid:17) (9) The one exception is a special copy mechanism for string-valued features. For these,  kw is not learned, but simply corresponds to an indicator function denoting if w matches the value of the feature. e.g.,  HasText(submit') , submit' = 1 . First, we evaluate the method for completion rates on tasks from the MiniWoB framework. Following Liu et al. (2018), we filtered 40 tasks from the MiniWoB framework (Shi et al., 2017) that require only clicking and typing actions. During training of the LED model, we sample an explained demonstration for each of the 40 tasks, and models are trained on the aggregate of these (the model sees one explanation-demonstration pair for a task). For testing, models are evaluated on a new instance of a task, where the model greedily computes the demonstration d (specifying a click or typing action on a web element in the current DOM) that would maximize p ( x | d, c ) (see Eqn 1) and executes the corresponding actions. The method then moves to the next explanations. This requires an enumeration of all possibly clicking and typing actions that can be performed in a context c at every step. 8 Since the number of actions in a demonstration can be different from the number of steps in the explanation, we heuristically align the sequence of actions in demonstrations to the sequence of sentences in the explanations in our dataset based on a small manually defined list of trigger words. A direct comparison of LED with other approaches is not possible, since they differ considerably in the type of supervision and resources used. Nonetheless, here we compare LED 's performance with the following two methods to get a coarse sense of its effectiveness: 8 This is possible since the set of actionable elements on a webpage, and the set of candidate strings that can be typed (up to two length tokens from task description) are not large. Figure 5 : Task-completion rates for MiniWoB tasks with varying difficulty. Rates are calculated over 100 new instances of each task 1. SemParse : This is a supervised semantic parsing baseline, trained on a manually annotated dataset of around 300 explanations labeled with their DSL logical forms (covering roughly one annotated explanation sequence for every task). The model is based on a sequence-to-sequence neural semantic parser from Jia and Liang (2016). During testing, the method parses the sequence of explanations to logical forms, and sequentially (attempts to) executes the predicted logical forms. In contrast, LED requires no logical form annotations. However, it leverages the inverse semantics of the DSL operators, which may not be feasible for every DSL. 2. BC+RL : This is the original approach from Shi et al. (2017), who proposed the MiniWoB framework and consists of behavior cloning and exploration. This learns a task by supervised learning on about 200 demonstrations, followed by exploration via reinforcement learning to fine-tune the learned policies. In comparison, LED requires no exploration of the environment but leverages additional supervision in the form of natural language explanations. Multiple methods have since explored other RL-based approaches, resulting in much improved performance (Liu et al., 2018; Luo, 2019; Jia et al., 2019). In particular, Liu et al. (2018) leverage a constraint language similar to our DSL to train a RL policy to get large gains in performance. However, all these methods require multiple demonstrations and exploration of the environment. Figure 5 shows task completion performance for different methods on a subset of tasks from the MiniWoB framework. We compute task comple-Approach Action-prediction accuracy LED (+Syntax) 0.45 LED 0.43 SemParse 0.39 Random 0.28 Table 2 : Semantic parsing performance (predicted action match) for interpreting individual explanations in a context tion rates over 100 randomly selected test instances of each task. The differences between instances involves different arguments for a task and differences in the state of the environment. Firstly, we note that the LED approaches consistently outperforms SemParse across all tasks. This is a strong result, since LED does not have access to logical form annotations for explanations as SemParse does. This strongly indicates that knowledge of the pragmatic context is important for language interpretation in this domain, since our approach which roots logical forms in observed demonstrations performs better or as well for all but one task. We note that there is a large variance among tasks in terms of amenability to learning from explanations or exploration. For tasks like tic-tac-toe , explanation-based methods perform poorly as expected, since learning the game involves reasoning that is hard to explain through step-wise explanation of a demonstration, but can be more naturally learned from exploration. On the other hand, explanation-based methods perform well on tasks that are easily expressed through language. On the whole, the LED approaches and are roughly competitive with BC+RL , while requiring no exploration and only a single demonstration. Note that unlike exploration-based methods, LED and SemParse can potentially generalize to new tasks during testing (where no demonstration is seen during training) from explanations and context only. We also note that LED(+Syntax) generally outperforms vanilla LED , although the effect size is not large. However, this trend is statistically significant (binomial test, p < 0 . 1 ). Next, we quantitatively evaluate the parsing performance of our method at the level of individual explanations (rather than task completion rate). For this, we evaluate the trained models on explanations from a set of 80 demonstrations from the dataset (unseen during training), where we calculate the match between the predicted action from an explanation in the context, and the actual action in the logged demonstration (accuracy of preFigure 6 : Heatmap showing learned values of  kw for 20 frequent words w in and representative values of k . Darker shades correspond to higher probability values. dicted action in a context). Table 2 summarizes this performance, which shows a similar trend as Section 5.1. Both LED methods perform substantially better than SemParse , and all three methods perform much better than randomly choosing the next executable action in the context ( Random ). We note again that LED 's involves no logical form annotations, and is driven purely by grounding explanations in observed demonstrations. Figure 6 depicts the learned lexicon by visualizing a representative subset of learned  kw values for LED (+Syntax) (from Sec 4.2) as a heatmap. We note that the model correctly induces mappings between words and DSL operators. The rows and columns are manually ordered to emphasize the block diagonal structure. Table 3 shows the learned transition probabilities, T k 1 ,k 2 , for LED (+Syntax) . To reduce model size, we share parameters for values of k corresponding to types f 0 , r , f 1 , f t and a . A common template about the general structure of user explanations is reflected from the parameter values. Most explanations start with the description of the action a , followed by mentioning features that identify the relevant element f 0 . In fact, f 0 distributions generate the majority of words in most explanations. Relation mentions, when present, usually follow this, in turn followed by features corresponding to f 0 , reflective of a VSO word order in most explanations. Diagonal values are substantially higher, indicating that words describing specific objects and actions tend to cluster together, as would be expected from the semantics of natural language. From a qualitative error analysis, we note that most errors in task learning come from three sources. Firstly, although the method learns reasonable mappings between words and semantic operators, the method often misaligns attributes of different ele-a f 0 r f 1 f t a 0.12 0.70 0.07 0.04 0.07 f 0 0.05 0.82 0.08 0.04 0.01 r 0.01 0.12 0.57 0.26 0.04 f 1 0.03 0.07 0.22 0.63 0.04 f t 0.08 0.30 0.09 0.06 0.47 Table 3 : Learned transition probabilities between latent variable categories for LED (+Syntax) . These reflect a prominence of VSO sentence structures in user explanations. ments, even with the LED(+Syntax) model. This is likely because the training data is not adequate to learn these constraints, and methods that enforce these through informed priors maybe more effective. Another common error is due to challenges with anaphora resolution and discourse referents. Finally, a large number of explanations are not explicit in describing the sequence of actions required to perform a task, and some needed actions remain unmentioned. While this would be expected in realistic computer-human interactions, fixing these errors is beyond the scope of the current method. Our work here is a step in the direction of teachable AI agents that can learn new behavior from conversational interactions with ordinary users. In terms of technique, our bottom-up approach to generating logical forms ensures consistency between interpretations and the ambient context during search. Conversely, this would be complicated in domains with rich composition and nesting in logical forms, which go beyond simple features and relations. e.g.,  click the third email from Jeanette , and where modeling inverse semantics is infeasible. Here, we posed the learning of web-based tasks as similar to instruction-following problem, with no aspect of interactivity or exploration of the environment. In future work, the possibility of learning from a mix of explanations, exploration and a limited budget of interaction with the environment can be explored. Also, language grounding models that incorporate richer alignments between explanations and demonstrations can lead to more effective learning. Since LED only requires tok-enization as pre-processing, it can possibly extend to low resource scenarios. In terms of problem framing, interactive use-cases that enable the agent to ask questions when it is confused may also be realistic. Future work can also explore curriculum learning in this domain, by first learning simpler tasks, which can be compositionally invoked in explanations for complex tasks.
objective objective method method result abstain abstain abstain abstain abstain objective abstain objective abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain abstain objective objective objective objective other other other other other abstain other abstain other other method other other other objective other objective other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain other abstain method abstain method method abstain abstain method abstain abstain method method other other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective method abstain abstain method abstain abstain abstain abstain abstain