In classic instruction following, language like I'd like the JetBlue flight maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flightbooking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning). Language is a natural interface for systems like robots or personal assistants that interact with human users. One way to interpret language in these interactive settings is to train an instruction following agent: a model that learns to map commands like go three steps forward to the door to a sequence of actions in context (e.g., Brana-van et al. 2009; Tellex et al. 2011, inter alia ). Instructions describe how an agent should act in an immediate context, but to build models that can generalizecarrying out a user's goals in new contexts and learning user preferences over repeated interactionsagents should also infer why actions are taken. Grounding language to reward functions extends the standard instruction following setup in this way, representing the goals and preferences that underlie actions, and allowing agents to autonomously carry out correct actions in new contexts (e.g., Fu et al. 2019). However, when people interact with systems they often primarily aim to achieve specific tasks, Available flights: JetBlue Delta price stop length User's true reward function: rather than literally describing their preferences in full. How do we infer general goals and preferences from utterances in these settings? Consider a flight booking agent like the one in Figure 1. By inferring the user's reward function (indicating their preference for carrier, price, and other flight features) beyond just selecting the right flight, such a system would be able to autonomously book flights on behalf of the user in other instances. To do so, the system might use the actions the user commands as evidence about what they prefer, recovering rewards from actions using (language-free) techniques like inverse reinforcement learning (IRL; Ng and Russell 2000). For example, the system can select a flight the user might like in a new instance by 8546 matching features from their past flight bookings. The key idea of our work is that the way that a user refers to their desired actions with language also reveals important information about their reward: the fact that they said  the JetBlue flight  and not  the expensive flight  conveys what matters to them. Intuitively, in settings with repeated interactions, utterances are optimized to communicate information that is generalizableimplicitly helping listeners make useful inferences for acting on a longer horizon. We implement this idea with a pragmatic model of how speakers (humans) generate such language: speakers choose utterances that both elicit reward-maximizing actions in a particular context and faithfully describe the reward. Given an utterance, our model infers that the most likely rewards are the ones that would have made a speaker likely to choose that utterance. To evaluate our model, we construct and release a dataset for mapping language to rewards, FLIGHTPREF , containing natural language utterances from humans with underlying preferences. Humans interact in a multi-turn flight booking game similar to Figure 1, where we provide a user player with a reward function representing flight preferences. The goal of the game is for the user to communicate these preferences in natural language to an assis-tant player, who is tasked with booking preferred flights for the user. We present this dataset as a challenging benchmark for reward learning from language and interaction. In our experiments, we show that our model can infer reward functions from natural language, improve reward estimates consistently over repeated interactions, and use inferred rewards to accurately select optimal actions in held-out environments. Our full model obtains relative accuracy improvements of 12% when compared to models that only treat language as descriptions of actions. 1 2 Related Work Instruction following. A long line of work on grounded instruction following has developed various methods for producing actions from language, including approaches that use intermediary structured semantic representations (MacMahon et al., 2006; Tellex et al., 2011; Chen and Mooney, 2011; Matuszek et al., 2013; Artzi and Zettlemoyer, 2013; She et al., 2014; Thomason et al., 2015; Wang et al., 1 We release our code and dataset at https://github.com/ jlin816/rewards-from-language . 2016; Fried et al., 2018a; Arumugam et al., 2017; Suhr et al., 2018) or map directly to primitive actions (Branavan et al., 2009; Andreas and Klein, 2015; Mei et al., 2016; Bisk et al., 2016; Misra et al., 2017; Guu et al., 2017; Suhr and Artzi, 2018; Anderson et al., 2018; Shridhar et al., 2020). All of these approaches interpret any given utterance (in-struction) solely in the context that elicited the utterance, producing one particular sequence of actions. The method we present extends these approaches, using utterances to infer the rewards that underlie the actions that should be taken across a range of environments: both the context that elicited the utterance, and other unseen environments. Reward learning. The majority of work on reward learning has been in the robotics and reinforcement learning communities and has not incorporated language, rather using techniques such as inverse reinforcement learning (IRL; Ng and Russell 2000; Ratliff et al. 2006; Ziebart et al. 2008; Hadfield-Menell et al. 2017; Jeon et al. 2020) to infer the rewards that underlie human demonstrations of actions. Even works that incorporate language into reward learning also take this primarily action-centric approach: either by using datasets pairing utterances with trajectories and using (language-free) IRL to then recover reward functions from trajectories (MacGlashan et al., 2015; Fu et al., 2019), or learning an instruction-following model guided by a language-conditioned discriminator (Bahdanau et al., 2019). The language in these settings are unambiguous commands, giving a complete description of a goal (e.g., go to the red door). In contrast, we are concerned with language used to guide agents in repeated interactions (where language may be a partial or ambiguous mix of instructions and reward descriptions). Pragmatics. A long line of work on pragmatics (Grice, 1975), particularly in the Rational Speech Acts (RSA) framework (Goodman and Frank, 2016), has developed computational models for inferring the behavior or belief that a speaker wishes to induce in a listener. However, the majority of this work has only focused on single-turn interactions, where an utterance conveys an action in a single context, e.g., choosing the correct referent in signaling games (Golland et al., 2010; Frank and Goodman, 2012; Degen et al., 2013; Monroe et al., 2017; McDowell and Goodman, 2019), interpreting implicatures (Goodman and Stuhlmller, 8547 What reward would make the referenced trajectory optimal? 2013; Bergen et al., 2016), or generating (Fried et al., 2018a; Sumers et al., 2021) or interpreting grounded instructions (Fried et al., 2018b). Our work extends this past work by showing that in repeated interactions, listeners can also benefit by reasoning pragmatically about how speakers communicate information about and over longer time horizons. Problem Formulation. We parameterize the user's preference as a reward function r  with parameters  . In our flight booking domain from Figure 1,  is a weight vector which specifies preferences over flight features (carrier, price, etc.). We formalize the general reward inference problem as sequence of Markov decision processes (MDPs) M 1 , . . . , MI that share the same reward function r  . In each MDP M i , the agent receives an utterance u i from the user and must execute a trajectory  . The agent's goal is to infer  over the sequence of interactions, which should allow the agent to execute trajectories with high reward in as-yet unseen contexts. The agent maintains an estimate over  over the course of interactions. We introduce a model p (  | u, M ) that the agent will use to perform Bayesian updates of a posterior over  : p (  | u 1: i , M 1: i )  p (  | u i , M i )  p (  | u 1: i  1 , M 1: i  1 ) In the flight domain, we specialize this formulation to study a one-step MDP (contextual bandit). Trajectories  consist of a single action, choosing one of the available flights. Over a series of these rounds where the agent books a flight given the user's utterance u i , the agent must infer the user's flight preferences  to book flights from other unseen sets of options, without explicit language instruction from the user. Our model, summarized in Figure 2, defines a rational listener , L 2 , which predicts a distribution over rewards  , conditioned on an utterance u and a context M . (The terminology we use for listeners and speakers follows Bergen et al. 2016.) The rational listener uses Bayesian reasoning about a speaker model, S 1 , which produces utterances conditioned on a reward function and context: p L 2 (  | u, M )  p S 1 ( u | , M ) p (  | M ) Key to our model is that the S 1 speaker distribution p S 1 ( u | , M ) defines how speakers produce language that functions both to elicit correct actions and describe their underlying reward: p S 1 ( u | , M ) = p action ( u | , M ) + (1   ) p reward ( u |  ) , where  controls the speaker's nearsightedness how much does the speaker care about the listener choosing the correct action in the current context, rather than describing the reward in a context-independent way so that the agent can make good choices in future contexts? Optimizing for action. The behavior-optimizing term p action specifies that the speaker chooses utterances that elicit reward-maximizing behavior from a listener in the current environment: p action ( u | , M ) = (cid:88)  p refer ( u | , M ) p opt (  | , M ) , 8548 where the optimality model p opt (  | , M ) specifies the probability the speaker refers to trajectory  if their true reward is  . We can formulate the optimality model with the Boltzmann distribution common in IRL, where speakers are noisily-rational about which trajectories to refer to: p opt (  | , M )  exp( r  (  ; M )) , with rationality parameter  . This term specifies that utterances are more likely to refer to trajectories that have high reward according to the speaker's  , compared to other trajectories in M . Then, for a particular trajectory  , p refer ( u | , M ) specifies what utterances are likely to refer to that trajectory. In particular, we model that speakers choose utterances that would make a listener execute that trajectory: p refer ( u | , M )  p L base (  | u, M ) using a base listener model L base of the type common in past work on instruction following. We provide details on L base in Section 5. Optimizing for reward descriptiveness. Finally, we model p reward ( u |  ) , the second term in PS 1 , with a base speaker model, S base , that maps rewards to reward descriptive utterances: p S base ( u |  ) 2 . We also provide details on S base in Section 5. Our account of pragmatic generation can also be viewed as the graphical model in Figure 3(c), where, importantly, the reward influences the utterance both directly and via the action that the 2 In principle, p reward could also do pragmatic reasoning to optimize a listener's reward belief, but we did not find an improvement from doing so empirically. speaker refers to. We define p ( u | , , M ) to be: p ( u | , , M ) = p ( u | , M ) + (1   ) p ( u | , M ) and assume that utterances are reward-descriptive in a way that is independent of the current context, p ( u | , M ) = p ( u |  ) . We can confirm this leads us back to p S 1 by marginalizing out  : p ( u | , M ) = (cid:88)  p ( u | , , M ) p (  | , M ) =  (cid:88)  (cid:16) p ( u | , M ) p (  | , M ) (cid:17) + (1   ) p ( u | , M ) = p action ( u | , M ) + (1   ) p reward ( u |  ) Using this graphical model, we illustrate how our model differs from prior work in similar settings: Classic reference game pragmatics collapses belief and behavior. In general, RSA allows the speaker to optimize for any utility function, and in the simplest form the utility function optimizes for the listener's belief over world states (Good-man and Frank, 2016). However, in most work on RSA the only relevant world-state belief is belief about behavior, e.g., the referent that should be selected (Figure 3a). Instead, our setting disentangles communication about intended referents in a single context and communication about (reward) beliefs, which influence behavior on longer horizons. Andreas et al. (2017); Sumers et al. (2021) have made the same observation: reference games conflate whether the speaker's objective is to influence beliefs or actions, and modeling the speaker as one or the other produces distinct interpretations of utterances (e.g., speakers that only optimize for correct behavior may do so at the cost of being truthful about the reward). IRL assumes all information about the reward function is modulated by the trajectory. Prior work (MacGlashan et al., 2015; Fu et al., 2019) uses IRL to recover rewards from trajectories (e.g., from datasets pairing utterances with trajectories), and then supervising a model with these induced (utterance, reward) pairs. While prior work has not specifically considered pragmatics (i.e., speaker models), their implicit speaker model amounts to assuming that all information about the reward 8549 comes from trajectories, as in Figure 3b. In our experiments we compare against a pragmatic version of this action-centric speaker, which is equivalent to setting  = 1 in our model (only using p action ). In realistic settings where utterances are not unambiguous commands like go to the red door, it becomes important to model how actions and utterances reveal complementary information about rewards. We design FLIGHTPREF , a task for reward inference from natural language in the flight booking domain. FLIGHTPREF is designed to simulate a simplified interaction with a flight booking agent, where users communicate with the agent via language to book flights from a set of options. Effective agents must not only learn to book the preferred flight given an instruction in the immediate context (instruction following), but also learn the user's preferences over repeated interactions to book preferred flights in unseen contexts. We collect a dataset of natural language in a multi-turn game between a user (the speaker) and an assistant (the listener agent). Each flight is represented by a feature vector  (  )  R 8 (e.g., features of carrier, price, etc.). We assume the user has a linear reward function with parameters   R 8 , specifying a reward for a particular flight r  (  ) =  (cid:124)  (  ) . In the first round of the game, the user and assistant observe a set of three flight options and the user provides an utterance to describe the flight they want (the optimal flight under the reward function), e.g., the flight with the most stops. In each of the subsequent rounds, the user and assistant are presented with a new set of three flights. The assistant can either choose by guessing the user's preferred flight (under the same reward function), or prompt the user for another utterance describing the desired flight in the new set. If the assistant chooses but does so incorrectly, the user is prompted for another utterance describing the correct flight. Both players are penalized if the assistant chooses incorrectly, and earn points if the assistant chooses correctly (with more points for each round the assistant can do so without asking for help). The user is thus incentivized to provide utterances that inform the agent which flight to choose, while enabling long-term success over later rounds. that is the cheapest and has less stops. anything but american jetblue one i need a flight with any airline but jet blue, price and number of stops are a bad factor for me also. i prefer delta if aordable and low layovers. can you help me? even american is undesirable, paying more is important i like the flight that is $64 Figure 4: Sample text from the task, exhibiting a diversity of instructive and reward-descriptive language. To collect data for the task, we recruit Amazon Mechanical Turk workers and randomly pair them to play six games (i.e., six different reward functions) of six rounds each. Each game thus consists of 1-6 utterances describing options for the same reward function in different contexts. One person plays the role of the user and the other acts as the assistant. The user has access to a hidden reward function, which is a discretized, randomly-sampled vector   { 1 ,  0 . 5 , 0 , 0 . 5 , 1 } 8 . In total, we collected 2,568 utterances across 813 games, of which we split off the 91 games with the highest score (where the speaker and listener were able to communicate most effectively) for the evaluation set. More details about the data collection process can be found in Section A of the appendix. A sampling of text is shown in Figure 4. Utterances exhibit a range of phenomena: some users lean towards describing very option-specific features (e.g. i like the flight that is $64 ). Other users attempt to describe as much of their reward function as possible (e.g. i need a flight with any airline but jetblue,. . .  )we note that even when they did so, the user's tradeoffs between features remain ambiguous. Many of the utterances are neither fully option-specific nor fully reward-descriptive: instructions like one stop that is short both instruct the agent which flight to select in the present context, while communicating some generalizable (but incomplete) information about the user's preferences. Our pragmatic model (Section 3.1) relies on base listener and speaker models L base and S base . In this section, we describe implementations of these models for the FLIGHTPREF dataset. To train the base models, we use the speaker-side data of (utterance, 8550 option set, reward function) tuples from each round. Our base listener and speaker models assume that the utterances are generated conditionally independently given the reward; we capture the dynamics of multiple turns in the posterior reward inference. Both base models learn neural encodings of utterances u , actions  , and rewards  , and produce distributions by applying softmax functions to inner products between these encodings. We use   to denote the optimal action in each context, i.e.,   = arg max  r  (  ) . Base listener model. The base listener model L base is defined using inner product similarities between learned representations of actions  produced by an MLP encoder, and learned representations of utterances produced by a BERT-base (De-vlin et al., 2019) encoder: p L base (  | u, M )  exp( MLPL base (  )  BERTL ( u )) where the distribution is normalized over all actions (flights) available in the context,  (cid:48)  M . We set the rationality parameter  =  in p opt as speakers tend to refer primarily to the optimal option in our domain. Base speaker model. The base reward speaker model S base is defined using an inner product between representations of rewards  from an MLP encoder, and utterance representations from a BERT encoder: p S base ( u |  )  exp( MLPS base (  )  BERTS ( u ) / ) where p S base is normalized over a set of utterances taken from the training data (see Section C in the appendix), and  = 3 is a temperature parameter. Training. We fine-tune all model parameters, including the parameters of the initially-pretrained BERT utterance encoders in the listener and speaker on ( u, , M ) pairs from the training data using the AdamW optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019). The listener and speaker models are trained separately, without sharing any parameters between the encoders used in the two models. We independently train 5 random seeds of each base model and ensemble them together in evaluation by averaging their output probabilities, which we found improved performance of all models (both our full model and baselines). See Section C in the appendix for details and model hyperparameters. Pragmatic inference We follow previous work (Fried et al., 2018a; Monroe et al., 2017) and approximate the S 1 distribution by normalizing over a fixed set of utterances: the de-duplicated set of short utterances (less than 8 tokens, making up the majority of utterances) with no digits from the training data. We implement the full pragmatic model p L 2 (  | u, M ) in Pyro (Bingham et al., 2018) and use importance sampling to generate samples from the posterior over rewards. Given our dataset collection procedure (where we uniformly sample re-wards), we model an uniform prior over rewards p (  | M ) for the first interaction. We evaluate models in the same repeated turn setup that humans carried out in the task. For each game, models play the role of the listener in that game, updating the reward posterior (Section 3.1) after observing the utterance and option set in each round. Our goal is to estimate rewards that allow the agent to carry out the person's preferences: choosing the optimal option (flight) in unseen contexts (sets of flight options). To that end, we directly compare models on held-out accuracy : on 1,000 randomly-generated sets of three options, how often the model's estimate of the reward,   , selects the option that is optimal under the true reward. 3 We use the model's reward posterior mean as the estimate,   = E p   . We additionally provide com-3 Note that when collecting the dataset, we also tested human listeners's ability to generalize, but only had them select an option on a single unseen option setthe next one in the sequenceto make data collection tractable. parisons of reward L2 distance between the estimated reward and the true reward as a context-independent metric: (cid:113)(cid:80) 8 i =1 (   i    i ) 2 , where   is the true reward. For our full action + reward model, we set the nearsightedness parameter  = 0 . 5 for all posterior updates. We compare to an action-only model that uses only p action (i.e., setting  = 1 . 0 ). This model is representative of approaches from past work on language-conditioned reward learning (e.g., Mac-Glashan et al. 2015; Fu et al. 2019) that infer rewards purely from the actions that utterances refer to. We also compare to a reward-only model that uses only p reward (inferring rewards purely from the utterance, without conditioning on actions, i.e., setting  = 0 . 0 ). For comparison to versions of our approach that remove pragmatic modeling, see Section D.1 in the appendix. In Table 1 we compare all models on held-out accuracy averaged over all rounds in the evaluation set (for each round, having observed all previous rounds in that game). Note that because held-out accuracy is assessed by the proportion of randomly-generated flight sets (out of 1,000) where the true reward function and the inferred reward function pick out the same optimal flight, it is significantly more difficult than achieving high accuracy on a single three-choice instance. Our full action+reward model achieves a held-out accuracy of 59.1%, +6.3% over the action-only model and +1.3% over the reward-only model, indicating that combining both sources of information allows better inference of rewards that enable optimal actions in novel contexts. For reference, an oracle baseline that infers the value of k randomly chosen features perfectly and is uniform on the other features obtains the following held-out accuracies: k = 1 (43%), 2 (51%), 3 (60%), 4 (65%), showing that our model is able to attain similar generalization performance even in the presence of uncertainty (without receiving oracle information about the true value of any feature). We analyze why our model benefits from both components in Section 6.3, and discuss potential for further improvements in Section 6.4. We explore how each model's reward inferences change as more observations are obtained over the course of a game. In Figure 5, we plot held-out accuracy and L2 distance to the true reward as a function of number of observed utterances. Our model outperforms the action-only and reward-only models for all numbers of observed utterances. is most important when there are few observations. While our full action+reward model improves substantially over the action-only model at all points, this improvement generally decreases as more utterances are observed (Figure 5). Conversely, the improvement of the full model over reward-only generally increases. Qualitatively, we observe that this occurs because utterances tend to mention the most extreme features of the reward 8552 price .59 # stops .50 longest stop .96 arrival time 0.0 Southwest price .07 # stops .50 longest stop .56 arrival time .92 Delta price .78 # stops 0.0 longest stop .80 arrival time .08 Delta want a long time before meeting Arrival time Action-only Belief-only Ours (a) Both the described action (the referenced flight is the one with the highest arrival time) and the explicit reward description in the utterance provide evidence that the user's true reward on arrival time is positive, leading the posterior in our model to (correctly) place more probability mass on positive values of this feature. (b) Evidence from actions and from the utterance complement each other: the action-based model captures that rewards that are positive on arrival time make the selected flight optimal, even though it is unmentioned, while the reward-based model captures evidence about the reward from the user's utterance. function, which allow our model to estimate the values of these important features. When there are few observations, inferring reward information from utterances in this way is more informative than using only the option implied by the user's utterance, which does not disambiguate between rewards that select the same option (a commonly discussed problem in IRL; Ziebart et al. (2008)). Inferring evidence from actions is most important when there are more observations. We observe that the action-only model improves more consistently over rounds. Qualitatively, the information that utterances provides about rewards is correlated across multiple roundsspeakers frequently mention salient reward features, whereas actions consistently provide new information about all features. This is particularly pronounced in our domain, due to a relatively small feature and action space. In other more complex domains, actions might provide even more benefits as they provide fine-grained information about reward values and tradeoff boundaries that are more difficult to communicate precisely in language. In this section, we investigate why our model benefits from both the action and reward models. show the reward posteriors for each model after a single update on a round (starting from a uniform prior). In Figure 6a, we observe how the action-and reward-only models can make correlated updates on an utterance and context where both the action (a flight with a high value on arrival time) and the utterance provide evidence about the arrival time feature. This leads our model's posteriors to aggregate more probability mass on positive values of that feature. In Figure 6b, we show how each model can make inferences about different features for the same contextthe action-only model inferring positive values for arrival time given the observed flight and the reward-only model updating on flight price and stops. Our model posterior aggregates information from both. Some utterances are primarily nearsighted, and others primarily farsighted. Another reason our full model improves is because some utterances are particularly farsightedmentioning a great deal of explicit information about the reward (which the action-only model cannot take advantage of)while other utterances are more nearsightedspecialized to the particular action, e.g., saying just enough to uniquely identify the optimal flight. Sorting the utterances by difference in accuracy between the action-only and reward-only models confirms that they exhibit qualitatively different phenomena: examples where the reward-only model helps the most are highly reward-8553 descriptive (e.g., if i had a choice, i would never fly with delta and american! get me jetblue or southwest. . . ) while examples where the action-only model helps most have less informative utterances (e.g., the cheaper the better ). Our full model is able to handle both kinds of language use. To further analyze the influence of the action and reward component, we evaluate an oracle model that switches between the action-only and reward-only models, choosing the model with highest held-out accuracy in each round. This model outperforms our action+reward model (improving from 59.1 to 62.9% on overall held-out accuracy), suggesting that further improvements could be obtained by integrating evidence from the two models. Doing so optimally is challenging in our setting: when a user says  i like the cheap jetblue flight , do they mean to say they like JetBlue generally, or just that they want to choose a desirable flight that happens to be uniquely identified by JetBlue? Future work might explore adaptively switching policies (e.g., using the utterance, or knowledge about the user). While our base models have fairly high performance (e.g., the base listener model L base has an average accuracy of 74% at selecting the optimal choice in each option set that has an utterance in the evaluation data), they naturally have some errors which lead to errors in reward inference. We test the influence of this underlying prediction error by skipping posterior updates on all rounds where the base listener predicts the incorrect option for the true reward function. This change improves held-out accuracy by 6% over the reward-only model after six observations (+4% from the original gap), indicating (1) that dataset affords future work on improved instruction following models and (2) that our reward inference procedure benefits from base model improvements. We note that in our task design, the user does not provide a demonstration (i.e., a choice of flight) to the model. However, if it is convenient to obtain demonstrations from users (e.g., a flight booking interface could let the person click on the flight they want in addition to specifying what they want in natural language), demonstrations would effectively serve as an oracle instruction-following model for that context, which could be incorporated into our full reward inference model. We presented a method for using natural language to infer reward functions: representing the goals, preferences, and intents underlying action. Conceptually, our work builds on previous work on language grounding by exploring how language serves a dual purpose. Utterances can refer directly to actions to be taken, as studied in instruction following. Beyond that, they communicate information about why those actions should be taken, and what actions may be desirable in new contexts. To build language-guided agents that can interact with people over longer horizons, it may be useful to model this relationship between language, actions, and rewards. Furthermore, language is ambiguous about both actions and goals. Standard settings for studying pragmatics (e.g., reference games) address how to resolve ambiguity about what object or action the speaker is choosing to refer to. We have explored how these settings can be extended by considering the preferences underlying those choices. We introduced FLIGHTPREF , a new dataset of naturalistic interactions between people in a multi-turn flight booking game. FLIGHTPREF uses held-out accuracy as a metric for evaluating interpretation success beyond selecting the right action in a single environment. Future work can build on the task by 1) learning or evaluating with more complex reward functions (e.g., using deep reward representations); 2) exploring how people communicate about their real preferences and modeling a natural prior (e.g., that people tend to prefer cheaper flights), instead of providing annotators with ground-truth preferences; 3) allowing other ways to handle uncertainty, e.g., leveraging the reward posterior to interactively learn to ask; or 4) extending these approaches to other domains where modeling goals and preferences may be important (e.g., language-conditioned robotics). We thank Eric Wallace, Jerry He, and the other members of the Berkeley NLP group and InterACT Lab for helpful feedback and discussion. This work is supported by a grant from the Office of Naval Research (ONR-YIP).
abstain abstain result objective abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain result abstain method method method method abstain method result result abstain other abstain objective abstain abstain abstain abstain objective abstain abstain method abstain objective abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain other method abstain abstain method method abstain abstain method abstain abstain abstain method other abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain other abstain abstain other abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain objective abstain result abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain abstain objective abstain result abstain result abstain abstain result other abstain abstain result abstain abstain abstain abstain result abstain result objective objective abstain abstain abstain abstain abstain objective objective abstain other other other