summaries: This paper addresses the problem of deriving the correct answer when contradicting examples are presented to the model. First, it introduces a dataset for the task. The dataset, ContraQA is built on SQuAD, and it contains contradicting contexts produced by humans and neural-models. Then, it presents a model for generating contradicting examples. The model, BART-FG, generates fake contexts by iteratively modifying and original input paragraph. The procedure starts by applying a constituency parsing to extract constituency spans from the input sentence. Then, it randomly masks some of these constituency spans, that are eventually fill by a BART model fine-tuned on Wikipedia dump. To study how QA models behave with contradicting examples, this work evaluates the performances in a scenario where the correct and the fake contexts are presented to the model. In order to make the QA system robust to fake contexts, it proposes a misinformation-aware framework that combines the score of the model with a trust score outputs by a fake detector, which is a transformer-based model trained to classify if a context is real or fake. The results show that under this setting, model performance decreases, and that the reduction can be mitigate by applying the fake detector model. Finally, it shows a comparison, between BART-FG and GPT-2, to identify which of the two can generate more impactful fake contexts for the QA model.

summaries: The work investigates closed-domain Question Answering under contradicting contexts by introducing a new task ContraQA—an extension of SQuAD1.1—which includes contradicting contexts for the SQuAD articles, produced by both humans and neural models. The work also proposes a neural framework, BART-FG, to automatically generate these contradicting contexts by iteratively modifying constituency spans on the original context. Finally, the work gives a brief analysis on how SOTA QA systems perform on the new task, ContraQA, and proposes a misinformation detecting system which when unified with a Machine Reader performs significantly better than SOTA systems over ContraQA.

summaries: The authors studied how contradictory information affects the accuracy of QA systems.  They created a new dataset of ~10k from SQuAD with added contradictory paragraphs (context). The contradictory data was generated by two different ways. The authors employed Amazon Mechanical Turks raters to rewrite the original context. They also designed a BART-FG model to automatically produce such context by replacing spans with model generated value.   The authors also proposed a way to help QA systems avoid contradictory information. A RoBERTa-based model was used to classify a context into trustworthy or not, with an accuracy around 80%. The trustworthiness score is then used to weigh the final result, together with existing scores of quality/confidence.  Evaluation was done with the new dataset. It showed that 1) adding contradictory information hurts QA performance badly, 2) the RoBERTa-based classifier can help regain some of the loss, but not all of them. The authors also measured the effectiveness of contradictory information creation, where human takes the top place by producing the least altered context with largest effect on final outcome.  The authors promised to share the dataset and the source code / weights of the proposed models. They further discussed the potential ethical impact of releasing the data, arguing that it's net beneficial.

summaries: This paper releases a new dataset with human and machine generated contradictory contexts for QA pairs from SQuAD 1.1. Amazon Mechanical Turk Workers are shown a paragraph and are asked to make edits such that it contradicts the original paragraph with respect to elements such as time, outcome, purpose, location, etc. In addition, the authors fine tune BART on a collection of masked constituent parses of Wikipedia sentences and it is then trained to fill the mask with an alternative phrase.  To automatically generate contradictory contexts, the authors use this fine tuned the BART model on the masked constituency parse of paragraph sentences. A dataset of 10,000 paragraphs from SQuAD  are transformed (once by mechanical Turk workers, and the rest by three different transformations by the BART Model). The paper presents experiments on this dataset for the task of QA -- specifically, machine reading comprehension. In one experiment, the QA system is first trained to predict which of the 5 (1 real + 4 contradictory) is correct. Then an off-the-shelf span based passage reader returns spans as answers. In the second experiment, the performance of QA models is compared on the unmodified SQuAD dataset as well as a version where a distracting passage is also added to context (by randomly choosing a different passage). The authors experiment using BERT, ROBERTA and SPAN-BERT and report a drop in performance in both experimental settings. In addition experiments reveal that models return worse performance on the subset of the data created by human workers ( perhaps unsurprising). Additional studies on the nature of edits have also been presented.   Overall a well written and easy to read paper. However, I am not sure I am clear about the goals of the paper -- I elaborate further in the rest of the review.

abstractText: With a rise in false, inaccurate, and misleading information in propaganda, news, and social media, real-world Question Answering (QA) systems face the challenges of synthesizing and reasoning over contradicting information to derive correct answers. This urgency gives rise to the need to make QA systems robust to misinformation, a topic previously unexplored. We study the risk of misinformation to QA models by investigating the behavior of the QA model under contradicting contexts that are mixed with both real and fake information. We create the first large-scale dataset for this problem, namely CONTRAQA, which contains over 10K human-written and model-generated contradicting pairs of contexts. Experiments show that QA models are vulnerable under contradicting contexts brought by misinformation. To defend against such threat, we build a misinformation-aware QA system as a counter-measure that integrates question answering and misinformation detection in a joint fashion.

authors:
Max Bartolo
Alastair Roberts
Johannes Welbl
Sebastian Riedel
Pontus Stenetorp

title: Beat the AI: investigating adversarial human annotation for reading comprehension

venue: Transactions of the Association for Computational Linguistics (TACL),

authors:
Ilya Sutskever
Dario Amodei

title: Language models are few-shot learners

venue: In Annual Conference on Neural Information Processing Systems (NeurIPS),

authors:
Danqi Chen
Adam Fisch
Jason Weston
Antoine Bordes

title: Reading wikipedia to answer opendomain questions

venue: In Annual Meeting of the Association for Computational Linguistics (ACL),

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: pre-training of deep bidirectional transformers for language understanding

authors:
Wee Chung Gan
Hwee Tou Ng

title: Improving the robustness of question answering systems to question paraphrasing

authors:
Cristina Garbacea
Samuel Carton
Shiyan Yan
Qiaozhu Mei

title: Judge the judges: A largescale evaluation study of neural language models for online review generation

venue: In Conference on Empirical Methods in Natural Language Processing (EMNLP),

authors:
Robin Jia
Percy Liang

title: Adversarial examples for evaluating reading comprehension systems

authors:
Yichen Jiang
Mohit Bansal

title: Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA

authors:
Mandar Joshi
Danqi Chen
Yinhan Liu
Daniel S. Weld
Luke Zettlemoyer
Omer Levy

title: Spanbert: Improving pre-training by representing and predicting spans

authors:
Vidur Joshi
Matthew E. Peters
Mark Hopkins

title: Extending a parser to distant domains using a few dozen partially annotated examples

authors:
Amita Kamath
Robin Jia
Percy Liang

title: Selective question answering under domain shift

authors:
Vladimir Karpukhin
Barlas Oguz
Sewon Min
Patrick S.H. Lewis
Ledell Wu
Sergey Edunov
Danqi Chen
Wen-tau Yih

title: Dense passage retrieval for open-domain question answering

authors:
Divyansh Kaushik
Zachary C. Lipton

title: How much reading does reading comprehension require? A critical investigation of popular benchmarks

authors:
Mike Lewis
Yinhan Liu
Naman Goyal
Marjan Ghazvininejad
Abdelrahman Mohamed
Omer Levy
Veselin Stoyanov
Luke Zettlemoyer

title: BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension

authors:
Patrick S.H. Lewis
Ethan Perez
Aleksandra Piktus
Fabio Petroni
Vladimir Karpukhin
Naman Goyal
Heinrich Küttler
Mike Lewis
Wen-tau Yih
Tim Rocktäschel
Sebastian Riedel
Douwe Kiela

title: Retrieval-augmented generation for knowledge-intensive NLP tasks

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized BERT pretraining approach

authors:
Shayne Longpre
Kartik Perisetla
Anthony Chen
Nikhil Ramesh
Chris DuBois
Sameer Singh

title: Entity-based knowledge conflicts in question answering

authors:
Adyasha Maharana
Mohit Bansal

title: Adversarial augmentation policy search for domain and cross-lingual generalization in reading comprehension

venue: In Conference on Empirical Methods in Natural Language Processing (EMNLP): Findings,

authors:
Pramod Kaushik Mudrakarta
Ankur Taly
Mukund Sundararajan
Kedar Dhamdhere

title: Did the model understand the question

authors:
Timothy Niven
Hung-Yu Kao

title: Probing neural network comprehension of natural language arguments

authors:
Alec Radford
Jeffrey Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI blog,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J. Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer

venue: Journal of Machine Learning Research (JMLR),

authors:
Pranav Rajpurkar
Jian Zhang
Konstantin Lopyrev
Percy Liang

title: Squad: 100, 000+ questions for machine comprehension of text

authors:
Marco Túlio Ribeiro
Sameer Singh
Carlos Guestrin

title: Semantically equivalent adversarial rules for debugging NLP models

authors:
Marco Túlio Ribeiro
Carlos Guestrin
Sameer Singh

title: Are red roses red? evaluating consistency of question-answering models

authors:
Yicheng Wang
Mohit Bansal

title: Robust machine comprehension models via adversarial training

authors:
Wei Yang
Yuqing Xie
Aileen Lin
Xingyu Li
Luchen Tan
Kun Xiong
Ming Li
Jimmy Lin

title: End-to-end open-domain question answering with bertserini

venue: In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Demonstrations,

authors:
Rowan Zellers
Ari Holtzman
Hannah Rashkin
Yonatan Bisk
Ali Farhadi
Franziska Roesner
Yejin Choi

title: Defending against neural fake news

heading: 1 INTRODUCTION

text:
A typical Question Answering (QA) system (Chen et al., 2017; Yang et al., 2019; Karpukhin et al., 2020; Lewis et al., 2020b) starts by retrieving a set of relevant context documents from the Web, which are then examined by a machine reader to identify the correct answer. Existing work equate Wikipedia as the web corpus. Therefore, all retrieved context documents are assumed to be clean and trustable. However, real-world QA faces a much noisier environment, where the web corpus is tainted with misinformation. This includes unintentional factual mistakes made by human writers and deliberate disinformation intended to deceive. Aside from human-created misinformation, we are also facing the inevitability of AI-generated misinformation. With the continuing progress in text generation (Radford et al., 2019; Brown et al., 2020; Lewis et al., 2020a), realistic-looking fake web documents can be generated at scale by malicious actors (Zellers et al., 2019).
The presence of misinformation — no matter deliberately created or not, no matter human-written or machine-generated — affects the reliability of the QA system by bringing in contradicting information in the context documents. Figure 1 shows a question and five context documents, which give contradicting answers to the question. Only one context document (in green) is factually correct, while the rest are human-written or machine-generated fake information. Faced with such contradicting contexts, even human readers must be familiar with “Super Bowl 50” or rely on Web search to invalidate these fake contexts. Although current QA models often achieve super-human performance under the idealized case of clean contexts, we argue that they may easily fail under the more realistic case of contradicting contexts, especially when they do not have the ability to identify fake information and reason over contradicting contexts.
We seek to study risks of misinformation to QA models by investigating how QA models behave under contradicting contexts that are mixed with both real and fake information. Since there is no public QA dataset that marks and introduces contradicting contexts, we construct one ourselves: CONTRAQA, a large-scale dataset that specifically serves this need. Our dataset is constructed on top of SQuAD 1.1 (Rajpurkar et al., 2016). For each context paragraph P in SQuAD, we create a fake version P ′ by modifying information in P , such that: 1) certain information in P ′ contradicts with the information in P , and 2) P ′ is fluent, consistent, and looks realistic. We include both human-written and model-generated fake contexts in our dataset to simulate a realistic environment. For the human-written part, we ask Mechanical Turkers to write fake contexts by modifying
the original context. For the model-generation part, we propose a strong rewriting model, namely BART-FG, which can controllably mask and re-generate constituency spans in the original context to produce fake contexts. The original contexts, human-written and model-generated fake contexts are mixed together blindly and presented to QA model to answer a given question. A sample of CONTRAQA is shown in Figure 1, where the model is presented with multiple contradicting contexts to predict the answer for a given question. This stimulates a real-world situation where both real and fake information are retrieved as the context documents for open-domain QA. A robust QA model should be able to deal with misinformation and properly handle contradictory information.
Unfortunately, from extensive experiments, we find that existing QA models are vulnerable under contradicting contexts brought about by misinformation, regardless of whether the fake contexts are manually-written or model-generated. State-of-the-art QA models (Devlin et al., 2019; Liu et al., 2019; Joshi et al., 2020) all suffer from a significant drop in their exact match (EM) score by 20–30 when presented with contradicting contexts in CONTRAQA. Our analyses show that 1) our proposed context-rewriting BART-FG can create more deceiving misinformation than GPT-2 (Radford et al., 2019), and that 2) though machine-generated fake contexts are deceiving, they are generally not on par with human-written ones. We find that humans are still much better at making subtle and efficient edits to create contradicting information.
To defend against the potential threat of misinformation, we build a more robust QA system that integrates question answering with a misinformation discriminator in a joint fashion. Decisions made by discriminator assist QA models in identifying likely misinformation, which in turn improves the EM score by 17.2%, assuming access to a sufficient level of training data.
We plan to release CONTRAQA publicly, helping pave the way for building more robust QA systems. Although there are multiple types of misinformation in the real world, such as hoaxes, rumors, or false propaganda, we focus on the misinformation that brings conflicting information to the QA contexts. However, our work gives a general threat model for QA under misinformation, including an attack model that mixes both real and fake information in the QA context, and a defense model that combines question answering and misinformation detection. Followup research can easily build upon this threat model by studying how other types of misinformation mislead the QA systems. We summarize our contributions as follows:
• To the best of our knowledge, this is the first work to investigate QA under contradicting contexts. •We construct a large-scale dataset CONTRAQA that includes contradicting contexts produced by both humans and neural models.
•We propose BART-FG, a novel framework that generates fake contexts by iteratively modifying constituency spans of the original context.
• To defend against the threat of misinformation, we propose a model that unifies question answering and misinformation discrimination.

heading: 2 RELATED WORK

text:
Adversarial Attacks in QA. Adversarial attacks aim to trick QA models intentionally by perturbing their input contexts. These perturbations are expected to preserve the meaning of contexts while
degrading model performance. Ribeiro et al. (2018) make grammar perturbations such as replacing What has with What’s. Other perturbations include adding distractor sentences (Jia & Liang, 2017; Wang & Bansal, 2018), modifying phrases (Maharana & Bansal, 2020), and human-in-theloop edits (Bartolo et al., 2020). Although we also make modifications to the QA contexts, our setting differs from the task of adversarial attacks in QA. First, we aim to make the modified contents contradict with the original paragraph, while adversarial perturbations are expected to preserve the original meaning. Second, the goal of generating contradicting contexts is to study the impact of misinformation on QA models. In contrast, adversarial contexts aim to reveal the weaknesses of systems upon local changes that are imperceptible by humans.
Improving Robustness for QA. Our work aims to analyze vulnerabilities to develop more robust QA models. Current QA models demonstrate brittleness in different aspects. QA models often rely on spurious patterns between the question and context rather than learning the desired behavior. They might ignore the question entirely (Kaushik & Lipton, 2018), focus primarily on the answer type (Mudrakarta et al., 2018), or ignore the “intended” mode of reasoning for the task (Jiang & Bansal, 2019; Niven & Kao, 2019). QA models also generalize badly to out-of-domain (OOD) data (Kamath et al., 2020). For example, they often make inconsistent predictions for different semantically equivalent questions (Gan & Ng, 2019; Ribeiro et al., 2019). Notably, a concurrent work (Longpre et al., 2021) shows QA models are less robust to OOD data where the contextual information contradicts with the learned information. Different from that work, we propose a brandnew angle for QA robustness which studies the vulnerability of QA models under misinformation. Although our work and Longpre et al. (2021) both create contradictory contexts, we use them to study different problems. Moreover, we include both human-created and model-generated contradictory contexts, which are more flexible and diverse than their entity-based contradictions.
Combating Neural-generated Misinformation. While we are excited about the recent progress in neural text generation, they can be misused to generate realistic-looking and catchy hallucinations, such as fake news (Zellers et al., 2019) and fake online reviews (Garbacea et al., 2019). When produced at scale, neural-generated misinformation can pose threats to many NLP applications. We believe we are the first to study the risk of neural-generated misinformation to QA models and propose a misinformation-aware QA system as a countermeasure.

heading: 3 DATASET: CONTRAQA

text: We construct the CONTRAQA dataset as follows. We first select 10,026 unique context paragraphs from SQuAD 1.1, including all the 2,036 unique paragraphs from the validation set and 8,000 paragraphs randomly sampled from the training set. For each selected paragraph CR, we create a set of N fake contexts (CF1 , · · · , CFN ) by modifying some information in CR, with the requirement that each fake context look realistic while containing contradicting information with CR. We use two different ways to create fake contexts: 1) via human edits: we ask online workers from Amazon Mechanical Turk (AMT) to produce fake contexts by modifying the original context, and 2) via BART-FG: our novel generative model BART-FG, which iteratively masks and re-generates constituency spans from the original context to produce fake contexts.

heading: 3.1 MANUAL CREATION OF FAKE CONTEXTS

text:
To solicit human-written deceptive fake contexts, we release 10K HITs (human intelligence tasks) on the AMT platform, where each HIT presents the crowd-worker with one context paragraph CR we selected. We ask workers to modify the contents of the given paragraph to create a fake version, following the below guidelines:
• The worker should make at least M edits at different places, where M equals to one plus the number of sentences in the contexts CR. • The worker should make at least one long edit that rewrites at least half of a sentence. • The edits should modify key information to make it contradict with the original, such as time, location, purpose, outcome, reason, etc.
1
• The modified paragraph should be fluent and look realistic, without commonsense errors.
To select qualified workers, we restrict our task to workers who are located in five native Englishspeaking countries1, and who maintain an approval rating of at least 90%. To ensure the annotations fulfil our guidelines, we give ample examples in our annotation interface with detailed explanations to help workers understand the requirements. The detailed annotation guideline is in Appendix B. We also hired three computer science major graduate students as human experts to validate a HIT’s annotation. In the end, 104 workers participated in the task. The average completion time for one HIT is 5 minutes, and payment is $1.0 U.S. dollars/HIT. The average acceptance rate was 93.75%.

heading: 3.2 MODEL GENERATION OF FAKE CONTEXTS

text:
Aside from human-written fake contexts, we also want to explore the threat of machine-generated fake contexts to QA. This source may be more of a concern than human-created misinformation, since they can easily be produced at scale. Recently introduced large-scale generative models, such as GPT2 (Radford et al., 2019), BART (Lewis et al., 2020a), and Google T5 (Raffel et al., 2020), can produce realistic-looking texts, but they do not lend themselves to producing controllable generation that only replaces only the key information with contradicting contents. Therefore, to evaluate the efficacy of realistic-looking neural fake contexts, we propose BART Fake Contexts Generator (BART-FG), which produces both realistic and controlled generated text by iteratively modifying the original paragraph. As shown in Figure 2, for each sentence S of the original paragraph, BARTFG produces its fake version S′ via a three-step process:
1) Constituency Parsing. We first apply constituency parsing to extract constituency spans from the input sentence to represent its syntactic structure. We use the off-the-shelf constituency parser from Joshi et al. (2018) in AllenNLP2, which achieved 94.11 F1 on the Penn Treebank.
2) Constituency Masking. We then randomly select a constituency phrase3 (a non-terminal in the parse tree) and replace it with a special mask token [MASK]. We choose to mask constituency phrases instead of random spans as: 1) constituents represent complete semantic units such as “Super Bowl 50”, which avoids meaningless random phrases such as “Bowl 50”; and 2) constituents often represent important information in the sentence — such as time, location, cause, etc.
3) Mask Filling. We fill in the mask by generating a phrase different with the masked phrase. The mask is filled by the BART model fine-tuned on the Wikipedia dump with a new self-supervised task called gap constituency filling, introduced later.
The above pipeline is iteratively run forK times to generate sentence S′ from S. We choose to make the edits iteratively rather than in parallel to model interaction between multiple edits. For example, in Figure 2, where the previous edit changed “Santa Clara” to “Atlanta”, the next edit may change “California” into “Georgia” to make the contents more consistent and realistic.
1Australia, Canada, Ireland, United Kingdom, USA 2https://demo.allennlp.org/constituency-parsing 3Our considered constituency types: ADJP, ADVP, NP, PP, SBAR, SBARQ, SINV, VP, SQ, WHNP, WHPP
Gap Constituency Filling (GCF) Pre-Training. To train the BART model to learn how to fill in a masked constituency phrase, we propose a novel pre-training task named Gap Constituency Filling (GCF). For each article in the Wikipedia dump that consists of T sentences [S1, S2, · · · , ST ], where each sentence is a word sequence St = [wt1, · · · , wt|St|], we construct the following training data for t = 2, · · · , T − 1:
Input: S1, St−1, wt1:a−1,[MASK], w t b+1:|St|, St+1 Output: w t a:b = [w t a, · · · , wtb]
where the output represents a masked constituency span that starts with the a-th word and ends with b-th word. The input is the concatenation of the first sentence S1, the previous sentence St−1, the current sentence St with one constituency being masked, and the subsequent sentence St+1. The BART model is fine-tuned to predict the output given the input on the entire Wikipedia dump. This task trains the BART model to predict the masked constituent, given both global contexts (S1) and local contexts (St−1, St+1).

heading: 3.3 DATA STATISTICS AND ANALYSIS

text:
We collect 10,026 unique context paragraphs from the SQuAD dataset (8,000 from the train set, 2,026 from the dev set). For each paragraph CR, we create four fake contexts {CF1 , · · · , CF4 }, where one fake context is written manually and the other three are generated via BART-FG. Afterward, each paragraph is paired with its corresponding question–answer pairs. Therefore, each data sample in CONTRAQA is a tuple of (Q,A, C = {CR, CF1 , · · · , CF4 }), where C is the contradicting contexts that are mixed with one original context and four fake contexts. Since each context C is paired with multiple (Q,A) in SQuAD, we finally obtain 36,447 and 10,218 data samples for the train and dev set, respectively. Since the SQuAD test set is not released, we only create contradictory examples for the full SQuAD dev set. For a fair comparison, we compare the performance of QA models between the original and contradictory versions of the SQuAD dev set.
Table 1 shows six original contexts with their corresponding contradicting contexts, which represent six common types of modifications, explained in the following:
(1) Entity Replacement: replacing entities (e.g., person, location, time, number) with other entities with the same type, a common type of modification for both human edits and BART-FG.
(2) Verb Replacement: replacing verb or verb phrase with its antonymic meaning, e.g., “force these children to”→ “prevent these children from”. (3) Adding Restrictions: create contradiction by inserting additional restrictions to the original content, e.g., “every day”→ “every day but Sunday”. (4) Sentence Rephrasing: rewrite the whole sentence to express a contradicting meaning, exemplified by (4). This is common in human edits but rarely seen in model-generated contexts, since this requires deep reading comprehension.
(5) Disrupting Orders: make a contradiction by disrupting some property of the entities; e.g., example (5) switches the property of “analysis of algorithms” and “complexity theory”.
(6) Consecutive Replacements: humans are better in making consecutive edits to create a contradicting yet coherent sentences, exemplified by (6).

heading: 4 MODELS AND EXPERIMENTS

text:
We now study how extractive QA models behave under such contradicting contexts. Extractive QA aims to answer a given questionQ by selecting a span from a set of context paragraphs C. We apply this to contradicting contexts by proposing two settings as follows:
1. Contra-QA. In this setting, QA is conducted under contradicting contexts, i.e., C = {CR, CF1 , · · · , CFN}, where N = 4 in our CONTRAQA dataset. We shuffle the context paragraphs so that the model does not know which context is real. Following Chen et al. (2017), a QA module (a.k.a. passage reader) is applied to each of the paragraph Ci to select the best answer span Ai and provides its confidence score Si. Afterward, the system returns the answer with the highest (normalized) span score.
2. Contra-QA (w/ Detection). In order to mitigate the harm of misinformation, we propose a misinformation-aware QA framework that integrates question answering with fake context detector. For each context paragraph Ci, a fake context detector outputs its trust score Ri; i.e., the confidence score that information in Ci is trustable. We then combine the trust score Ri with the confidence score Si for the best answer span in Ci via linear interpolation: Pi = λ · Si + (1− λ) ·Ri, where µ ∈ [0, 1] is a hyperparameter which we set as 0.5. Finally, we select the answer span with the best Pi across all context paragraphs as the final prediction.
The above settings are compared with two settings without contradictory contexts. 1) SQuAD: the traditional QA setting in which only the real context is given. 2) SQuAD + Random Ctx.: the real context is paired with N randomly sampled other contexts irrelevant to the question. This helps to differentiate whether the model is really distracted by the contradicting contexts or simply by the fact that there are other contexts.
We consider four state-of-the-art QA models with public code that achieved strong results on the public leader board of SQuAD: BERT-base (Devlin et al., 2019), RoBERTa-base, RoBERTalarge (Liu et al., 2019), and Span-BERT (Joshi et al., 2020). We use their implementations from the Hugging Face library, fine-tuned on the SQuAD-1.1 training set. We use the standard Exact Match (EM) and F1 metrics to measure QA performance. We train the fake context detector by fine-tuning the RoBERTa-large model to differentiate real and fake paragraphs (binary classification) using a moderate level of labeled real/fake paragraphs in the CONTRAQA training set. The classifier achieves an detection accuracy of 80.57% with 10,000 training examples.

heading: 4.1 MAIN RESULTS

text:
In Table 4, we show the performance of different QA models on the CONTRAQA test set. We have two major observations.
Misinformation can easily mislead QA models. When moving from clean SQuAD contexts to contradicting contexts, all four QA models suffer from large performance drops between 23.4 (BERT-base) and 25.91 (RoBERTa-large) in absolute EM value, and drops between 40.67% and 42.24% in relative EM value. This reveals the serious impact of misinformation and its potential
threat to current QA systems. As QA models are not trained to differentiate fake contexts, they can be easily mislead by misinformation.
QA models are mainly distracted by contradictions brought by misinformation. By pairing the original context with four randomly sampled other contexts (SQuAD + Random Ctx.), we only observe an average EM drop of 9.4%, showing that the QA models are relatively robust when using random distractor contexts. However, when using the contradictory contexts as the distractor (Contra-QA), we observe an average 41.21% drop in EM. The results show that the models are largely distracted by the contradictory contexts rather than by the presence of additional contexts.
Misinformation-aware QA models are more robust under contradicting contexts. When the RoBERTa-large QA model is equipped with the fake context detector trained with 10,000 examples, the EM score improves from 61.34% to 71.92% (+17.2%). Similar improvements are observed for other QA models. This result shows that the decisions of the fake context detector can assist QA models in identifying likely misinformation, thus improving the QA performance under contradicting contexts. However, the performance is still much lower than the results under clean context (87.25% EM), due to the imperfect accuracy of the detector. Moreover, training an effective detector is often difficult in the real world, as it requires the time-consuming and costly labeling of large amounts of in-domain real and fake contexts as training data. Therefore, how to effectively fight against the threat of misinformation in QA remains challenging and deserves future attention.

heading: 4.2 IMPACT OF THE NUMBER OF FAKE CONTEXTS

text: Given the contradicting contexts C = {CR, CF1 , · · · , CFN} (cf Figure 3), we plot the EM and F1 for the RoBERTa-large model with N = 0, 1, 2, 3, 4. The results show a linear downward trend of both EM and F1 as N increases. Therefore, misinformation may have more severe impact on QA systems when they are produced at scale. With the availability of pretrained text generation models, producing fluent and realistic-looking contexts now has little marginal cost. This brings an urgent need to effectively defend against neural-generated misinformation.

heading: 4.3 WHICH IS MORE DECEIVING: HUMAN- OR MODEL-GENERATED MISINFORMATION?

text:
We further investigate which is more deceiving to QA models: human or neural misinformation? To study this, based on the real contexts CR in the CONTRAQA test set, we generate their fake contexts {CF1 , · · · , CFN} using different methods and then evaluate the QA performance of the RoBERTalarge model. The methods we consider are human and BART-FG; where in the case of BART-FG we consider three variants in which the number of iterations K is set as 1, 2, and 3, respectively.
Table 3 shows the QA performance for different methods. We introduce a metric called average edit distance percentage (Edit(G)) to measure the average number of edits a generation method G needs to make to the original contexts in order to generate the fake paragraph, defined as follows: Edit(G) = ∑M i=1
∣∣edit distance(G(CRi ), CRi )/length(CRi )∣∣, where CRi is the i-th real context in the test set, and G(CRi ) is the fake context generated by method G for CRi . This metric measures the relative edit distance between the true and fake context, taking average in the test set. From the results in Table 3, we make two major observations:
Humans can create more misleading contradictions with fewer edits. Table 3 shows that when pairing the real context with the human-written fake context, the models underperform (an EM 72.02%) compared against pairings with model-written fake contexts. Interestingly, the average edit distance percentage for humans is also the lowest (19.19%) among all methods. This indicates that humans create more challenging fake contexts that trick QA models with fewer edits. From our observations in Table 1, humans make more subtle and deceiving edits to create contradictions, such as switching “former” and “latter” (Example 4), and changing “every day” to “every day but Sunday” (Example 3). Such edits requires a deep level of reading comprehension not currently achieved by text generation models.
Error analysis also reveals that human-created contradictions are more deceiving. As shown in Figure 4, for each real context, we pair it with five fake contexts produced by five different methods. We then analyze the source (which fake context) of the incorrect answer when the RoBERTa-large QA model makes an error. If all five methods create equally deceiving fake contexts, we expect to observe a uniform distribution. However, the distribution in Figure 4 shows that the most (31.8%) wrong answers are extracted from the human-created fake context.
BART-FG creates more deceiving contradictions at the cost of more edits. As the number of iterations K increases, the BART-FG model makes more edits to the original contexts, as reflected by the increasing Edit in Table 3. The resultant QA performance also falls, as more contradicting information is likely to appear when more edits are made. Therefore, generation models can produce more deceiving fake contexts by making more edits to the real contexts (certainly worrying from a dual-use perspective). However, we will show in Section 4.5 that more edits make the generated texts less realistic and more easy to detect.

heading: 4.4 BART-FG VERSUS GPT2

text:
Is GPT2 also good at generating deceiving fake contexts? To investigate this, we introduce a baseline — namely GPT2-FG— to compare against our proposed BART-FG model. GPT2-FG applies the pretrained GPT2-large model from Hugging Face to generate the rest of the contexts given the first 20% of the real contexts CR as the prompt.
Setting Detector Accuracy EM score before → after adding detector
Human 72.86% 72.02 → 78.92 (K=1) 78.04% 81.40 → 85.26 (K=2) 84.17% 78.85 → 85.95 (K=3) 90.10% 77.51 → 85.91
Figure 6: Independent evaluation of the detector accuracy for fake contexts generated by different methods and the benefits to QA models. The second to fourth rows denote the BART-FG model with different parameter K.
We find that BART-FG is able to create more contradictions with less edits compared with the GPT2-FG baseline (Table 3). We attribute this to the iterative modification strategy of BART-FG, which selectively replaces text spans that convey key information. In contrast, while GPT2-FG generates the whole passage without any explicit control except for the given prompt. This often makes the generated contents deviate the original topic, yielding fewer contradictions. The error analysis in Figure 4 also shows that GPT2-FG generates the least deceiving fake contexts compared with human and BART-FG.

heading: 4.5 EVALUATION OF MISINFORMATION DETECTION

text:
Finally, we evaluate effectiveness of integrating a misinformation detector of varying model architecture, and their sensitivity to training data size. Figure 5 shows the EM score achieved by the RoBERTa-large QA model after incorporating different detectors. While preliminary, the results show that we can train an effective detector only when we have sufficient number of in-domain labeled real/fake contexts. The benefit of the detector becomes quite limited given with insufficient training data (e.g., +4.0% with 500 training samples). This reveals the difficulty of defending against misinformation in the real world: ideally a good detector helps, but we usually do not have large-scale in-domain labeled data to train an effective detector.
Through a separate evaluation for different types of fake contexts, we further show in Figure 6 that human-written misinformation has the lowest detection accuracy, showing that they are more difficult to detect than the machine-generated misinformation. This further validates the observation in Section 4.3 that humans can create subtle misinformation that require a high-level understanding. For BART-FG, we find a trade-off between contradiction power and realism. As K increases, the model makes more edits, creating more contradicting information, which lowers the EM score for QA. However, more edits will make the generated fake contexts less realistic, leaving more of a “trace” for the detector to track, thus increasing the fake detection accuracy. When a detector is applied, these two factors cancel out and give us a similar EM score of around 85.

heading: 5 CONCLUSION AND FUTURE WORK

text:
We study the potential threat of misinformation on question answering models by creating a largescale dataset CONTRAQA, containing over 10K human-written and model-generated contradicting contexts that are mixed with both real and fake information. Our studies reveal that QA models are indeed vulnerable under contradicting contexts. While integrating a misinformation detector into the QA system mitigates the problem, this solution requires the labeling of large-scale real/fake paragraphs which may not be feasible nor generalizable. We believe urgent further work is required to study this problem under the more realistic open-domain QA setting, to propose more effective counter-measures to build a robust misinformation-aware QA system. We make our dataset and codes publicly available to further this important agenda.
ETHICS STATEMENT
We plan to publicly release the CONTRAQA dataset and open-source the code and model weights for our BART-FG model. We note that open-sourcing the BART-FG model may bring the potential for deliberate misuse to generate disinformation for harmful applications. Since our CONTRAQA dataset contains large-scale human-written and model-generated fake contexts, it can also be misused to generate disinformation. We deliberated carefully on the reasoning for open-sourcing and share here our three reasons for publicly releasing our work.
First, the danger of BART-FG in generating disinformation is limited. Disinformation is a subset of misinformation that is spread deliberately to deceive. Although we utilize the innate “hallucination” ability of current pretrained language models to create misinformation, our model are not specialized to generate harmful disinformation such as hoaxes, rumors, or false propaganda. Instead, our model focuses on generating conflicting information by iteratively editing the original passage to test the ability of QA models to handle contradictory information.
Second, our model is based on the open-sourced BART model, which makes our model easy to replicate even without the released code. Given the fact that our model is a revised version of an existing publicly available model, it is unnecessary to conceal code or model weights.
Third, our decision to release follows the similar stance of the full release of another strong detector and state-of-the-art generator of neural fake news: Grover (Zellers et al., 2019)4. The authors claim that to defend against potential threats, we need threat modeling, in which a crucial component is a strong generator or simulator of the threat. In our work, we build an effective threat model for QA under misinformation. Followup research can build on our model transparency, further enhancing the threat model.

heading: A FIGURE ILLUSTRATION OF EXPERIMENTAL SETTINGS

heading: B HUMAN ANNOTATION GUIDELINE

text:
B.1 JOB DESCRIPTION
Given a paragraph from Wikipedia, modify some information in the paragraph to create a fake version of it. Here are the general requirements:
• You should make at least M edits at different places, where M is determined by the length of the passage and will show on the screen when you annotate each passage.
• You should make at least one long edit that rewrites at least half of a sentence. • The edits should modify key information to make it contradict with the original, such as time, location, purpose, outcome, reason, etc.
• The modified paragraph should be fluent and look realistic, without commonsense errors.
Under review as a conference paper at ICLR 2022 Annotation Guideline Job Description Given a paragraph from Wikipedia, modify some information in the paragraph to create a fake version of it. For example, given the following passage:
Modify some key information of it to create the following fake version:
Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50.
Super Bowl 50 was the 48th Super Bowl Game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion San Francisco 49ers defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title. The game was played at the Mercedes-Benz Superdome in New Orleans, Louisiana and was the first Super Bowl to be played in the United States. As this was the NFL's 48th Super Bowl, the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming Super Bowls with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the game would be known as the "Super Bowl of the Century".
B.2 DETAILED REQUIREMENTS
Here we give an example of modification as follows.
Detailed annotation instructions are as follows.
1) At least make N edits at different places. In the above example, there are a total of 5 edits:
• “an American football game”→ “the 48th Super Bowl Game” • “Denver Broncos”→ “San Francisco 49ers” • “on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara,
California.” → “Mercedes-Benz Superdome in New Orleans, Louisiana and was the first Super Bowl to be played in the United States.”
• “the 50th”→ “the NFL’s 48th” • “so that the logo could prominently feature the Arabic numerals 50.” → “so that the game
would be known as the ”Super Bowl of the Century.”
2) There should be at least one long edit. Among all your edits, there should be at least one long edit, which rewrites the whole sentence or at least half of the sentence.
In the above example, the long edit is: “on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California.” → “Mercedes-Benz Superdome in New Orleans, Louisiana and was the first Super Bowl to be played in the United States.”
3) The edits should create contradicting information. After your edits, the original passage and the modified passage should have contradicting information. One way to test it is that: when you ask questions about your modified information, the original passage and the modified passage gives contradicting answers.
For example: after you edit “Denver Broncos” to “San Francisco 49ers”, the original and modified passages are shown in the Figure below:
Original Text:
The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title.
Modified Text:
The American Football Conference (AFC) champion San Francisco 49ers defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title.
When you ask the question: “Which NFL team won Super Bowl 50?”, the original passage gives you the answer “Denver Broncos”, and the modified passage gives you the answer “San Francisco 49ers”. This is a contradiction.
Another example is the following edit: “so that the logo could prominently feature the Arabic numerals 50.” → “so that the game would be known as the “Super Bowl of the Century”.
Original Text:
... the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50.
Modified Text:
... the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the game would be known as the "Super Bowl of the Century".
When you ask the question: “Why the league suspended the tradition of naming Super Bowls with Roman numerals?” the original passage and the modified passage also give you contradicting answers.
However, the following passage does NOT create any contradiction, because the modified information is just a paraphrasing of the original information.
Original Text:
The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title.
Modified Text:
The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to win the Super Bowl.
4) The edits should modify important information in the passage. Your edits should focus on important information in the passage, i.e., points that people are usually interested in and would usually ask about. For example, time, location, purpose, outcome, reason, etc. Please avoid editing trivial and unimportant details.
For example, the following trivial edit is not supported:
Original Text:
the game would have been known as "Super Bowl L"...
Modified Text:
the game would have been known as "Super Bowl H"...
5) The modified passage should look “realistic”. The final modified passage should look “realistic”. Don’t make obvious logic or commonsense mistakes to make the reader easily know that this is a fake passage by simply going through it.
For example, the following edit is not supported.
Original Text:
The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California.
Modified Text:
The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at New York City, California.
People can easily tell the modified passage is fake since everybody knows that New York is not a city in California.
B.3 ANNOTATION INTERFACE
The original passage is shown on the left for your reference, you should modify the passage in the text box on the right to make the fake passage. After you finished the edits, Click “Submit”.


summaries: This paper frames cross-domain imitation learning as an optimal transport problem using the Gromov-Wasserstein distance. This problem is highly relevant to imitation learning settings where there is often substantial domain mismatch between action and state spaces, eg. a humanoid robot learning to walk from a human demonstrator. The paper introduces a reward function that can be optimised and proves that this is equivalent to minimising the Gromov-Wasserstein distance between state action occupancies of an agent and expert. Substantial discussion/ proofs are included to show that minimising the Gromov-Wasserstein distance is equivalent to recovering an optimal policy up to an isometry. This is both a blessing and a curse, as it allows for optimal policies to be recovered under extreme changes in domain or differences, but does mean that recovered policies could be entirely unsuitable due to isometry.   The paper is well written and concisely written, although does get excessively mathy at times, when a figure could be more helpful. Experimental results corroborate the proofs and propositions, and highlight the value of the proposed approach.

summaries: This paper presents a method to transfer policies between different MDPs based on the minimization of Gromov-Wasserstein distance. This distance provides a pseudo-reward that can be used to learn via RL the optimal policy in the target MDP given an optimal policy in the original MDP. The method is optimal if the MDPs can be mapped into each other through an isometry, but works also empirically in other cases.

summaries: In this paper, the authors focus on a more general cross domain imitation learning problem where only expert demonstrations from one domain is available. To solve such a problem, the authors use the Gromov-Wasserstein distance to align and compare states between tasks from different domains and propose a Gromov-Wasserstein Imitation Learning (GWIL).  They also show theoretically the possibilities and limitations of GWIL.

summaries: A method is proposed for cross-domain imitation learning, without resorting to any form of correspondence. This is done using a Gromov-Wasserstein distance between policies (in practice, Euclidiean distances on collected state-action pairs,  within a given domain), which finds isometric transformations that best preserve distance measures, between the two domains. Given an imitation domain and an expert domain with example trajectories, a pseudo-reward is computed  based on the degree to which the distances from a state to its neighbors in the imitation domain, are preserved in the expert domain. Given these pseudo-rewards, as computed for collected episodes, SAC is used as an RL algorithm to optimize the policy.  The paper contributes both a theoretical analysis and experiments with:  U-maze, pendulum-to-cart-and-pole, and half-cheetah-to-fallen-walker.

abstractText: Cross-domain imitation learning studies how to leverage expert demonstrations of one agent to train an imitation agent with a different embodiment or morphology. Comparing trajectories and stationary distributions between the expert and imitation agents is challenging because they live on different systems that may not even have the same dimensionality. We propose Gromov-Wasserstein Imitation Learning (GWIL), a method for cross-domain imitation that uses the GromovWasserstein distance to align and compare states between the different spaces of the agents. Our theory formally characterizes the scenarios where GWIL preserves optimality, revealing its possibilities and limitations. We demonstrate the effectiveness of GWIL in non-trivial continuous control domains ranging from simple rigid transformation of the expert domain to arbitrary transformation of the state-action space. 1

authors:
Pieter Abbeel
Andrew Y Ng

title: Apprenticeship learning via inverse reinforcement learning

venue: In Proceedings of the twenty-first international conference on Machine learning,

authors:
Pieter Abbeel
Adam Coates
Andrew Y Ng

title: Autonomous helicopter aerobatics through apprenticeship learning

venue: The International Journal of Robotics Research,

authors:
David Alvarez-Melis
Stefanie Jegelka
Tommi S. Jaakkola

title: Towards optimal transport with global invariances

venue: Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics,

authors:
Haitham Bou Ammar
Eric Eaton
Paul Ruvolo
Matthew E Taylor

title: Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment

venue: In Twenty-Ninth AAAI Conference on Artificial Intelligence,

authors:
Yusuf Aytar
Tobias Pfaff
David Budden
Tom Le Paine
Ziyu Wang
Nando de Freitas

title: Playing hard exploration games by watching youtube

venue: arXiv preprint arXiv:1805.11592,

authors:
Christopher Berner
Greg Brockman
Brooke Chan
Vicki Cheung
Przemysław Dębiak
Christy Dennison
David Farhi
Quirin Fischer
Shariq Hashme
Chris Hesse

title: Dota 2 with large scale deep reinforcement learning

venue: arXiv preprint arXiv:1912.06680,

authors:
Samuel Cohen
Giulia Luise
Alexander Terenin
Brandon Amos
Marc Deisenroth

title: Aligning time series on incomparable spaces

venue: In International Conference on Artificial Intelligence and Statistics,

authors:
Robert Dadashi
Léonard Hussenot
Matthieu Geist
Olivier Pietquin

title: Primal wasserstein imitation learning

venue: arXiv preprint arXiv:2006.04678,

authors:
Sira Ferradans
Nicolas Papadakis
Julien Rabin
Gabriel Peyré
Jean-François Aujol

title: Regularized discrete optimal transport

venue: In International Conference on Scale Space and Variational Methods in Computer Vision,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

venue: Advances in neural information processing systems,

authors:
Abhishek Gupta
Coline Devin
YuXuan Liu
Pieter Abbeel
Sergey Levine

title: Learning invariant feature spaces to transfer skills with reinforcement learning

venue: arXiv preprint arXiv:1703.02949,

authors:
Tuomas Haarnoja
Aurick Zhou
Kristian Hartikainen
George Tucker
Sehoon Ha
Jie Tan
Vikash Kumar
Henry Zhu
Abhishek Gupta
Pieter Abbeel

title: Soft actor-critic algorithms and applications

venue: arXiv preprint arXiv:1812.05905,

authors:
Nicolas Heess
Dhruva TB
Srinivasan Sriram
Jay Lemmon
Josh Merel
Greg Wayne
Yuval Tassa
Tom Erez
Ziyu Wang
SM Eslami

title: Emergence of locomotion behaviours in rich environments

venue: arXiv preprint arXiv:1707.02286,

authors:
Donald Hejna
Lerrel Pinto
Pieter Abbeel

title: Hierarchically decoupled imitation for morphological transfer

venue: In International Conference on Machine Learning,

authors:
Jonathan Ho
S. Ermon

title: Generative adversarial imitation learning

venue: In NIPS,

authors:
Kuno Kim
Yihong Gu
Jiaming Song
Shengjia Zhao
Stefano Ermon

title: Domain adaptive imitation learning

authors:
Jens Kober
J Andrew Bagnell
Jan Peters

title: Reinforcement learning in robotics: A survey

authors:
Sergey Levine
Chelsea Finn
Trevor Darrell
Pieter Abbeel

title: End-to-end training of deep visuomotor policies

venue: The Journal of Machine Learning Research,

authors:
Fangchen Liu
Zhan Ling
Tongzhou Mu
Hao Su

title: State alignment-based imitation learning

venue: arXiv preprint arXiv:1911.10947,

authors:
YuXuan Liu
Abhishek Gupta
Pieter Abbeel
Sergey Levine

title: Imitation from observation: Learning to imitate behaviors from raw video via context translation

venue: IEEE International Conference on Robotics and Automation (ICRA),

title: Gromov–wasserstein distances and the metric approach to object matching

venue: Foundations of computational mathematics,

title: Dynamic time warping. Information retrieval for music and motion, pp

authors:
Andrew Y. Ng
Stuart J. Russell

title: Algorithms for inverse reinforcement learning

venue: In Proceedings of the Seventeenth International Conference on Machine Learning,

authors:
Georgios Papagiannis
Yunpeng Li

title: Imitation learning with sinkhorn distances

venue: arXiv preprint arXiv:2008.09167,

authors:
Xue Bin Peng
Pieter Abbeel
Sergey Levine
Michiel van de Panne

title: Deepmimic: Exampleguided deep reinforcement learning of physics-based character skills

venue: ACM Transactions on Graphics (TOG),

authors:
Gabriel Peyré
Marco Cuturi
Justin Solomon

title: Gromov-wasserstein averaging of kernel and distance matrices

title: Alvinn: An autonomous land vehicle in a neural network

title: Efficient training of artificial neural networks for autonomous navigation

venue: Neural Computation,

title: Markov decision processes: discrete stochastic dynamic programming

authors:
Dripta S Raychaudhuri
Sujoy Paul
Jeroen van Baar
Amit K Roy-Chowdhury

title: Cross-domain imitation from observations

venue: arXiv preprint arXiv:2105.10037,

authors:
Bradly C Stadie
Pieter Abbeel
Ilya Sutskever

title: Third-person imitation learning

venue: arXiv preprint arXiv:1703.01703,

authors:
Richard S Sutton
Andrew G Barto

title: Reinforcement learning: An introduction

venue: MIT press,

authors:
Emanuel Todorov
Tom Erez
Yuval Tassa

title: Mujoco: A physics engine for model-based control

venue: In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems,

title: Optimal transport: old and new, volume 338

authors:
Oriol Vinyals
Igor Babuschkin
Wojciech M Czarnecki
Michaël Mathieu
Andrew Dudzik
Junyoung Chung
David H Choi
Richard Powell
Timo Ewalds
Petko Georgiev

title: Grandmaster level in starcraft ii using multi-agent reinforcement learning

authors:
Kevin Zakka
Andy Zeng
Pete Florence
Jonathan Tompson
Jeannette Bohg
Debidatta Dwibedi

title: Xirl: Cross-embodiment inverse reinforcement learning

venue: arXiv preprint arXiv:2106.03911,

authors:
Feng Zhou
Fernando Torre

title: Canonical time warping for alignment of human behavior

authors:
Yuke Zhu
Ziyu Wang
Josh Merel
Andrei Rusu
Tom Erez
Serkan Cabi
Saran Tunyasuvunakool
János Kramár
Raia Hadsell
Nando de Freitas

title: Reinforcement and imitation learning for diverse visuomotor skills

venue: arXiv preprint arXiv:1802.09564,

heading: 1 INTRODUCTION

text:
Reinforcement learning (RL) methods have attained impressive results across a number of domains, e.g., Berner et al. (2019); Kober et al. (2013); Levine et al. (2016); Vinyals et al. (2019). However, the effectiveness of current RL method is heavily correlated to the quality of the training reward. Yet for many real-world tasks, designing dense and informative rewards require significant engineering effort. To alleviate this effort, imitation learning (IL) proposes to learn directly from expert demonstrations. Most current IL approaches can be applied solely to the simplest setting where the expert and the agent share the same embodiment and transition dynamics that live in the same state and action spaces. In particular, these approaches require expert demonstrations from the agent domain. Therefore, we might reconsider the utility of IL as it seems to only move the problem, from designing informative rewards to providing expert demonstrations, rather than solving it. However, if we relax the constraining setting of current IL methods, then natural imitation scenarios that genuinely alleviate engineering effort appear. Indeed, not requiring the same dynamics would enable agents to imitate humans and robots with different morphologies, hence widely enlarging the applicability of IL and alleviating the need for in-domain expert demonstrations.
This relaxed setting where the expert demonstrations comes from another domain has emerged as a budding area with more realistic assumptions (Gupta et al., 2017; Liu et al., 2019; Sermanet et al., 2018; Kim et al., 2020; Raychaudhuri et al., 2021) that we will refer to as Cross-Domain Imitation Learning. A common strategy of these works is to learn a mapping between the expert and agent domains. To do so, they require access to proxy tasks where both the expert and the agent act optimally in there respective domains. Under some structural assumptions, the learned map enables to transform a trajectory in the expert domain into the agent domain while preserving the optimality. Although these methods indeed relax the typical setting of IL, requiring proxy tasks heavily restrict the applicability of Cross-Domain IL. For example, it rules out imitating an expert never seen before as well as transferring to a new robot.
In this paper, we relax the assumptions of Cross-Domain IL and propose a benchmark and method that do not need access to proxy tasks. To do so, we depart from the point of view taken by previous work and formalize Cross-Domain IL as an optimal transport problem. We propose a method, that we call Gromov Wasserstein Imitation Learning (GWIL), that uses the Gromov-Wasserstein distance
1Project site with videos: https://sites.google.com/view/gwil/home
to solve the benchmark. We formally characterize the scenario where GWIL preserves optimality (theorem 1), revealing the possibilities and limitations. The construction of our proxy rewards to optimize optimal transport quantities using RL generalizes previous work that assumes uniform occupancy measures (Dadashi et al., 2020; Papagiannis & Li, 2020) and is of independent interest. Our experiments show that GWIL learns optimal behaviors with a single demonstration from another domain without any proxy tasks in non-trivial continuous control settings.

heading: 2 RELATED WORK

text:
Imitation learning. An early approach to IL is Behavioral Cloning (Pomerleau, 1988; 1991) which amounts to training a classifier or regressor via supervised learning to replicate the expert’s demonstration. Another key approach is Inverse Reinforcement Learning (Ng & Russell, 2000; Abbeel & Ng, 2004; Abbeel et al., 2010), which aims at learning a reward function under which the observed demonstration is optimal and can then be used to train a agent via RL. To bypass the need to learn the expert’s reward function, Ho & Ermon (2016) show that IRL is a dual of an occupancy measure matching problem and propose an adversarial objective whose optimization approximately recover the expert’s state-action occupancy measure, and a practical algorithm that uses a generative adversarial network (Goodfellow et al., 2014). While a number of recent work aims at improving this algorithm relative to the training instability caused by the minimax optimization, Primal Wasserstein Imitation Learning (PWIL) (Dadashi et al., 2020) and Sinkhorn Imitation Learning (SIL) (Papagiannis & Li, 2020) view IL as an optimal transport problem between occupancy measures to completely eliminate the minimax objective and outperforms adversarial methods in terms of sample efficiency. Heess et al. (2017); Peng et al. (2018); Zhu et al. (2018); Aytar et al. (2018) scale imitation learning to complex human-like locomotion and game behavior in non-trivial settings. Our work is an extension of Dadashi et al. (2020); Papagiannis & Li (2020) from the Wasserstein to the GromovWasserstein setting. This takes us beyond limitation that the expert and imitator are in the same domain and into the cross-domain setting between agents that live in different spaces.
Transfer learning across domains and morphologies. Work transferring knowledge between different domains in RL typically learns a mapping between the state and action spaces. Ammar et al. (2015) use unsupervised manifold alignment to find a linear map between states that have similar local geometry but assume access to hand-crafted features. More recent work in transfer learning
across viewpoint and embodiment mismatch learn a state mapping without handcrafted features but assume access to paired and time-aligned demonstration from both domains (Gupta et al., 2017; Liu et al., 2018; Sermanet et al., 2018). Furthermore, Kim et al. (2020); Raychaudhuri et al. (2021) propose methods to learn a state mapping from unpaired and unaligned tasks. All these methods require proxy tasks, i.e. a set of pairs of expert demonstrations from both domains, which limit the applicability of these methods to real-world settings. Stadie et al. (2017) have proposed to combine adversarial learning and domain confusion to learn a policy in the agent’s domain without proxy tasks but their method only works in the case of small viewpoint mismatch. Zakka et al. (2021) take a goal-driven perspective that seeks to imitate task progress rather than match fine-grained structural details to transfer between physical robots. In contrast, our method does not rely on learning an explicit cross-domain latent space between the agents, nor does it rely on proxy tasks. The GromovWasserstein distance enables us to directly compare the different spaces without a shared space. The existing benchmark tasks we are aware of assume access to a set of demonstrations from both agents whereas the experiments in our paper only assume access to expert demonstrations.

heading: 3 PRELIMINARIES

text:
Metric Markov Decision Process. An infinite-horizon discounted Markov decision Process (MDP) is a tuple (S,A,R, P, p0, γ) where S and A are state and action spaces, P : S × A → ∆(S) is the transition function, R : S×A→ R is the reward function, p0 ∈ ∆(S) is the initial state distribution and γ is the discount factor. We equip MDPs with a distance d : S × A → R+ and call the tuple (S,A,R, P, p0, γ, d) a metric MDP.
Gromov-Wasserstein distance. Let (X , dX , µX ) and (Y, dY , µY) be two metric measure spaces, where dX and dY are distances, and µX and µY are measures on their respective spaces2. The Gromov-Wasserstein distance (Mémoli, 2011) extends the Wasserstein distance from optimal transportation (Villani, 2009) to these spaces and is defined as
GW((X , dX , µX ), (Y, dY , µY))2 = min u∈U(µX ,µY) ∑ X 2×Y2 |dX (x, x′)− dY(y, y′)|2ux,yux′,y′ , (1)
where U(µX , µY) is the set of couplings between the atoms of the measures defined by
U(µX , µY) = u ∈ RX×Y ∣∣∣∣∣∣ ∀x ∈ X , ∑ y∈Y ux,y = µX (x),∀y ∈ Y, ∑ x∈X ux,y = µY(y)  . GW compares the structure of two metric measure spaces by comparing the pairwise distances within each space to find the best isometry between the spaces. Figure 1 illustrates this distance in the case of the metric measure spaces (SE ×AE , dE , ρπE ) and (SA ×AA, dA, ρπA).

heading: 4 CROSS-DOMAIN IMITATION LEARNING VIA OPTIMAL TRANSPORT

heading: 4.1 COMPARING POLICIES FROM ARBITRARILY DIFFERENT MDPS

text:
For a stationary policy π acting on a metric MDP (S,A,R, P, γ, d), the occupancy measure is:
ρπ : S ×A→ R ρ(s, a) = π(a|s) ∞∑ t=0 γtP (st = s|π).
We compare policies from arbitrarily different MDPs in terms of their occupancy measures. Definition 1 (Gromov-Wasserstein distance between policies). Given an expert policy πE and an agent policy πA acting, respectively, on
ME = (SE , AE , RE , PE , TE , dE) and MA = (SA, AA, RA, PA, TA, dA).
We define the Gromov-Wasserstein distance between πE and πA as the Gromov-Wasserstein distance between the metric measure spaces (SE ×AE , dE , ρπE ) and (SA ×AA, dA, ρπA):
GW(π, π′) = GW((SE ×AE , dE , ρπE ), (SA ×AA, dA, ρπA)). (2) 2We use discrete spaces for readability but show empirical results in continuous spaces.
We now define an isometry between policies by comparing the distances between the state-action spaces and show that GW defines a distance up to an isometry between the policies. Figure 2 illustrates examples of simple isometric policies. Definition 2 (Isometric policies). Two policies πE and πA are isometric if there exists a bijection φ : supp[ρπE ]→ supp[ρπA ] that satisfies for all (sE , aE), (sE ′, aE ′) ∈ supp[ρπE ]2:
dE ((sE , aE), (sE ′, aE ′)) = dA (φ(sE , aE), φ(sE ′, aE ′))
In other words, φ is an isometry between (supp[ρπE ], dE) and (supp[ρπA ], dA). Proposition 1. GW defines a metric on the collection of all isometry classes of policies.
Proof. By definition 1, GW(πE , πA) = 0 if and only if GW((SE , dE , ρπE ), (SA, dA, ρπA)) = 0. By Mémoli (2011, Theorem 5.1), this is true if and only if there is an isometry that maps supp[ρφE ] to supp[ρφA ]. By definition 2, this is true if and only if πA and πE are isometric. The symmetry and triangle inequality follow from Mémoli (2011, Theorem 5.1).
The next theorem3 gives a sufficient condition to recover, by minimizing GW , an optimal policy4 in the agent’s domain up to an isometry. Theorem 1. Consider two MDPs
ME = (SE , AE , RE , PE , pE , γ) and MA = (SA, AA, RA, PA, pA, γ).
Suppose that there exists four distances dSE , d A E , d S A, d A A defined on SE ,AE , SA andAE respectively, and two isometries φ : (SE , dSE) → (SA, dSA) and ψ : (AE , dSE) → (AS , dSA) such that for all (sE , aE , s ′ E) ∈ SE ×AE × SE the three following conditions hold:
R(sE , aE) = RA(φ(sE), ψ(aE)) (3)
PEsE ,aE (s ′ E) = PAφ(sE)ψ(aE)(φ(s ′ E)) (4)
pE(sE) = pA(φ(sE)). (5) Consider an optimal policy π∗E in ME . Suppose that πGW minimizes GW(π∗E , πGW ) with
dE : (sE , aE) 7→ dSE(sE) + dAE(aE) and dA : (sA, aA) 7→ dSA(sA) + dAA(aA). Then πGW is isometric to an optimal policy in MA.
Proof. Consider the occupancy measure ρ∗A : SA ×AA → R given by (sA, aA) 7→ ρπ∗E (φ −1(sA), ψ −1(aA)). We first show that ρ∗A is feasible in MA, i.e. there exists a policy π ∗ A acting in MA with occupancy measure ρ∗A (a). Then we show that π ∗ A is optimal in MA (b) and is isometric to π ∗ E (c). Finally we show that πGW is isometric to π∗A, which concludes the proof (d). (a) Consider sA ∈ SA. By definition of ρ∗A,∑ aA∈AA ρ∗A(sA) = ∑ aA∈AA ρπ∗E (φ −1(sA), ψ −1(aA)) = ∑ aE∈AE ρπ∗E (φ −1(sA), aE). Since ρπ∗E is feasible in M , it follows from Puterman (2014, Theorem 6.9.1) that∑ aE∈AE ρπ∗E (φ −1(sA), aE) = pE(φ −1(sA)) + γ ∑ sE∈SE ,aE∈AE PEsE ,aE (φ −1(sA)) + ρπ∗E (sE , aE). By conditions 4 and 5 and by definition of ρ∗A,
pE(φ −1(sA)) + γ ∑ sE∈SE ,aE∈AE PEsE ,aE (φ −1(sA)) + ρπ∗E (sE , aE)
= pA(sA) + γ ∑
sE∈SE ,aE∈AE
PAφ(sE),ψ(aE)(sA) + ρ ∗ A(φ(sE), ψ(aE))
= pA(sA) + γ ∑
s′A∈SA,aA∈AA
PAs′A,aA(sA) + ρ ∗ A(s ′ A, aA).
3Our proof is in finite state-action spaces for readability and can be directly extended to infinite spaces. 4A policy is optimal in the MDP (S,A,R, P, γ, d) if it maximizes the expected return E ∑∞ t=0R(st, at).
It follows that∑ aA∈AA ρ∗A(sA) = pA(sA) + γ ∑
s′A∈SA,aA∈AA
PAs′A,aA(sA) + ρ ∗ A(s ′ A, aA).
Therefore, by Puterman (2014, Theorem 6.9.1), ρ∗A is feasible in MA, i.e. there exists a policy π ∗ A acting in MA with occupancy measure ρ∗A.
(b) By condition 5 and definition of ρ∗A, the expected return of π ∗ A in MA is then∑
sA∈SA,aA∈AA
ρ∗A(sA, aA)RA(sA, aA)
= ∑
sA∈SA,aA∈AA
ρ∗E(φ −1(sA), ψ −1(aA))RE(φ −1(sA), ψ −1(aA))
= ∑
sE∈SE ,aE∈AE
ρ∗E(sE , aE)RE(sE , aE)
Consider any policy πA in M ′. By condition 5, the expected return of πA is∑ sA∈SA,aA∈AA ρπA(sA, aA)RA(sA, aA) = ∑ sE∈SE ,aE∈AE ρπA(φ(sE), ψ(aE))RE(sE , aE).
Using the same arguments that we used to show that ρ∗A is feasible in M ′, we can show that
(sE , aE) 7→ ρπA(φ(sE), ψ(aE))
is feasible in M . It follows by optimality of π∗E in M that∑ sE∈SE ,aE∈AE ρπA(φ(sE), ψ(aE))RE(sE , aE) ≤ ∑ sE∈SE ,aE∈AE ρπ∗E (φ(sE), ψ(aE))RE(sE , aE)
= ∑
sA∈SA,aA∈AA
ρ∗A(sA, aA)RA(sA, aA).
It follows that π∗A is optimal in M ′.
(c) Notice that ξ : (sE , aE) 7→ (φ(sE), ψ(aE))
is an isometry between (SE ×AE , dE) and (SA ×AA, dA), where dE and dA and given, resp., by
(sE , aE) 7→ dSE(sE) + dAE(aE) and (sA, aA) 7→ dSA(sA) + dAA(aA).
Therefore by definition of ρ∗A, π ∗ A is isometric to π ∗ E .
(d) Recall from the statement of the theorem that πGW is a minimizer of GW(π∗E , πGW ). Since π∗A is isometric to π∗E , it follows from prop. 1 that GW(π∗E , π∗A) = 0. Therefore GW(π∗E , πGW ) must be 0. By prop. 1, it follows that there exists an isometry
χ : (supp[ρ∗E ], dE)→ (supp[ρπGW ], dA).
Notice that χ ◦ ξ−1|supp[ρ∗A] is an isometry from (supp[ρ ∗ A], dA) to (supp[ρπGW ], dA). It follows that πGW is isometric to π∗A, an optimal policy in MA, which concludes the proof.
Remark 1. Theorem 1 shows the possibilities and limitations of our method. It shows that our method can recover optimal policies even though arbitrary isometries are applied to the state and action spaces of the expert’s domain. Importantly, we don’t need to know the isometries, hence our method is applicable to a wide range of settings. We will show empirically that our method produces strong results in other settings where the environment are not isometric and don’t even have the same dimension. However, a limitation of our method is that it recovers optimal policy only up to isometries. We will see that in practice, running our method on different seeds enables to find an optimal policy in the agent’s domain.
Algorithm 1 Gromov-Wasserstein imitation learning from a single expert demonstration. Inputs: expert demonstration τ , metrics on the expert (dE) and agent (dA) space Initialize the imitation agent’s policy πθ and value estimates Vθ while Unconverged do
Collect an episode τ ′ Compute GW(τ, τ ′) Set pseudo-rewards r with eq. (7) Update πθ and Vθ to optimize the pseudo-rewards
end while

heading: 4.2 GROMOV-WASSERSTEIN IMITATION LEARNING

text:
Minimizing GW between an expert and agent requires derivatives through the transition dynamics, which we typically don’t have access to. We introduce a reward proxy suitable for training an agent’s policy that minimizes GW via RL. Figure 1 illustrates the method. For readability, we combine expert state and action variables (sE , aE) into single variables zE , and similarly for agent state-action pairs. Also, we define ZE = SE ×AE and ZA = SA ×AA. Definition 3. Given an expert policy πE and an agent policy πA, the Gromov-Wasserstein reward of the agent is defined as rGW : supp[ρπA ]→ R given by
rGW(zA) = − 1
ρπ(zA) ∑ zE∈ZE z′E∈ZE z′A∈ZA |dE(zE , z′E))− dA(zA, z′A)|2u?zE ,zAu ? z′E ,z ′ A
where u? is the coupling minimizing objective 1.
Proposition 2. The agent’s policy πA trained with rGW minimizes GW(πE , πA). Proof. Suppose that πA maximizes E( ∑∞ t=0 γ trGW(s A t , a A t )) and denote by ρπA its occupancy measure. By Puterman (2014, Theorem 6.9.4), πA maximizes the following objective:
E zA∼ρπA
rGW(zA) =− ∑
zA∈supp[ρπA ]
ρπA(zA) ρπA(zA) ∑ zE∈ZE z′E∈ZE z′A∈ZA |dE(zE , z′E)− dA(zA, z′A)|2u?zA,zEu ? z′ A ,z′ E
=− ∑
zE∈ZE z′E∈ZE zA∈ZA z′A∈ZA
|dE(zE , z′E)− dA(zA, z′A)|2u?zA,zEu ? z′ A ,z′ E
=− GW2(πE , πA)
In practice we approximate the occupancy measures of π by ρ̂π(s, a) = 1T ∑T t=1 1(s = st∧a = at) where τ = (s1, a1, .., sT , aT ) is a finite trajectory collected with π. Assuming that all state-action pairs in the trajectory are different5, ρ̂ is a uniform distribution. Given an expert trajectory τE and an agent trajectory τA 6, the (squared) Gromov-Wasserstein distance between the empirical occupancy measures is
GW2(τE , τA) = min θ∈ΘTE×TA ∑ 1≤i,i′≤TE 1≤j,j′≤TA |dE((sEi , aEi ), (sEi′ , aEi′ ))− dA((sAj , sAj ), (sAj′ , aAj′))|2θi,jθi′,j′
(6)
5We can add the time step to the state to distinguish between two identical state-action pairs in the trajectory. 6Note that the Gromov-Wasserstein distance defined in equ. (6) does not depend on the temporal ordering
of the trajectories.
where Θ is the set of is the set of couplings between the atoms of the uniform measures defined by
ΘT×T ′ = θ ∈ RT×T ′ ∣∣∣∣∣∣ ∀i ∈ [T ], ∑ j∈[T ′] θi,j = 1/T, ∀j ∈ [T ′], ∑ i∈[T ] θi,j = 1/T ′  . In this case the reward is given for every state-action pairs in the trajectory by:
r(sAj , s A j ) = −TA ∑ 1≤i,i′≤TE 1≤j′≤TA |dE((sEi , aEi ), (sEi′ , aEi′ ))− dA((sAj , sAj ), (sAj′ , aAj′))|2θ?i,jθ?i′,j′ (7)
where θ? is the coupling minimizing objective 6.
In practice we drop the factor TA because it is the same for every state-action pairs in the trajectory.
Remark 2. The construction of our reward proxy is defined for any occupancy measure and extends to previous work optimizing optimal transport quantities via RL that assumes uniform occupancy measure in the form of a trajectory to bypass the need for derivatives through the transition dynamics (Dadashi et al., 2020; Papagiannis & Li, 2020).
Computing the pseudo-rewards. We compute the Gromov-Wasserstein distance using Peyré et al. (2016, Proposition 1) and its gradient using Peyré et al. (2016, Proposition 2). To compute the coupling minimizing 6, we use the conditional gradient method as discussed in Ferradans et al. (2013).
Optimizing the pseudo-rewards. The pseudo-rewards we obtain from GW for the imitation agent enable us to turn the imitation learning problem into a reinforcement learning problem (Sutton & Barto, 2018) to find the optimal policy for the Markov decision process induced by the pseudorewards. We consider agents with continuous state-action spaces and thus do policy optimization with the soft actor-critic algorithm (Haarnoja et al., 2018). Algorithm 1 sums up GWIL in the case where a single expert trajectory is given to approximate the expert occupancy measure.

heading: 5 EXPERIMENTS

text:
We propose a benchmark set for cross-domain IL methods consisting of 3 tasks and aiming at answering the following questions:
1. Does GWIL recover optimal behaviors when the agent domain is a rigid transformation of the expert domain? Yes, we demonstrate this with the maze in sect. 5.1.
2. Can GWIL recover optimal behaviors when the agent has different state and action spaces than the expert? Yes, we show in sect. 5.2 for slightly different state-action spaces between the cartpole and pendulum, and in sect. 5.3 for significantly different spaces between a walker and cheetah.
To answer these three questions, we use simulated continuous control tasks implemented in Mujoco (Todorov et al., 2012) and the DeepMind control suite (Tassa et al., 2018). In all settings we use the Euclidean metric within the expert and agent spaces for dE and dA.

heading: 5.1 AGENT DOMAIN IS A RIGID TRANSFORMATION OF THE EXPERT DOMAIN

text: We evaluate the capacity of IL methods to transfer to rigid transformation of the expert domain by using the PointMass Maze environment from Hejna et al. (2020). The agent’s domain is obtained by applying a reflection to the expert’s maze. This task satisfies the condition of theorem 1 with φ being the reflection through the central horizontal plan and ψ being the reflection through the x-axis in the action space. Therefore by theorem 1, the agent’s optimal policy should be isometric to the policy trained using GWIL. By looking at the geometry of the maze, it is clear that every policy in the isometry class of an optimal policy is optimal. Therefore we expect GWIL to recover an optimal policy in the agent’s domain. Figure 3 shows that GWIL indeed recovers an optimal policy.

heading: 5.2 AGENT AND THE EXPERT HAVE SLIGHTLY DIFFERENT STATE AND ACTION SPACES

text: We evaluate here the capacity of IL methods to transfer to transformation that does not have to be rigid but description map should still be apparent by looking at the domains. A good example of such transformation is the one between the pendulum and cartpole. The pendulum is our expert’s domain while cartpole constitutes our agent’s domain. The expert is trained on the swingup task. Even though the transformation is not rigid, GWIL is able to recover the optimal behavior in the agent’s domain as shown in fig. 4. Notice that pendulum and cartpole do not have the same state-action space dimension: The pendulum has 3 dimensions while the cartpole has 5 dimensions. Therefore GWIL can indeed be applied to transfer between problems with different dimension.

heading: 5.3 AGENT AND THE EXPERT HAVE SIGNIFICANTLY DIFFERENT STATE AND ACTION SPACES

text:
We evaluate here the capacity of IL methods to transfer to non-trivial transformation between domains. A good example of such transformation is two arbitrarily different morphologies from the DeepMind Control Suite such as the cheetah and walker. The cheetah constitutes our expert’s domain while the walker constitutes our agent’s domain. The expert is trained on the run task.
Although the mapping between these two domains is not trivial, minimizing the GromovWasserstein solely enables the walker to interestingly learn to move backward and forward by imitating a cheetah. Since the isometry class of the optimal policy – moving forward– of the cheetah and walker contains a suboptimal element –moving backward–, we expect GWIL to recover one of these two trajectories. Indeed, depending on the seed used, GWIL produces a cheetah-imitating walker moving forward or a cheetah-imitating walker moving backward, as shown in fig. 5.

heading: 6 CONCLUSION

text:
Our work demonstrates that optimal transport distances are a useful foundational tool for crossdomain imitation across incomparable spaces. Future directions include exploring:
1. Scaling to more complex environments and agents towards the goal of transferring the structure of many high-dimensional demonstrations of complex tasks into an agent.
2. The use of GW to help agents explore in extremely sparse-reward environments when we have expert demonstrations available from other agents.
3. How GW compares to other optimal transport distances that work apply between two metric MDPs, such as Alvarez-Melis et al. (2019), that have more flexibility over how the spaces are connected and what invariances the coupling has.
4. Metrics aware of the MDP’s temporal structure such as Zhou & Torre (2009); Cohen et al. (2021) that build on dynamic time warping (Müller, 2007). The Gromov-Wasserstein ignores the temporal information and ordering present within the trajectories.

heading: A OPTIMIZATION OF THE PROXY REWARD

text: In this section we show that the proxy reward introduced in equation 7 constitutes a learning signal that is easy to optimize using standards RL algorithms. Figure 6 shows proxy reward curves across 5 different seeds for the 3 environments. We observe that in each environment the SAC learner converges quickly and consistently to the asymptotic episodic return. Thus there is reason to think that the proxy reward introduced in equation 7 will be similarly easy to optimize in other crossdomain imitation settings.

heading: B TRANSFER TO SPARSE-REWARD ENVIRONMENTS

text: In this section we show that GWIL can be used to facilitate learning in sparse-reward environments when the learner has only access to one expert demonstration from another domain. We compare GWIL to a baseline learner having access to a single demonstration from the same domain and minimizing the Wasserstein distance, as done in Dadashi et al. (2020). In these experiments, both agents are given a sparse reward signal in addition to their respective optimal transport proxy reward. We perform experiments in two sparse-reward environment. In the first environment, the agent controls a point mass in a maze and obtain a non-zero reward only if it reaches the end of the maze. In the second environment, which is a sparse version of cartpole, the agent controls a cartpole and obtains a non-zero reward only if he can maintain the cartpole up for 10 consecutive time steps. Note that a SAC agent fails to learn any meaningful behavior in both environments. Figure 7 shows that GWIL is competitive with the baseline learner in the sparse maze environment even though GWIL has only access to a demonstration from another domain, while the baseline learner has access to a demonstration from the same domain. Thus there is reason to think that GWIL efficiently and reliably extracts useful information from the expert domain and hence should work well in other cross-domain imitation settings.

heading: C SCALABILITY OF GWIL

text:
In this section we show that our implementation of GWIL offers good performance in terms of wallclock time. Note that the bottleneck of our method is in the computation of the optimal coupling which only depends on the number of time steps in the trajectories, and not on the dimension of the expert and the agent. Hence our method naturally scales with the dimension of the problems. Furthermore, while we have not used any entropy regularizer in our experiments, entropy regularized methods have been introduced to enable Gromov-Wasserstein to scale to demanding machine learning tasks and can be easily incorporated into our code to further improve the scalability. Figure 8 compares the time taken by GWIL in the maze with the time taken by the baseline learner introduced in the previous section. It shows that imitating with Gromov-Wasserstein requires the same order of time than imitating with Wasserstein. Figure 9 compares the wall-clock time taken by a walker imitating a cheetah using GWIL to reach a walking speed (i.e., a horizontal velocity of 1) and the wall-clock time taken by a SAC walker trained to run. It shows that a GWIL walker imitating a cheetah reaches a walking speed faster than a SAC agent trained to run. Even though the SAC agent is optimizing for standing in addition to running, it was not obvious that GWIL could compete with SAC in terms of wall-clock time. These results gives hope that GWIL has the potential to scale to
more complex problems (possibly with an additional entropy regularizer) and be a useful way to learn by analogy.