summaries: This paper proposes a fine-tune technique to help BERT models to learn & capture form and content information on textual data (without any form of structural parsing needed). They key addition to the classic BERT model is the introduction of the R and S embeddings. R &S are supposed to learn the information in text that is traditionally represented as the structural positions and the content-bearing  symbols in those positions.

summaries: This paper proposes a layer on top of BERT which is motivated by a desire to disentangle content (meaning of the tokens) and form (structural roles of the tokens).  Figure 1 shows this clearly. The paper considers two variants of the disentangling layer (TPR), one with LSTMs (figure 2) and the other with attention (figure 3). The aim in both is to obtain a decomposition of the form x(t) = S a_s(v_t) a_r(v_t) R where S and R are shared matrices of parameters and v is the output of BERT.

abstractText: We introduce HUBERT1 which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional Transformer language model. We show that there is shared structure between different NLP datasets that HUBERT, but not BERT, is able to learn and leverage. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks.2

authors:
Nitin Bansal
Xiaohan Chen
Zhangyang Wang

title: Can we gain more from orthogonality regularizations in training deep cnns

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,

authors:
Samuel R Bowman
Gabor Angeli
Christopher Potts
Christopher D Manning

title: A large annotated corpus for learning natural language inference

venue: arXiv preprint arXiv:1508.05326,

authors:
K. Clark
U. Khandelwal
O. Levy
C.D. Manning

title: What Does BERT Look At? An Analysis of BERT’s Attention

authors:
Andy Coenen
Emily Reif
Ann Yuan
Been Kim
Adam Pearce
Fernanda Viégas
Martin Wattenberg

title: Visualizing and measuring the geometry of BERT

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: BERT: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
K. He
X. Zhang
S. Ren
J. Sun

title: Deep Residual Learning for Image Recognition

authors:
John Hewitt
Christopher D Manning

title: A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Qiuyuan Huang
Paul Smolensky
Xiaodong He
Li Deng
Dapeng Oliver Wu

title: Tensor product generation networks for deep NLP modeling

venue: Volume 1 (Long Papers),

authors:
Nitish Shirish Keskar
Bryan McCann
Caiming Xiong
Richard Socher

title: Unifying question answering and text classification via span extraction

authors:
D.P. Kingma
J. Ba

title: Adam: A Method for Stochastic Optimization

authors:
O. Kovaleva
A. Romanov
A. Rogers
A. Rumshisky

title: Revealing the Dark Secrets of BERT

authors:
Q.V. Le
N. Jaitly
G.E. Hinton

title: A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

authors:
Moontae Lee
Xiaodong He
Wen-tau Yih
Jianfeng Gao
Li Deng
Paul Smolensky

title: Reasoning in vector space: An exploratory study of question answering

venue: In 4th International Conference on Learning Representations,

authors:
Simon D Levy
Ross Gayler

title: Vector symbolic architectures: A new building material for artificial general intelligence

venue: In Proceedings of the 2008 Conference on Artificial General Intelligence

authors:
Yongjie Lin
Yi Chern Tan
Robert Frank

title: Open sesame: Getting inside BERT’s linguistic knowledge

venue: arXiv preprint arXiv:1906.01698,

authors:
Xiaodong Liu
Pengcheng He
Weizhu Chen
Jianfeng Gao

title: Multi-task deep neural networks for natural language understanding

venue: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

venue: arXiv preprint arXiv:1907.11692,

authors:
R Thomas McCoy
Ellie Pavlick
Tal Linzen

title: Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference

authors:
P. Michel
O. Levy
G. Neubig

title: Are Sixteen Heads Really Better than One

title: Physical symbol systems

venue: Cognitive science,

authors:
Hamid Palangi
Paul Smolensky
Xiaodong He
Li Deng

title: Question-answering with grammatically-interpretable representations

authors:
Jason Phang
Thibault Févry
Samuel R Bowman

title: Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks

venue: arXiv preprint arXiv:1811.01088,

title: Holographic reduced representations

venue: IEEE Transactions on Neural networks,

authors:
P. Rajpurkar
J. Zhang
K. Lopyrev
P. Liang

title: SQuAD: 100,000+ Questions for Machine Comprehension of Text

authors:
Pranav Rajpurkar
Jian Zhang
Konstantin Lopyrev
Percy Liang

title: Squad: 100, 000+ questions for machine comprehension of text

venue: In EMNLP,

authors:
Imanol Schlag
Jürgen Schmidhuber

title: Learning to reason with third order tensor products

venue: In Advances in Neural Information Processing Systems

title: Tensor product variable binding and the representation of symbolic structures in connectionist systems

venue: Artificial Intelligence,

authors:
Paul Smolensky
Géraldine Legendre

title: The Harmonic Mind: From Neural Computation to Optimality-Theoretic GrammarVolume I: Cognitive Architecture (Bradford Books)

authors:
I. Sutskever
O. Vinyals
Q.V. Le

title: Sequence to Sequence Learning with Neural Networks

authors:
Ian Tenney
Dipanjan Das
Ellie Pavlick

title: BERT rediscovers the classical NLP pipeline

venue: arXiv preprint arXiv:1905.05950,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: In Advances in neural information processing systems,

authors:
E. Vorontsov
C. Trabelsi
S. Kadoury
C. Pal

title: On orthogonality and learning recurrent networks with long term dependencies

authors:
Alex Wang
Amanpreet Singh
Julian Michael
Felix Hill
Omer Levy
Samuel R Bowman

title: Glue: A multi-task benchmark and analysis platform for natural language understanding

venue: arXiv preprint arXiv:1804.07461,

authors:
Alex Wang
Jan Hula
Patrick Xia
Raghavendra Pappagari
R Thomas McCoy
Roma Patel
Najoung Kim
Ian Tenney
Yinghui Huang
Katherin Yu

title: Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling

authors:
Jason Weston
Antoine Bordes
Sumit Chopra
Tomas Mikolov

title: Towards ai-complete question answering: A set of prerequisite toy tasks

authors:
Zhuosheng Zhang
Yuwei Wu
Hai Zhao
Zuchao Li
Shuailiang Zhang
Xi Zhou
Xiang Zhou

title: Semantics-aware bert for language understanding

heading: 1 INTRODUCTION

text:
Built on the Transformer architecture (Vaswani et al., 2017), the BERT model (Devlin et al., 2018) has demonstrated great power for providing general-purpose vector embeddings of natural language: its representations have served as the basis of many successful deep Natural Language Processing (NLP) models on a variety of tasks (e.g., Liu et al., 2019a;b; Zhang et al., 2019). Recent studies (Coenen et al., 2019; Hewitt & Manning, 2019; Lin et al., 2019; Tenney et al., 2019) have shown that BERT representations carry considerable information about grammatical structure, which, by design, is a deep and general encapsulation of linguistic information. Symbolic computation over structured symbolic representations such as parse trees has long been used to formalize linguistic knowledge. To strengthen the generality of BERT’s representations, we propose to import into its architecture this type of computation.
Symbolic linguistic representations support the important distinction between content and form information. The form consists of a structure devoid of content, such as an unlabeled tree, a collection of nodes defined by their structural positions or roles (Newell, 1980), such as root, left-child-of-root, right-child-of-left-child-of root, etc. In a particular linguistic expression such as “Kim referred to herself during the speech”, these purely-structural roles are filled with particular content-bearing symbols, including terminal words like Kim and non-terminal categories like NounPhrase. These role fillers have their own identities, which are preserved as they move from role to role across expressions: Kim retains its referent and its semantic properties whether it fills the subject or the object role in a sentence. Structural roles too maintain their distinguishing properties as their fillers change: the root role dominates the left-child-of-root role regardless of how these roles are filled.
Thus it is natural to ask whether BERT’s representations can be usefully factored into content × form, i.e., filler× role, dimensions. To answer this question, we recast it as: can BERT’s representations be usefully unpacked into Tensor-Product Representations (TPRs)? A TPR is a collection of constituents, each of which is the binding of a filler to a structural role. Specifically, we let BERT’s final-layer vector-encoding of each token of an input string be factored explicitly into a filler bound to a role: both the filler and the role are embedded in a continuous vector space, and they are bound together according to the principles defining TPRs: with the tensor product. This factorization effectively untangles the fillers from their roles, these two dimensions having been fully entangled in the BERT encoding itself. We then see whether disentangling BERT representations into TPRs facilitates their general use in a range of NLP tasks.
Concretely, as illustrated in Figure 1, we create HUBERT by adding a TPR layer on top of BERT; this layer takes the final-layer BERT embedding of each input token and transforms it into the tensor
1HUBERT is a recursive acronym for HUBERT Untangles BERT. 2Our code and models will be made available after publication.
product of a filler embedding-vector and a role embedding-vector. The model learns to separate fillers from roles in an unsupervised fashion, trained end-to-end to perform an NLP task.
If the BERT representations truly are general-purpose for NLP, the TPR re-coding should reflect this generality. In particular, the formal, grammatical knowledge we expect to be carried by the roles should be generally useful across a wide range of downstream tasks. We thus examine transfer learning, asking whether the roles learned in the service of one NLP task can facilitate learning when carried over to another task.
In brief, overall we find in our experiments on the NLP benchmarks of GLUE (Wang et al., 2018) and HANS (McCoy et al., 2019) that HUBERT’s recasting of BERT encodings as TPRs does indeed lead to effective knowledge transfer across NLP tasks, while the bare BERT encodings do not. Specifically, after pre-training on the MNLI dataset in GLUE, we observe positive gains ranging from 0.60% to 12.28% when subsequently fine-tuning on QNLI, QQP, RTE, SST, and SNLI tasks. This is due to transferring TPR knowledge—in particular the learned roles—relative to transferring just BERT parameters which have gains ranging from minus 0.33% to positive 2.53%.
Additionally, on average, we gain 5.7% improvement on the demanding non-entailment class of the HANS challenge dataset. Thus TPR’s disentangling of fillers from roles, motivated by the nature of symbolic representations, does yield more general deep linguistic representations as measured by cross-task transfer.
The paper is structured as follows. First we discuss the prior work on TPRs in deep learning and its previous applications in Section 2. We then introduce the model design in Section 3 and present our experimental results in Section 4. We conclude in Section 5.

heading: 2 RELATED WORK

text:
Building on the successes of symbolic AI and linguistics since the mid-1950s, there has been a long line of work exploiting symbolic and discrete structures in neural networks since the 1990s. Along with Holographic Reduced Representations (Plate, 1995) and Vector-Symbolic Architectures (Levy & Gayler, 2008), Tensor Product Representations (TPRs) provide the capacity to represent the discrete linguistic structure in a continuous, distributed manner, where grammatical form and semantic content can be disentangled (Smolensky, 1990; Smolensky & Legendre, 2006). In Lee et al. (2016), TPR-like representations were used to solve the bAbI tasks (Weston et al., 2016), achieving close to 100% accuracy in all but one of these tasks. Schlag & Schmidhuber (2018) also achieved success on the bAbI tasks, using third-order TPRs to encode and process knowledgegraph triples. In Palangi et al. (2018), a new structured recurrent unit (TPRN) was proposed to learn grammatically-interpretable representations using weak supervision from (context, question, answer) triplets in the SQuAD dataset (Rajpurkar et al., 2016). In Huang et al. (2018), unbinding operations of TPRs were used to perform image captioning. None of this previous work, however, examined the generality of learned linguistic knowledge through transfer learning.
Transfer learning for transformer-based models has been studied recently: Keskar et al. (2019) and Wang et al. (2019) report improvements in accuracy over BERT after training on an intermediate task from GLUE; an approach which has come to be known as Supplementary Training on Intermediate Labeled data Tasks (STILTs). However, as shown in more recent work (Phang et al., 2018), the results do not follow a consistent pattern when using different corpora for fine-tuning BERT, and degraded downstream transfer is often observed. Even for data-rich tasks like QNLI, regardless of the intermediate task and multi-tasking strategy, the baseline results do not improve. This calls for new model architectures with better knowledge transfer capability.

heading: 3 MODEL DESCRIPTION

text:
Applying the TPR scheme to encode the individual words (or sub-word tokens) fed to BERT, a word is represented as the tensor product of a vector embedding its content—its filler (or symbol)3 aspect—and a vector embedding the structural role it plays in the sentence. Given the results of
3We use filler or symbol to indicate the content symbolic representation throughout the paper.
Palangi et al. (2018), we expect the symbol to capture the semantic contribution of the word while the structural role captures its grammatical role:
x(t) = s(t) ⊗ r(t) (1)
Assuming we have nS symbols with dimension dS and nR roles with dimension dR, x(t) ∈ RdS×dR is the tensor representation for token t, s(t) ∈ RdS is the (presumably semantic) symbol representation and r(t) ∈ RdR is the (presumably grammatical) role representation for token t. s(t) may be either the embedding of one symbol or a linear combination of different symbols using a softmax symbol selector, and similarly for r(t). In other words, Eq. 1 can also be represented as x(t) = SB(t)R> where S ∈ RdS×nS and R ∈ RdR×nR are matrices the columns of which contain the global symbol and role embeddings, common for all tokens, and either learned from scratch or initialized by transferring from other tasks, as explained in Section 4. B(t) ∈ RnS×nR is the binding matrix which selects specific roles and symbols (embeddings) from R and S and binds them together. We assume that for a single-word representation, the binding matrix B(t) is rank 1, so we can decompose it into two separate vectors, one soft-selecting a symbol and the other a role, and rewrite equation (1) as x(t) = S(a(t)S a (t)> R )R > where a(t)R ∈ RnR and a (t) S ∈ RnS can respectively be interpreted as attention weights over different roles (columns of R) and symbols (columns of S). For each input token x(t), we get its contextual representations of grammatical role (a(t)R ) and semantic symbol (a(t)S ) by fusing the contextual information from the role and symbol representations of its surrounding tokens.
We explore two options for mapping the input token from the current time-step, and the tensor representation from the previous time-step, to a(t)R and a (t) S : a Long Short-Term Memory (LSTM) architecture (Hochreiter & Schmidhuber, 1997) and a one-layer Transformer. All the models share the general architecture depicted in Figure 1 except for BERT and BERT-LSTM, where the TPR layer is absent. Our conclusion based on initial experiments was that the Transformer layer results in better integration and homogeneous combination with the other Transformer layers in BERT, as will be described shortly.
The TPR layer with LSTM architecture works as follows (see also Figure 2, discussed in Sec. 4.2). We calculate the hidden states (h(t)S , h (t) R ) and cell states (c (t) S ,c (t) R ) and for each time-step according to the following equations:
h (t) S , c (t) S = LSTMS(vt, (vec(x (t)), c (t−1) S )); h (t) R , c (t) R = LSTMR(vt, (vec(x (t)), c (t−1) R )) (2)
where vt is the final-layer BERT embedding of the t-th word, and vec(.) flattens the input tensor into a vector. Each LSTM’s input cell state is the previous LSTM’s cell output state. Each LSTM’s input hidden state, however, is calculated by binding the previous cell’s role and symbol vectors.
In the TPR layer with Transformer architecture, we calculate the output representations (h(t)S , h (t) R ) using a Transformer Encoder layer:
h (t) S = TransformerS(vt); h (t) R = TransformerR(vt) (3)
Each Transformer layer consists of a multi-head attention layer, followed by a residual block (with dropout), a layer normalization block, a feed-forward layer, another residual block (with dropout), and a final layer normalization block. (See Figure 3, discussed in Sec. 4.2, and the original Transformer paper, Vaswani et al. (2017), for more details.)
Given that each word is usually assigned to a few grammatical roles and semantic concepts (ideally one), an inductive bias is enforced using a softmax temperature (T ) to make a(t)R and a (t) S sparse. Note that in the limit of very low temperatures, we will end up with one-hot vectors which pick only one filler and one role.4
a (t) S = softmax(WSh (t) S /T ); a (t) R = softmax(WRh (t) R /T ) (4)
Here WS and WR are linear-layer weights. For the final output of the transformer model, we explored different aggregation strategies to construct the final sentence embedding:
P (c|f) = softmax(WfAgg(x(0),x(1), ...,x(N))) (5)
where P (c|f) is a probability distribution over class labels, f is the final sentence representation, Wf is the classifier weight matrix, and N is the maximum sequence length. Agg(.) defines the merging strategy. We experimented with different aggregation strategies: max-pooling, meanpooling, masking all but the input-initial [CLS] token, and concatenating all tokens and projecting down using a linear layer. In Devlin et al. (2018), the final representation for the [CLS] token is used as the sentence representation. However, during our experiments, we observed better results when concatenating the final embeddings for all tokens and then projecting down to a smaller dimension, as this exposes the classifier to the full span of token information.
The formal symmetry between symbols and roles evident in Eq. 1 is broken in two ways.
First, we choose hyper-parameters so that the number of symbols is greater than the number of roles. Thus each role is on average used more often than each symbol, encouraging more general information (such as structural position) to be encoded in the roles, and more specific information (such as word semantics) to be encoded in the symbols. (This effect was evident in the analogous TPR learning model of Palangi et al. (2018).)
Second, to enable the symbol that fills any given role to be exactly recoverable from a TPR in which it appears along with other symbols, the role vectors should be linearly independent: this expresses the intuition that distinct structural roles carry independent information. Fillers, however, are not expected to be independent in this sense, since many fillers may have similar meanings and be quasi-interchangeable. So for the role matrix R, but not the filler matrix S, we add a regularization term to the training loss which encourages the R matrix to be orthogonal:
L = − ∑ c 1[c = c∗] logP (c|f) + λ(||RR> − IdR ||2F + ||R>R− InR ||2F ) (6)
HereL indicates the loss function, Ik is the identity matrix with k rows and k columns, and 1[.] is the indicator function: it is 1 when the predicted class c matches the correct class c∗ label, and 0 otherwise. Following the practice in Bansal et al. (2018) we use double soft orthogonality regularization to handle both over-complete and under-complete matrices R.
4Note that bias parameters are omitted for simplicity of presentation.

heading: 4 EXPERIMENTS

text:
We performed extensive experiments to answer the following questions:
1. Does adding a TPR layer on top of BERT (as in the previous section) impact its performance positively or negatively? We are specifically interested in MNLI for this experiment because it is large-scale compared to other GLUE tasks and is more robust to model noise (i.e., different randomly-initialized models tend to converge to the same final score on this task). This task is also used as the source task during transfer learning. This experiment is mainly a sanity check to verify that the specific TPR decomposition added does not hurt source-task performance.
2. Does transferring the BERT model’s parameters, fine-tuned on one of the GLUE tasks, help the other tasks in the Natural Language Understanding (NLU) benchmarks (Bowman et al., 2015; Wang et al., 2018)? Based on our hypothesis of the advantage of disentangling content from form, the learned symbols and roles should be transferable across natural language tasks. Does transferring role (R) and/or symbol (S) embeddings (described in the previous section) improve transfer learning on BERT across the GLUE tasks?
3. Is the ability to transfer the TPR layer limited to GLUE tasks? Can it be generalized? To answer this question we evaluated our models on a challenging diagnostic dataset outside of GLUE called HANS (McCoy et al., 2019).

heading: 4.1 DATASET

text:
We conduct three major experiments to answer the above questions: a comparison of architectures on the MNLI dataset, a study of transfer learning between GLUE tasks (Wang et al., 2018), and finally model diagnosis using HANS (McCoy et al., 2019); these are discussed in Sections 4.2, 4.3, and 4.4, respectively.
Section A.1 provides a more detailed analysis of the dataset. Table 4 (see Appendix) shows the dataset statistics and details for GLUE, SNLI, and HANS.

heading: 4.2 ARCHITECTURE COMPARISON ON MNLI

text:
Our experiments are done with four different model architectures. The general architecture of the models is depicted in Figure 1. The TPR layer is absent in BERT and BERT-LSTM. In the figure, the BERT model indicates the pre-trained off-the-shelf BERT base model which has 12 Transformer encoder layers. The aggregation layer computes the final sentence representation (see Eq. 5). The linear classifier is task-specific and is not shared between tasks during transfer learning.
BERT: This is our baseline model which consists of BERT, an aggregation layer on top, and a final linear classifier.
BERT-LSTM: We augment the BERT model by adding a unidirectional LSTM Recurrent layer (Hochreiter & Schmidhuber, 1997; Sutskever et al., 2014) on top. The inputs to the LSTM are token representations encoded by BERT. We then take the final hidden state of the LSTM and feed it into a classifier to get the final predictions. Since this model has an additional layer augmented on top of BERT, it can serve as a baseline for TPR models introduced below.
HUBERT (LSTM): We use two separate LSTM networks to compute symbol and role representation for each token. Figure 2 shows how the final token embedding (x(t)) is constructed at each time step: this plays the role of the LSTM hidden state h(t). (In the figures, ‘©∗ ’ denotes matrix-vector multiplication.) The results (Table 1) show that this decomposition improves the accuracy on MNLI compared to both the BERT and BERT-LSTM models. Training recurrent models is usually difficult, due to exploding or vanishing gradients, and has been studied for many years (Le et al., 2015; Vorontsov et al., 2017). With the introduction of the gating mechanism in LSTM and GRU cells, this problem was alleviated. In our model, we have a tensor-product operation which binds role and symbol vectors. We observed that during training the values comprising these vectors can reach very small numbers (< 10−4), and after binding, the final embedding vectors have values roughly in the order of 10−8. This makes it difficult for the classifier to distinguish between similar but different sentences. Additionally, backpropagation is not effective since the gradients are too small.
We avoided this problem by linearly scaling all values by a large number (∼ 1K) and making that scaling value trainable so that the model can adjust it for better performance.
HUBERT (Transformer): In this model, instead of using a recurrent layer, we deploy the power of Transformers (Vaswani et al., 2017) to encode roles and symbols (see Figure 3). This lets us attend to all the tokens when calculating a(k)R and a (k) S and thus better capture long-distance dependencies. It also speeds up training as all embeddings are computed in parallel for each sentence. Furthermore, it naturally solves the vanishing and exploding gradients problem, by taking advantage of residual blocks (He et al., 2015) to facilitate backpropagation and Layer Normalization (Lei Ba et al., 2016) to prohibit value shifts. It also integrates well with the rest of the BERT model and presents a more homogeneous architecture.5
We first do an architecture comparison study on the four models, each built on BERT (base model). We fine-tune BERT on the MNLI task, which we will then use as our primary source training task for testing transfer learning. We report the final accuracy on the MNLI development set.
Table 1 summarizes the results. Both HUBERT models are able to maintain the same performance as our baseline (BERT). This confirms that adding TPR heads will not degrade the model’s accuracy and can even improve it (in our case when evaluated on MNLI matched development set). Although HUBERT (Transformer) and HUBERT (LSTM) have roughly the same accuracy, we choose HUBERT (Transformer) to perform our transfer learning experiments, since it eliminates the limitations of HUBERT (LSTM) (as discussed above) and reduces the training and inference time significantly (> 4X).

heading: 4.3 TRANSFER LEARNING

text:
We compare the transfer-learning performance of HUBERT (Transformer) against BERT. We follow the same training procedure for each model and compare the final development set accuracy on the target corpus. The training procedure is as follows: For Baseline, we train three instances of each model on the target corpus and then select the one with the highest accuracy on target dev set (We vary the random seed and the order in which the training data is sampled for each instance.) These results are reported for each model in the Baseline Acc. column in Table 2. For Fine-tuned, in a separate experiment, we first fine-tune one instance of each model on the source corpus and use these
5The results reported here correspond to an implementation using an additional Transformer encoder layer on top of the TPR layer; we scale the input values to this layer only. Future versions of the model will omit this layer.
updated parameters to initialize a second instance of the same model. The initialized model will then be trained and tested on the target corpus. In this setting, we have three subsets of parameters to choose from when transferring values from the source model to the target model: BERT parameters, the Role embeddings R, and the Filler embeddings F . Each of these subsets can independently be transferred or not, leading to a total of 7 combinations excluding the option in which none of them are transferred. We chose the model which has the highest absolute accuracy on the target dev dataset. These results are reported for each model under Fine-tuned Acc. Note that the transferred parameters are not frozen, but updated during training on the target corpus.
MNLI as source: Table 2 summarizes the results for these transfer learning experiments when the source task is MNLI. Gain shows the difference between Fine-tuned model’s accuracy and Baseline’s accuracy. For HUBERT (Transformer), we observe substantial gain across all 5 target corpora after transfer. However, for BERT we have a drop for QNLI, QQP, and SST.
These observations confirm our hypothesis that recasting the BERT encodings as TPRs leads to better generalization across down-stream NLP tasks.
Almost all tasks benefit from transferring roles except for QNLI. This may be due to the structure of this dataset, as it is a modified version of a question-answering dataset (Rajpurkar et al., 2016) and has been re-designed to be an NLI task. Transferring the filler embeddings helps with only QNLI and RTE. Transferring BERT parameters in conjunction with fillers or roles surprisingly boosts accuracy for QNLI and SST, where we had negative gains for the BERT model, suggesting that TPR decomposition can also improve BERT’s parameter transfer.
QQP as source: The patterns here are quite different as the source task is now a paraphrase task (instead of inference) and TPR needs to encode a new structure. Again transferring roles gives positive results except for RTE. Filler vectors learned from QQP are more transferable compared to MNLI and gives a boost to all tasks except for SNLI. Surprisingly, transferring BERT parameters is hurting the results now even when TPR is present. However, for cases in which we also transferred BERT parameters (not shown), the Gains were still higher than for BERT, confirming the results obtained when MNLI was the source task.6
6Baseline results slightly differ from Table 2 due to using a different scaling value for this each source task.
We also verified that our TPR layer is not hurting the performance by comparing the test set results for HUBERT (Transformer) and BERT. The results are obtained by submitting models to the GLUE evaluation server. The results are presented in Table 6.

heading: 4.4 MODEL DIAGNOSIS

text:
We also evaluated HUBERT (Transformer) on a probing dataset outside of GLUE called HANS (McCoy et al., 2019) Results are presented in Table 3. HANS is a diagnosis dataset that probes various syntactic heuristics which many of the state-of-the-art models turn out to exploit, and thus they perform poorly on cases that don’t follow those heuristics. There are three heuristics measured in HANS which are as follows: Lexical overlap where a premise entails any hypothesis built from a subset of words in the premise, Subsequence where a premise entails any contiguous subsequences of it, and Constituent where a premise entails all complete subtrees in its parse tree. Our results indicate that TPR models are less prone to adopt these heuristics, resulting in versatile models with better domain adaptation. Following McCoy et al. (2019), we combined the predictions of neutral and contradictory into a non-entailment class, since HANS uses two classes instead of three. Note that no subset of the HANS data is used for training.7
We observed that our HUBERT (Transformer) model trained on MNLI did not diminish BERT’s near-perfect performance on correctly-entailed cases (which follow the heuristics). In fact, it increased the accuracy of Lexical and Subsequence heuristics. On the problematic Non-Entailment cases, however, BERT outperforms HUBERT (Transformer). Since HUBERT has more parameters than BERT it can better fit the training data. Thus, we suspect that HUBERT attends more to the heuristics that MNLI has in its design, and gets a lower score on sentences that don’t follow those heuristics. But to examine the knowledge-transfer power of TPR, we additionally fine-tuned each model on SNLI and tested again on HANS. (For HUBERT (Transformer), we only transfer roles and fillers). On Non-entailment cases, for the HUBERT model, the Lexical accuracy improved drastically: by 61.62% (6,162 examples). Performance on cases violating the Subsequence heuristic improved by 1.44% (144 examples) and performance on those violating the Constituent heuristic improved by 5.4% (540 examples). These improvements on Non-entailment case came at the cost of small drops in Entailment accuracy. This pattern of transfer is in stark contrast with the BERT results. Although the results on Entailment cases are improved, the accuracies for Subsequence and Constituent Non-Entailment cases drop significantly, showing that BERT is failing to integrate new knowledge gained from SNLI with previously learned information from MNLI. This shows that here, HUBERT (Transformer) can leverage information from a new source of data efficiently. The huge improvement on the Lexical Non-entailment cases speak to the power of TPRs to generate role-specific word embeddings: the Lexical heuristic amounts essentially to performing inference
7We observed high variance in the results on HANS for both BERT and HUBERT. For instance, two models that achieve similar scores on the MNLI dev set can have quite different accuracies on HANS. To account for this, we ran our experiments with at least 3 different seeds and reported the best scores for each model.
on a bag-of-words representation, where mere lexical overlap between a premise and a hypothesis yields a prediction of entailment.

heading: 5 CONCLUSION

text: In this work we showed that BERT cannot effectively transfer its knowledge across NLP tasks, even if the two tasks are fairly closely related. To resolve this problem, we proposed HUBERT: this adds a decomposition layer on top of BERT which disentangles symbols from their roles in BERT’s representations. The HUBERT architecture exploits Tensor-Product Representations, in which each word’s representation is constructed by binding together two separated properties, the word’s (semantic) content and its structural (grammatical) role. In extensive empirical studies, HUBERT showed consistent improvement in knowledge-transfer across various linguistic tasks. HUBERT+ outperformed BERT+ on the challenging HANS diagnosis dataset, which attests to the power of its learned, disentangled structure. The results from this work, along with recent observations reported in Kovaleva et al. (2019); McCoy et al. (2019); Clark et al. (2019); Michel et al. (2019), call for better model designs enabling synergy between linguistic knowledge obtained from different language tasks.

heading: ACKNOWLEDGMENTS

text: We would like to thank R. Thomas McCoy from Johns Hopkins University and Alessandro Sordoni from Microsoft Research for sharing and discussing their recent results on HANS, and Xiaodong Liu from Microsoft Research for thoughtful discussions.

heading: A APPENDIX

text:
A.1 DATASET DETAILS
In this section, we briefly describe the datasets we use to train and evaluate our model. GLUE is a collection of 9 different NLP tasks that currently serve as a good benchmark for different proposed language models. The tasks can be broadly categorized into single sentence tasks (e.g. CoLA and SST) and paired sentence tasks (e.g. MNLI and QQP). In the former setting, the model makes a binary decision on whether a single input satisfies a certain property or not. For CoLA, the property is grammatical acceptability; for SST, the property is positive sentiment.
The 7 other tasks in GLUE are paired sentence tasks in which the model strives to find a relationship (binary or ternary) between two sentences. QNLI, WNLI, and RTE are inference tasks, in which given a premise and a hypothesis, the model predicts whether the hypothesis is congruent with the premise (i.e. entailment) or not (i.e. conflict). Although QNLI and WNLI are not originally designed as inference tasks, they have been re-designed to have a similar configuration as other
NLI tasks. This way, a single classifier can be used to judge whether the right answer is in the hypothesis (e.g. for QNLI) or whether a pronoun is replaced with the correct antecedent (e.g. for WNLI). MNLI is an additional NLI task in which three classes are being used instead of two to represent the relation between two sentences. The third class shows neutrality when the model is not confident that the relation is either entailment or contradiction. The last three tasks measure sentence similarity. In MRPC the model decides if two sentences are paraphrases of each other. In QQP, given two questions, the model decides whether they are equivalent and are asking for the same information. All the tasks discussed so far fall under the classification category, where the model produces a probabilistic distribution over the possible class outcomes and the highest value is selected. STS-B, however, is a regression task where the model produces a real number between 1 and 5, indicating the two sentences’ semantic similarity. Since our model is designed only for classification tasks, we skip this dataset.
We observed a lot of variance in the accuracy (±5%) for models trained on WNLI, MRPC, and CoLA. As mentioned in the GLUE webpage8, there are some issues with the dataset, which makes many SOTA models perform worse than majority-voting. We found that MRPC results are highly dependent on the initial random seed and order of sentences in the shuffled training data which is mainly caused by the small number of training samples (Table 4). CoLA is the only task in GLUE which examines grammatical correctness rather than sentiment, and thus it makes it harder to benefit from the knowledge learned from other tasks. The train and test set are also constructed in an adversarial way which makes it very challenging. For example, the sentence “Bill pushed Harry off the sofa for hours.” is labeled as incorrect in the train split but a very similar sentence “Bill pushed Harry off the sofa.” is labeled as correct in the test split. Hence, we only conduct our experiments on the remaining 5 datasets from GLUE.
We also take advantage of an additional NLI dataset called SNLI. It is distributed in the same format as MNLI and recommended by Wang et al. (2018) to be used in conjunction with MNLI during training. However, in our experiments, we treat this dataset as a separate corpus and report our results on it individually.
To further test the capabilities of our model, we evaluate our model on a probing dataset (McCoy et al., 2019). It introduces three different syntactic heuristics and claims that most of SOTA neural NLI models exploit these statistical clues to form their judgments on each example. It shows through extensive experiments that these models obtain very low accuracies for sentences cleverly crafted to defeat the models which exploit these heuristics. Lexical overlap, Subsequence, and Constituent are the three categories examined, each containing 10 sub-categories.
A.2 TEST RESULTS
Table 5 shows the transfer learning results when the source corpus is QQP. Table 6 shows the test results for BERT and HUBERT (Transformer). The top 5 rows are for MNLI as source corpus and bottom 5 rows are for QQP as source corpus.
8https://gluebenchmark.com/faq
A.3 IMPLEMENTATION DETAILS
Our implementations are in PyTorch and based on the HuggingFace9 repository which is a library of state-of-the-art NLP models, and BERT’s original codebase10. In all of our experiments, we used bert-base-uncased model which has 12 Transformer Encoder layers with 12 attention heads each and the hidden layer dimension of 768. BERT’s word-piece tokenizer was used to preprocess the sentences. We used Adamax (Kingma & Ba, 2014) as our optimizer with a learning rate of 5× 10−5 and used a linear warm-up schedule for 0.1 proportion of training. In all our experiments we used the same value for dimension and number of roles and symbols (dS : 32, dR: 32, nS : 50, nR: 35). These parameters were chosen from the best performing BERT models over MNLI. We used the gradient accumulation method to speed up training (in which we accumulate the gradients for two consecutive batches and then update the parameters in one step). Our models were trained with a batch size of 256 distributed over 4 V100 GPUs. Each model was trained for 10 epochs, both on the source task and the target task (for transfer learning experiments).
We performed hyper-parameter tuning for both BERT and HUBERT models on the MNLI dev set. As for the dimension of roles and symbols we did grid search over these values: [10, 30, 60] We fixed the number of roles to 35 and searched among these values for number of fillers: [50, 100, 150]. We additionally performed some light tuning on learning rate, temperature value, and scaling value. To control for the randomness in our results, we ran our experiments by fine-tuning BERT with 3 different seeds and choosing the best results among them. However, for HUBERT we used the same seed for all 7 experiments and only changed the initial weights of layers in the model. We observed small variance in baseline and fine-tuned accuracy across different runs for BERT model. Specifically when MNLI is the source corpus, the standard deviation for QNLI, QPP, RTE, SNI, and SST as target corpora was 0.2%, 0.4%, 0.5%, 0.4%, 0.4% respectively.
A.4 INTERPRETATION OF LEARNED ROLES
To gain a better understanding of what R (the global role matrix) is learning we analyze the attention scores (aR) over this matrix generated by the model for each token. The final role vector (r(t)) for each token is being calculated by performing a matrix product between the attention vector and Role matrix as discussed in Section 3:
r(t) = Ra (t) R (7)
aR is a vector of nR dimension indicating the importance of each Role in constructing the final role vector r(t). First, for each sentence in the dataset we collect the Part Of Speech (POS) tags for each token (or sub-token) coming out of BERT model. We obtain the POS tags from the last sub-tokens of each word. This requires us to take extra care when processing the data and keep a dictionary that maps the index of each token in the original sentence to the indices of all its sub-tokens. Then, we gather the roles each sub-token is being attracted to. To account for the distributed nature of the attention scores, instead of choosing the role with highest value in the attention distribution, we select the top K values, concatenate them, and treat this new tuple as a single role. We then calculate the number of roles assigned to each POS tag. Figure 4 shows this distribution where each color shows a specific role tuple.
The number of all possible POS tags in MNLI is 36 which nicely aligns with the 35 number of roles we chose in our experiments. See this link11 for a description of each tag. To make the visualizations simpler we merged tags indicating similar grammar roles into one coming up with a total of 21 tags (e.g. combining NN, NNS, NNP, and NNPS into one category: NN).
We can observe some interesting patterns by looking at the frequency of the roles assigned to each tag. For example, we observe that the “yellow” role is attracted to ‘.’ tag which represents punctuation marks such as a full stop, question mark, exclamation mark, etc. On the other hand, the “blue” role is attracted almost exclusively to NN. Additionally the “green” role is mainly present in NN, VB, and JJ which are linguistically the three most important POS categories. However, the “purple”
9https://github.com/huggingface/pytorch-pretrained-BERT 10https://github.com/google-research/bert 11Penn Treebank POS
role is more or less used by all different POS categories suggesting low correlation between that role and POS tags.


summaries: The submission proposes a method for hierarchical RL in multiagent settings. In particular it proposes to explicitly decouple training of a high-level and low-level controller with grounded the controller interface as goals in the environment to reach for the low-level controller. The model is trained via PPO with GAE and evaluated on a small set of multi agent locomotion tasks.

summaries: This paper proposes a multi-agent hierarchical reinforcement learning algorithm so that multiple humanoid robots can navigate in multi-agent settings (e.g. avoid collisions, collaboration, chase and escape) in a physically simulated environment. The key difference of this paper with the prior work on MARL is that it used an accurate physics simulation of humanoid robots. This is the main reason of using the hierarchical RL.

abstractText: Multi-agent reinforcement learning is a particularly challenging problem. Current methods have made progress on cooperative and competitive environments with particle-based agents using communication and centralized training. Little progress has been made on solutions that could operate in the real world with interaction, dynamics, and humanoid navigation strategies. In this work, we make a significant step in multi-agent models on simulated humanoid navigation by combining Multi-Agent Reinforcement Learning with Hierarchical Reinforcement Learning. Specifically, we develop a partial parameter sharing approach wherein the lower level of the hierarchy is shared enabling learning using decentralized methods. This drastically reduces the overall parameter space in the multi-agent problem and introduces structure in the optimization problem. We build on top of prior foundational work in learning goal-conditioned policies to learn low-level physical controllers for balance and walking. These lower-level controllers are task-agnostic and can be shared by higher-level policies. Overtop of these goal conditioned policies, we can train decentralized heterogeneous policies for multi-agent goal-directed collision avoidance. Surprisingly, our results show that with this combination of methods, RL techniques can be used for finding strong policies. A video of our results on a multi-agent pursuit environment can be seen here1.

authors:
Andreas Argyriou
Theodoros Evgeniou
Massimiliano Pontil

title: Multi-task feature learning

authors:
Trapit Bansal
Jakub Pachocki
Szymon Sidor
Ilya Sutskever
Igor Mordatch

title: Emergent complexity via multi-agent competition

venue: arXiv preprint arXiv:1710.03748,

authors:
Hugo Bruggeman
Wendy Zosh
William H Warren

title: Optic flow drives human visuo-locomotor adaptation

venue: Current biology,

authors:
Lucian Bu
Robert Babu
Bart De Schutter

title: A comprehensive survey of multiagent reinforcement learning

venue: IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews),

title: Multitask learning

venue: Machine learning,

authors:
Caroline Claus
Craig Boutilier

title: The dynamics of reinforcement learning in cooperative multiagent systems

authors:
Peter Dayan
Geoffrey E Hinton

title: Feudal Reinforcement Learning, pp. 271278

venue: MorganKaufmann,

title: Hierarchical reinforcement learning with the maxq value function decomposition

venue: URL http://arxiv.org/abs/cs/9905014. arXiv: cs/9905014

authors:
Tom Erez
Yuval Tassa
Emanuel Todorov

title: Simulation tools for model-based robotics: Comparison of bullet, havok, mujoco, ode and physx

venue: In 2015 IEEE international conference on robotics and automation (ICRA),

authors:
Benjamin Eysenbach
Abhishek Gupta
Julian Ibarz
Sergey Levine

title: Diversity is all you need: Learning skills without a reward function

venue: In International Conference on Learning Representations,

authors:
Petros Faloutsos
Michiel Van de Panne
Demetri Terzopoulos

title: Composable controllers for physics-based character animation

venue: In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques,

authors:
Jakob Foerster
Ioannis Alexandros Assael
Nando de Freitas
Shimon Whiteson

title: Learning to communicate with deep multi-agent reinforcement learning

venue: In Advances in Neural Information Processing Systems,

authors:
Tao Geng
Bernd Porr
Florentin Wörgötter

title: A reflexive neural network for dynamic biped walking control

venue: Neural Computation,

authors:
Abhishek Gupta
Russell Mendonca
YuXuan Liu
Pieter Abbeel
Sergey Levine

title: Metareinforcement learning of structured exploration strategies

venue: Advances in Neural Information Processing Systems

authors:
Jayesh K. Gupta
Maxim Egorov
Mykel Kochenderfer

title: Cooperative multi-agent control using deep reinforcement learning

venue: Autonomous Agents and Multiagent Systems,

authors:
Dongge Han
Wendelin Boehmer
Michael Wooldridge
Alex Rogers

title: Multi-agent hierarchical reinforcement learning with dynamic termination

venue: In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2006–2008. International Foundation for Autonomous Agents and Multiagent Systems,

authors:
Nicolas Heess
Greg Wayne
Yuval Tassa
Timothy Lillicrap
Martin Riedmiller
David Silver

title: Learning and transfer of modulated locomotor controllers

venue: arXiv preprint arXiv:1610.05182,

authors:
Sander EM Jansen
Alexander Toet
Peter J Werkhoven

title: Human locomotion through a multiple obstacle environment: strategy changes as a result of visual field limitation

venue: Experimental brain research,

title: Learning to achieve goals

venue: In International Joint Conference on Artificial Intelligence (IJCAI), volume

authors:
Mubbasir Kapadia
Shawn Singh
William Hewlett
Petros Faloutsos

title: Egocentric affordance fields in pedestrian steering

venue: In Proceedings of the 2009 symposium on Interactive 3D graphics and games,

authors:
Mubbasir Kapadia
Matt Wang
Shawn Singh
Glenn Reinman
Petros Faloutsos

title: Scenario space: characterizing coverage, quality, and failure of steering algorithms

venue: In Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation,

authors:
Ioannis Karamouzas
Brian Skinner
Stephen J Guy

title: Universal power law governing pedestrian interactions

venue: Physical review letters,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Andrew Kun
W. Thomas Miller III

title: Adaptive dynamic balance of a biped robot using neural networks

venue: In Proceedings of the IEEE International Conference on Robotics and Automation, volume pages,

authors:
Andrew Levy
George Konidaris
Robert Platt
Kate Saenko

title: Learning multi-level hierarchies with hindsight

venue: [cs], Dec 2017a. URL http://arxiv.org/abs/1712.00948

venue: arXiv preprint arXiv:1712.00948, Dec 2017b. URL http://arxiv.org/abs/1712

title: Markov games as a framework for multi-agent reinforcement learning

venue: In Machine learning proceedings

authors:
Pinxin Long
Wenxi Liu
Jia Pan

title: Deep-learned collision avoidance policy for distributed multiagent navigation

venue: IEEE Robotics and Automation Letters,

authors:
Ryan Lowe
YI WU
Aviv Tamar
Jean Harb
OpenAI Pieter Abbeel
Igor Mordatch

title: Multiagent actor-critic for mixed cooperative-competitive environments

authors:
Yongxi Lu
Abhishek Kumar
Shuangfei Zhai
Yu Cheng
Tara Javidi
Rogerio Feris

title: Fullyadaptive feature sharing in multi-task networks with applications in person attribute classification

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Josh Merel
Arun Ahuja
Vu Pham
Saran Tunyasuvunakool
Siqi Liu
Dhruva Tirumala
Nicolas Heess
Greg Wayne

title: Hierarchical visuomotor control of humanoids

title: Real-time neural network control of a biped walking robot

venue: Control Systems, IEEE,

authors:
Volodymyr Mnih
Koray Kavukcuoglu
David Silver
Andrei A Rusu
Joel Veness
Marc G Bellemare
Alex Graves
Martin Riedmiller
Andreas K Fidjeland
Georg Ostrovski

title: Human-level control through deep reinforcement learning

authors:
Ofir Nachum
Shixiang Gu
Honglak Lee
Sergey Levine

title: Data-efficient hierarchical reinforcement learning. arXiv:1805.08296 [cs, stat], May 2018

venue: URL http://arxiv.org/abs/1805.08296

authors:
Ashvin Nair
Vitchyr Pong
Murtaza Dalal
Shikhar Bahl
Steven Lin
Sergey Levine

title: Visual reinforcement learning with imagined goals. arXiv:1807.04742 [cs, stat], Jul 2018

venue: URL http: //arxiv.org/abs/1807.04742

authors:
Liviu Panait
Sean Luke

title: Cooperative multi-agent learning: The state of the art

venue: Autonomous agents and multi-agent systems,

authors:
Xue Bin Peng
Glen Berseth
KangKang Yin
Michiel Van De Panne

title: Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning

venue: ACM Transactions on Graphics (TOG),

authors:
J. Schulman
F. Wolski
P. Dhariwal
A. Radford
O. Klimov

title: Proximal Policy Optimization Algorithms

venue: ArXiv e-prints,

authors:
John Schulman
Philipp Moritz
Sergey Levine
Michael Jordan
Pieter Abbeel

title: Highdimensional continuous control using generalized advantage estimation

venue: In International Conference on Learning Representations (ICLR

authors:
David Silver
Guy Lever
Nicolas Heess
Thomas Degris
Daan Wierstra
Martin Riedmiller

title: Deterministic policy gradient algorithms

venue: In Proc. ICML,

authors:
David Silver
Julian Schrittwieser
Karen Simonyan
Ioannis Antonoglou
Aja Huang
Arthur Guez
Thomas Hubert
Lucas Baker
Matthew Lai
Adrian Bolton

title: Mastering the game of go without human knowledge

authors:
Joseph Suarez
Yilun Du
Phillip Isola
Igor Mordatch

title: Neural MMO: A massively multiagent game environment for training and evaluating intelligent agents

authors:
Richard S. Sutton
Doina Precup
Satinder Singh

title: Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning

title: Multi-agent reinforcement learning: Independent vs. cooperative agents

venue: In Proceedings of the tenth international conference on machine learning,

authors:
Hongyao Tang
Jianye Hao
Tangjie Lv
Yingfeng Chen
Zongzhang Zhang
Hangtian Jia
Chunxu Ren
Yan Zheng
Changjie Fan
Li Wang

title: Hierarchical deep multiagent reinforcement learning

venue: arXiv preprint arXiv:1809.09332,

authors:
Y. Tassa
Y. Doron
A. Muldal
T. Erez
Y. Li
D. de Las Casas
D. Budden
A. Abdolmaleki
J. Merel
A. Lefrancq
T. Lillicrap
M. Riedmiller

title: DeepMind Control Suite

authors:
Yuandong Tian
Qucheng Gong
Wenling Shang
Yuxin Wu
C. Lawrence Zitnick

title: Elf: An extensive, lightweight and flexible research platform for real-time strategy games

authors:
Kathleen A Turano
Dylan Yu
Lei Hao
John C Hicks

title: Optic-flow and egocentric-direction strategies in walking: Central vs peripheral visual field

venue: Vision research,

authors:
Alexander Sasha Vezhnevets
Simon Osindero
Tom Schaul
Nicolas Heess
Max Jaderberg
David Silver
Koray Kavukcuoglu

title: Feudal networks for hierarchical reinforcement learning. arXiv:1703.01161 [cs], Mar 2017

venue: URL http://arxiv.org/abs/1703.01161

authors:
William H Warren Jr.
Bruce A Kay
Wendy D Zosh
Andrew P Duchon
Stephanie Sahuc

title: Optic flow is used to control human walking

venue: Nature neuroscience,

authors:
KangKang Yin
Kevin Loken
Michiel van de Panne

title: Simbicon: Simple biped locomotion control

venue: ACM Transactions on Graphics,

authors:
Petr Zaytsev
S Javad Hasaneini
Andy Ruina

title: Two steps is enough: no need to plan far ahead for walking balance

venue: In 2015 IEEE International Conference on Robotics and Automation (ICRA),

authors:
Amy Zhang
Nicolas Ballas
Joelle Pineau

title: A dissection of overfitting and generalization in continuous reinforcement learning

venue: arXiv preprint arXiv:1806.07937,

authors:
Lianmin Zheng
Jiacheng Yang
Han Cai
Ming Zhou
Weinan Zhang
Jun Wang
Yong Yu

title: Magent: A many-agent reinforcement learning platform for artificial collective intelligence

venue: In Thirty-Second AAAI Conference on Artificial Intelligence,

heading: 1 INTRODUCTION

text:
Deep Reinforcement Learning (DRL) has been successful in solving complex planning tasks. Given a sizeable computational budget DRL has displayed superhuman performance on many games (Mnih et al., 2015; Silver et al., 2017). However, less progress has been made on the Multi-Agent Reinforcement Learning (MARL) problem space, possibly due to the non-stationary optimization of multiple changing policies, which is not easily overcome by collecting more data (OpenAI, 2018). If the goal is to create agents that can operate in a dynamic multi-agent world, more stable methods with novel forms of communication are needed. The trend to make progress on MARL has been to simplify the optimization problem. For example, converting the multi-agent problem into a single agent centralized model results in large gains in performance but can increase the number of network parameters significantly and imposes a constraint on the generalization to the number of agents (Lowe et al., 2017). By using recurrent policies, significant computational power, and constraints on the amount the policy is allowed to change between updates, it is possible to beat the best humans at the multi-agent game of Dota (OpenAI, 2018). While these methods have shown promise, they require significant amounts of compute and have not yet displayed success in complex and dynamic multi-agent environments. While recent work has been successful in producing competitive behaviour using asymmetric self-play, this work is limited to two agents (Bansal et al., 2017).
In this work, we propose the integration of MARL with Hierarchical Reinforcement Learning (HRL) to produce heterogeneous humanoid agents that can both navigate and interact in dynamic simulation. Specifically, we propose a method to reduce the complexity in the MARL policy optimization problem using a type of parameter sharing. While previous methods have focused on re-framing the problem as a type of single-agent RL problem, we show that this does not scale. Our method
1https://sites.google.com/view/mahrl
preserves the essential features of heterogeneous agent behaviour without adding more network parameters (Lowe et al., 2017). This use of HRL with a goal conditioned lower layer (Kaelbling, 1993), has significant advantages over current methods. We use shared parameter methods for task-agnostic portions of the policy similar to multi-task learning (Caruana, 1997; Argyriou et al., 2007; Lu et al., 2017) where each high-level agent is a different task. Given the shared sub-policy, the optimization is simplified and allows learning complex multi-agent policies with significantly less data. This method represents, to the best of our knowledge, the first method for heterogeneous multi-agent physical character control for locomotion, navigation, and behaviour.

heading: 2 RELATED WORK

text: Simulated robot and physical character control is a rich area of research with many solutions and approaches. Neural models focused on training neural networks by receiving joint or body sensor feedback as input and producing appropriate joint angles as output (Geng et al., 2006; Kun & Miller III, 1996; Miller III, 1994). A biped character’s movement controller set can also be manually composed using simple control strategies and feedback learning (Yin et al., 2007; Faloutsos et al., 2001). HRL has been proposed as a solution to handling many of the issues with current RL techniques that have trouble with long horizons and weak signal. Many frameworks have been proposed for HRL, but none seem to be the obvious choice for any particular problem (Sutton et al., 1999; Dayan & Hinton, 1993; Dietterich, 1999). One difficulty in HRL design is finding a reasonable communication representation to condition the lower level. Some methods pretrain the lower level on a random distribution (Heess et al., 2016; Peng et al., 2017; Merel et al., 2018) and others learn a more constructive latent encoding (Nair et al., 2018; Eysenbach et al., 2019; Gupta et al., 2018). There is also the present challenge of learning multiple levels of the hierarchy concurrently (Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2017a). Here a goal-based approach that uses a footstep space representation that is pretrained to learn a task-agnostic lower level of control. This provides the high-level policies with a single shared model that is conditioned to produce diverse behaviour based on a latent goal input.

heading: 2.1 MULTI-AGENT DEEP REINFORCEMENT LEARNING

text:
There are many types of multi-agent learning problems, including cooperative-competitive and withwithout communication (Bu et al., 2008; Tan, 1993; Panait & Luke, 2005). Recent work converts the MARL problem to a single agent setting by using a single Q-function across all agents (Lowe et al., 2017). Additional work focuses on the problem of learning communication methods between agents (Foerster et al., 2016). While progress is being made, MARL is notoriously tricky due to the non-stationary optimization issue, even in the cooperative case (Claus & Boutilier, 1998). In this work, we apply a partial parameter sharing method assuming all agents share task-agnostic locomotion and optimize similar goals (Gupta et al., 2017).
There exists few environments specifically created for MARL evaluation (Zheng et al., 2018; Suarez et al., 2019). The focus of these environments is often a type of strategy learning and cooperation (Tian et al., 2017; Vinyals et al., 2017). MARL is a growing area of research and, as such, will need increasingly complex environments to evaluate algorithms. In this work, we are interested in the overlapping problems of control and perception. To this end, we have created the first simulation of its type that affords multiple physics-based control tasks with variable numbers of agents. Recent work has begun to combine MARL and HRL but is limited to simple environments, uses additional methods to stabilize the optimization, and includes communication (Tang et al., 2018; Han et al., 2019). This work focuses on a decentralized approach for interacting models that generalize across scenarios, agent numbers, and tasks.
Our work represents a new paradigm in learning within the multi-agent navigation domain that goes beyond simple navigation control strategies. We show compelling AI that learns navigation and gameplay strategies with fully articulated physical characters. This is achieved through a novel learning strategy that produces high-value policies for a complicated control problem.

heading: 3 POLICY REPRESENTATION AND LEARNING

text:
In this section, we outline key details of the general Reinforcement Learning (RL) framework. RL is formulated on the Markov Dynamic Process (MDP) framework: at every time step t, the world (including the agent) exists in a state st ∈ S, wherein the agent is able to perform actions at ∈ A, sampled from a policy π(st, at) which results in a new state st+1 ∈ S according to the transition probability function T (st+1|at, st). Performing action at in state st produces a reward rt from the environment; the expected future discounted reward from executing a policy π is:
J(π) = Er0,...,rT [ T∑ t=0 γtrt ] (1)
where T is the maximum time horizon, and γ is the discount factor, indicating the planning horizon length. The agent’s goal is to optimize its policy, π(·|θπ), by maximizing J(π). The policies in the work are trained using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017). The value function is trained using TD(λ).

heading: 3.1 HIERARCHICAL REINFORCEMENT LEARNING

text: In HRL the original MDP is separated into different MDPs that should be easier to solve separately. In practice, this is accomplished by learning two different policies in two different layers. The lower level policy is trained first and is often conditioned on a latent variable or goal g. The lower level policy πlo(a|s, g) is constructed in a way to give it temporally correlated behaviour depending on the g. After the lower level policy is trained, it is used to help solve the original MDP using a separate policy πhi(g|s). This policy learns to provide goals to the lower policy to maximize rewards.

heading: 4 MULTI-AGENT REINFORCEMENT LEARNING

text:
The extension to the MDP framework for MARL is a partially observable Markov game (Littman, 1994). A Markov game has a collection ofN agents, each with its own set of actionsA0, . . . , AN and partial observationsO0, . . . , ON of the full state space S. Each agent i has its own policy π(a|oi, θi) that models the probability of selecting an action. The environment transition function is a function of the full state and each agent’s actions T (S′|S,A0, . . . , AN ). Each agent i receives a reward ri for taking a particular action ai given a partial observation oi and its objective is to maximize this reward over time ∑T t=0 γ
trti , where γ is the discount factor and T is the time horizon. The policy gradient can be computed for each agent as
∇θiJ(π(·|θi)) = ∫ Oi dθi(oi) ∫ Ai ∇θi log(π(ai|oi, θi))Aπ(oi, ai) dai doi (2)
where dθ = ∫ S ∑T t=0 γ
tp0(o0)(o0 → o | t, π0) do0 is the discounted state distribution, p0(o) represents the initial state distribution, and p0(o0)(o0 → o | t, π0) models the likelihood of reaching state s by starting at state o0 and following the policy π(a, o|θπ) for T steps (Silver et al., 2014). Here Aπ(o, a) represents the advantage function estimator GAE(λ) (Schulman et al., 2016).
The challenge in MARL is that each agent learns separately, and this often results in a non-stationary learning problem. Each agent is learning how to estimate the dynamics and advantage of an environment that is affected by other actively learning agents. Data collected and used for learning becomes inaccurate after any learning update; thus, the problem is non-stationary. MADDPG reformulates the above problem into a fully observable problem with a centralized Q-function using the equation:
∇θiJ(θ) = Eoi∼pπ(·),ai∼π(·|oi)[∇θi log π(ai|oi, θi)Q π i (o, a)] (3)
Here Qπi (o, a) is a centralized action-valued function that gives the Q value for agent i. Where o = {o0t , . . . , oNt } is the observations of all agents and a = {a0t , . . . , aNt } all agent actions at time t and θ = {θ0, . . . , θN} is the collection of policy parameters for each agent. This framework will allow the method to train the policies together, but it treats the agents as disjoint distributions to optimize. Instead, we propose a decentralized method that does not need to learn a complex Qfunction and reduces the optimization challenges by using HRL.

heading: 4.1 TASK-AGNOSTIC LOCOMOTION CONTROLLER (LC)

text:
The LC, the lower-level policy in our design, is designed to learn a robust and diverse policy π(at|ot, gt, θlot ) based on a latent goal gt varaible. The footstep goals gL = {p̂0, p̂1, θ̂root} in Figure 1, consist of the agent root relative distances of the next two footsteps on the ground plane and the desired facing direction at the first step’s end. This goal description is motivated by work that shows people may plan steering decisions two foot placements ahead (Zaytsev et al., 2015).
The LC learns to place its feet as accurately as it can to match these step goals using the reward rLg = exp(−0.2||sgchar − gL||2). The better the LC learns this behaviour, the more responsive the controller will be to provided goals.

heading: 4.2 MULTI-AGENT HIERARCHICAL REINFORCEMENT LEARNING

text:
We construct a multi-agent learning structure that takes advantage of hierarchical design or MultiAgent Hierarchical Reinforcement Learning (MAHRL). Each agent has its own higher level policy (Multi-Agent Navigation Controller (NC)) π(g|o, θhii ) and a shared task agnositic lower level policy (LC) π(a|o, g, θlo). This method allows us to introduce more structure into the difficlut multi-agent optimization problem. This change alters the underlying MDP, such that the policy is queried for a new action every k timesteps. This also changes the MDP method by reducing the dimensoinality of the action space to specifiying goals g while using the low-level policy to produce more temporally consistent behaviour in the original action space. The low-level policy is used in a deterministic manner to futher reduce variance introduced into the problem. We will use the notation π(ai|oi, θhii , θlo) to denote the policy induced by the pair of policies. While current research shows that it is challenging but possible to train a two-level hierarchy concurrently (Nachum et al., 2018; Levy et al., 2017b), we instead pretrain the lower level policy and leave it fixed and shared across all agents. This reduces the MARL problem from learning the details of locomotion via joint torques for each agent to learning goal-based footstep plans for each agent.
The use of HRL is key to the method. When the challenge in MARL is dealing with what can be large changes in the distribution of states visited by the agent, the use of a temporally correlated structure given by the goal-conditioned LC significantly reduces the non-stationarity. Not only is each agent sharing its network parameters with each other agent, but this portion has also been carefully constructed to provide structured exploration for the task, thus greatly reducing the number of network parameters that need to be learned. This is in contrast to centralized methods that take a step away from the goal of solving the heterogeneous problem in a scalable way. We extend the analysis of this work by combining the PPO algorithm in a MARL partial parameter sharing setting. Details of the algorithm can be found in the supplementary material. The use of the LC controls the way dθi(oi) can change for each agent making it easier for each agent to estimate other agents potential changes in behaviour because the LC is already trained to produce a specific type of useful behaviour that is a subset of the full space. We also create a version of MADDPG that uses the HRL structure and show that it does not perform as well.

heading: 5 LEARNING HIERARCHICAL MULTI-AGENT NAVIGATION CONTROL

text: To solve the hierarchical learning problem, we train in a bottom-up fashion sharing the LC policy among heterogenous NC policies. The levels of the hierarchy communicate through shared actions and state in a feedback loop that is meant to reflect human locomotion. The NC’s objective is to provide footstep placement goals aH = gL for the LC as seen in the right hand side of Figure 1. These footstep goals are produced as two-step plans. Each step is parameterized with its root-relative placement, angle on the ground, centre of mass heading vector, and a time signature. The NCs are queried for a new footstep action every 0.5 s. The NCs decides what action to take based on the egocentric velocity field E in front of the agent, its pose schar and the NC goal gH , which together form the NC state C = {E, schar, gH}, seen in the left box of Figure 1.

heading: 5.1 STATE SPACE

text:
In several studies, it has been shown that optic flow, an inherently egocentric phenomenon of mapping velocities to regions of vision, is key to sighted locomotion (Bruggeman et al., 2007; Warren Jr
et al., 2001). Additionally, the visual field directly impacts walking ability and locomotion stability (Jansen et al., 2011). Taken together, vision’s role in locomotion forms an egocentric velocity field where perceived distance and movement play different roles in locomotion control (Turano et al., 2005). This evidence has been used previously to define multi-agent navigation models, both by constructing a discretized egocentric field (Kapadia et al., 2009) and by learning the discretization of an egocentric field (Long et al., 2017).
The NC uses as input an egocentric velocity field relative to the agent’s location and rotation. This egocentric velocity field E is 32× 32 samples over a space of 5x5 m, starting 0.5 m behind the agent and extending 4.5 m in front, shown in the left hand side of Figure 1. The velocity field consists of two image channels in the x and y directions of a square area directly in front of the agent, where each sample is calculated as the velocity relative to the agent (Bruggeman et al., 2007; Warren Jr et al., 2001). The current pose of the agent is included next, followed by the NC goal. The NC goal gH consists of two values, the agent relative direction and the distance to the current spatial goal location.

heading: 5.2 SIMULATION ENVIRONMENT & TRAINING

text:
We construct a collection of physics-based simulation tasks to evaluate the method. At initialization, each agent is randomly rotated, and the initial velocities of the agent’s links are determined by a selected motion capture clip using finite differences and rotated to align with the agent’s reference frame. Goal locations are randomly generated in locations that are at least 1 m away from any obstacle. Each agent is randomly placed in the scene such that it does not intersect with any other agent or obstacle. The number and density of agents in the simulation vary depending on the task. For training, we found that starting with 3 agents in the environment is a good trade-off between computation cost and the generalization ability of the resulting learned policy. Environment specifics are given in Table 1. We consider the simulation and training environment to be, to the best of our knowledge, another novel contribution. While some simulators exist that support physics-based simulation for robots (Brockman et al., 2016; Tassa et al., 2018), few support more than one agent, with at most 2 which interact beyond physical simulation and actually solve their tasks with respect to each other. Other libraries focus on supporting different kinds of MARL configurations for particle-based agents (Lowe et al., 2017). Our proposed approach represents the first physics-based simulation of its kind that supports MARL.
The NC uses convolutional layers followed by dense layers. The particular network used is as follows: 16 convolutional filters of size 6 × 6 and stride 2 × 2, 16 convolutional filters of size 3 × 3 and stride 1× 1, the structure is flattened and the character and goal features schar, gH are concatenated, a dense layer of 256 units and a dense layer of 128 units are used at the end. The network uses Rectified Linear Unit (ReLU) activations throughout except for after the last layer which uses a tanh activation that outputs values between [−1, 1]. All network inputs are standardized with respect
to all states observed so far. The rewards are scaled by a running variance. That is, the variance is computed from a running computation during training that is updated after every data collection step. The batch size used for PPO is 256, with a smaller batch size of 64 for the value function. The policy learning-rate and value function learning-rate are 0.0001 and 0.001, respectively. The value function is updated four times for every policy update. The NC also uses the Adam stochastic gradient optimization method (Kingma & Ba, 2014) to train the Artificial Neural Network (ANN) parameters.

heading: 6 RESULTS

text: In this section, we demonstrate the efficacy of MAHRL. We introduce a new set of challenging multi-agent simulations and show that the previous method can not solve these tasks. However, using MAHRL many of these tasks can be solved with a modest amount of data and computational power. We separate our evaluation into four sections. We examine the performance of MAHRL in terms of training, quantitative metrics, and learned policies. Then we examine the performance in terms of computation cost and generalizability over the number of agents in the environment.

heading: 6.1 LEARNING

text:
We evaluate MAHRL in complex multi-agent environments by first examining the performance of the reward function over training episodes. We show that MAHRL performs much better than the basic PPO algorithm without any hierarchical structure in the 5-agent Mall in Figure 2a. After a lengthy training session, the basic PPO is not able to even produce a standing behaviour. We find that MAHRL performs well and is able to learn navigation behaviour after a short amount of training. We find that the hierarchical design provides a significant improvement in this case. We also compare to MADDPG in Figure 2a,3a and find that, while it does better than PPO as it also uses the HRL structure, it struggles to produce good coordinated behaviour. We believe this is related to the large Q-network that needs to be learned for MADDPG. Qualitatively, throughout training, even with the increased control complexity, our method is able to learn successful navigation strategies shown in Figure 8. Each agent in the scenario quickly develops strong navigation behaviours that become more conservative over time as agents value avoiding collisions. This can result in agents taking longer paths in the environment.
MAHRL, is also applied to a Pursuit environment. In this environment, there is one agent (agent 0) with the same navigation goal as in previous environments. As well, two additional agents (pursuer 0,1) that have the goal of reaching agent 0. This is accomplished by setting a high-level goal gH for the pursuers to the location of agent 0. In Figure 3a, we show a comparative analysis of the learning curves of MAHRL, MADDPG, and PPO. We note that qualitatively, the three agents all begin to increase their average reward via their navigation objective, as learning progresses the pursuing agents outperform agent 0, shown in Figure 3b and as they get better agent 0 has an increasingly difficult time reaching its own navigation targets. An example of this behaviour is shown in Figure 7.
Last MAHRL is applied to a 2-on-2 soccer simulation, Soccer shown in Figure 4b. In this simulation, there are two teams of 2 bipedal agents. The goal of the members of each team is to kick the ball into the other team’s goal area (shown in green). The task is particularly challenging to learn as each agent now has to learn with one cooperative agent and two adversarial agents. We find that MAHRL works well at learning policies for agents to locate and kick the ball into the opposing team’s goal. The learning curves for two different methods that use MAHRL are shown in Figure 4a, on based on PPO and another based on TD3. Our results often produce imbalanced teams where one team becomes much better at the game than the other. This can be seen in the accompanying video on the website.
(a) ego field x (b) ego field y
By extracting the gradients on the input features for the value function in the learned model, we can examine some artifacts of what is learned. Recall that for input, we include in the state of the NC a simple model of an egocentric perceptual field– a rectangular region in front of the agent. We show that our models learns two important aspects of navigation with respect to this field. The magnitude of the velocity field gradients in our learned model reveals that the learning process has developed
an egocentric velocity field attenuated with distance. This can be seen in Figure 5a. Interestingly,
this field is biased toward the rightward direction (down in the figures). This bias supports reciprocal collision avoidance in counter flows.

heading: 6.2 QUANTITATIVE COMPARISON

text:
To evaluate learned navigation strategies quantitatively, we capture the mean number of collision events overall agents for each episode in several instantiations of the Mall environment. We perform this study over the state-of-the-art MADDPG method, MAHRL with and without Heterogeneous agents, and PPO. The results are shown in Figure 6a. For each model, we perform 155 policy rollouts over several random seeds. The basic PPO algorithm does not appear to learn anything. A KruskalWallis rank-sum test and post-hoc Conover’s test with both FDR and Holm corrections for ties show the MAHRL methods significantly outperform others (p ¡ 0.01).
To evaluate the pursuit and evade strategies quantitatively, we capture the mean number of collision events overall agents for each episode in several instantiations of the game. We perform this study over the MADDPG method, MAHRL, and PPO. The results can be seen in Figure 6. For each model, we run 155 simulations over several random seeds. A Kruskal-Wallis rank-sum test and post-hoc Conover’s test with both FDR and Holm corrections for ties show the MAHRL and MADDPG methods significantly outperform PPO (p < 0.01). However, MAHRL is not significantly different from MADDPG (p = 0.45), but the distribution is skewed lower than MADDPG with more zero collision samples.

heading: 6.3 QUALITATIVE RESULTS

text:
To evaluate the NC policies qualitatively, we show that agents learn to navigate arbitrary scenarios while avoiding collisions with both obstacles and other agents. First, we show a rasterized version of an example episode from the Pursuit environment in Figure 7 where agents learn to corner and tackle. Then, we show that agents can successfully and continuously navigate complicated environments of forced interactions, as seen in Figure 9a (Kapadia et al., 2011). Finally, robust clogging and trampling are shown in both low and high-density bottleneck egress scenarios, respectively Figure 9b.
6.4 COMPUTATION AND GENERALIZATION
The deep integration of physical character control and distributed multi-agent navigation comes with a cost directly dependent on the number of active agents. In this section, we show two results in the same experiment, the computational costs of increasing the number of agents and the model generalization to more difficult scenarios. For two scenarios, mall and bottleneck, the number of agents is increased, and we record the average reward and computation time from the simulation. The agent-computation curve in Figure 10(a) indicates a linear trend in computational cost. While at agents counts in the 20s the simulation is not real-time, the most computationally expensive part is not the learning system but the physics simulation. Computationally efficient articulated body simulation is an active area of research (Erez et al., 2015). For accurate and stable simulation, we use a physics time-step of 0.0005 s.
To evaluate the learned policy’s ability to generalize with group size, we vary the number of agents for select tasks after a homogeneous policy has been trained. The longer the policy can maintain a high average reward, the better the generalization. In addition, the average reward for two different types of policy training styles is compared. The first method trains on a single task at a time; the second method uses multi-task learning in hopes that a more generalizable task-independent structure is acquired. The multi-task method, often preferring to optimize easier tasks, does not appear to learn more robust
policies compared to the scenario-space based method (Kapadia et al., 2011). All generalization results can be seen in Figure 10(b). However, generalization remains a known and open issue of DRL methods (Zhang et al., 2018)

heading: 7 CONCLUSION

text:
In this paper, we present a novel method where, for the first time, multi-agent navigation and physical character control are integrated. Multiple heterogeneous interacting agents can experience physical interactions, handle physical perturbations, and produce robust physical phenomena like falling and slipping. To achieve this, we developed an integrated model of MARL and HRL with partial parameter sharing. The evaluation of this approach shows how valuable it is for addressing the non-stationary learning problem of MARL in complex multi-agent scenarios. In particular, the heterogeneous Pursuit and Soccer scenarios learn complex routing and tackling behaviours akin to state-of-the-art competitive self-play approaches.
While our method produces promising results, the work is limited by the fixed LC partial parameter sharing. There is room for research in the area of training the LC and NC concurrently. For the NC, we introduced a set of reward functions to encourage human-like behaviour while navigating with other agents. The literature motivates these rewards, but balancing them is its own challenge. In the future, it may be beneficial to use additional data-driven imitation terms to encourage humanlike paths. Finally, considerable effort was made to create combined locomotion, navigation, and behaviour controller that is robust to the number of agents in the simulation. However, robust generalization remains an open problem.

heading: 8 SUPPLEMENTARY MATERIAL

heading: 8.1 LOCOMOTION CONTROLLER

heading: 8.1.1 REWARD

text:
We found that a combination of imitation and end effector rewards leads to a robust and reactive motion. The goal is to reward joint angle targets which result in mimicking source motions. The
combination of these competing rewards leads to a complex balancing problem of many objectives– trading off imitation behaviour for footstep matching.

heading: 8.1.2 IMITATION

text:
The imitation method is predicated on time indexed 0.5 s long reference motions qi(t) created from motion capture data and augmented with footstep parameterized labels. The difference between the agent and the desired pose to imitate is computed as the weighted difference between the time indexed pose and the agent rLp = exp(−0.5 ∑ i ||sichar−qi(t)||2). The motion database contains 7 s of segmented motion that coarsely covers the space of possible footstep positions and the motions necessary to realize those footsteps. The accuracy and diversity of the Low-Level Controller (LLC)’s motion can be increased or made more robust with additional motion if desired, including stylized, disordered, running, and more.
However, if the agent matches these footstep goals too well, the smoothness of the motion will degrade. A balance is struck between learning robust footstep placement strategies and perfect mimicry–the reference motions and foot placements guide naturalness and should not be over-fitted.

heading: 8.1.3 END EFFECTOR

text:
An additional reward is given to encourage the policy to match the end-effector positions in the imitation motion rLe = exp(−0.15 ∑ e ||sechar − qe(t)||2). Reward is also given for how well the agent
matches the imitated motion’s Centre of Mass (COM) rLcom = exp(−0.2 ∑ ||scomchar − qcom(t)||2)

heading: 8.1.4 TORQUE PENALTY

text: To smooth the actions generated from the control policy, a reward for maximizing the above rewards with minimal torque cost is used rLτ = exp(−0.2 ∑ i τ(s i char)). These torques are normalized by specified torque limits that keep the agent from displaying unrealistic strength.

heading: 8.2 TRAINING & SIMULATION ENVIRONMENT

text:
The LLC state includes the proprioceptive-like joint information. In particular, the components of schar consist of the COM relative distances and velocities of each links as well as the rotation and angular velocities of each link as shown in Figure 1(d). The trained policy actions are in the form of angle-axis targets for per-joint Proportional Derivative (PD) controllers. In this way, the action is used to set the desired position of each joint of the agent. This can be seen in Figure 1(e). The method is also able to learn policies for agents of different types. In Figure 8 we additionally show the method using a full humanoid LLC character.
The agent is simulated with the physics-based rigid body simulation environment Bullet Bullet (2015). The policy is trained in an on-policy fashion. At the beginning of each episode a goal gL is sampled from the simulation and updated every 0.5 s. Episodes end when either a time limit T is reached or the agent falls. Between training rounds 4096 transition tuples are collected by simulating episodes in parallel. After enough tuples have been collected 16 minibatch updates are performed with a batch size of 256. The value function is updated 64 with a minibatch size of 64. For the policy and value function. The policy and value function are both modeled using a ANN with 2 hidden layers of size 512 and 256. The network uses ReLU activations through except for after the last layer where a tanh activation that outputs values between [−1, 1] is used. The Adam stochastic gradient optimization method is used to train the ANN parameters Kingma & Ba (2014). The learning rates for the policy and value function are 0.0001 and 0.001 respectively. Details related to the learning algorithm can be found in the supplementary material.
Many previous methods have created robust controllers via imitation. These methods were not intended to be used in crowds or dynamic environments with potentially random disturbances from other interacting agents in the simulation. Given that we intend to use this agent in a physical multiagent simulation where the agent may bump into other agents or obstacles when trained in a crowded setting, we apply additional methods to simulate pushes that may be encountered. That is, random pushes between 50 − 150 N are applied every 3 − 5 s for a duration of 0.1 − 0.3 s to increase the robustness of the controller. Similarly, the motions in the motion database start from the same facing
direction. To make the agent more robust to distributed crowd scenarios, the agent’s initial facing direction is randomized during training.

heading: 8.2.1 NC REWARD

text:
Navigation combines the desire to move towards goals, while avoiding collisions with other objects, in an energy efficient way. The methods purpose is to implicitly learn low effort local turning behaviour while avoiding collisions without the need to explicitly describe its operation. To elicit this behaviour, a combination of reward signals are used. Primarily, a dense reward is used to encourage the agent to walk in the direction of its current goal.
rHd = exp(−(min(0, (utar ∗ vcom)− vcom))2) (4) where vcom is the agent’s velocity and uTtar is a normalized vector in the direction of the goal. For this work, a desired speed of vcom = 1.0m/s is used.
A directional reward is not enough to encourage the agent to proactively reach its goal. To reinforce the importance of goal reaching behaviour, a large reward rgoal for reaching the goal is added–being within √ 2 m. In this work, the value rgoal = 20 is used and comes from horizon = 1/(1− γ) and the maximum reward the agent can otherwise receive is max(rd) = 1, a reward that will be more important than travelling to the goal should be at least horizon ∗ max(rd). This reward greatly increases the goal reaching behaviour. However, it can have the unintended effect of making the agent seek its goal aggressively, by trampling other agents. To reduce this behaviour, a repulsive cost was added when the agent is within √ 3 m other agents
rHa(a) = ∑
a′∈{A−a}
−(rs + (l − dist(a, a′)) (5)
where A is the set of agents in the simulation, and dist(a, a′) computes the Euclidean distance between the COM of agents a and a′. We empirically found rs = 2.5 and l = 3.0 work well for defining a distance dependent penalty in this case. This is intended to roughly approximate the power law of pedestrian interactions (Karamouzas et al., 2014). A similar repulsive cost is applied between the agent a and obstacles OB in the scene rHb(a) = ∑ ob∈OB −2.5 for each obstacle within 1m of a. The high reward for reaching the goal makes the agent very single-minded. The obstacle penalty is introduced to prevent the agents from using obstacles as affordances to regain balance.
Many RL simulation environments use a flag to indicate the episode or simulation end. This indicates that either the agent has reach its time limit or the agent has entered into an unrecoverable area of the state space, such as a fall. With multiple agents, the likelihood of a fall is high. It is not clear what to do when there are multiple agents being simulated. Terminating early, when one agent has fallen, is sub-optimal for other agents doing well, and waiting for every agent to fall wastes compute resources while most agents are collecting unhelpful data. We chose to reset the simulation when more than half the agents have fallen. However, fallen agents continue to act and need to be heavily penalized, so they receive a fixed reward of −5 which represents the largest penalty in our system. The NC’s final reward function is a combination of the task rewards and behavioural costs:
rH(a) = { −5 if fallen rHd + rHa(a) + rHb(a) otherwise.
(6)

heading: 8.3 LEARNING DETAILS

text:
The modification to this algorithm for the MARL method is to change line 7−8 to execute an action for each agent in the simulation in parallel.
The Inputs are standardized wrt to all states seen so far. The rewards are divided by the variance. The variance is computed from a running computation during training that is updated after every data collection step. The advantage is batch normed.
Hyper Parameter Exploration Parameter exploration is a key process in acheiving the best results for DRL methods. In this work we explored different network architectures, policy variances, annealing the variance, learning rates and activation types. The results with the best performance are reported in the paper.
Algorithm 1 Goal-Based Learning Algorithm 1: Randomly initialize model parameters θπ and θv 2: while not done do 3: for i ∈ {0, . . . N} do 4: τi ← {} 5: for t ∈ {0, . . . , T} do 6: at ← π(·|st, θπ) 7: st+1, rt ← execute at in environment 8: τi,t ←< st, at, rt > 9: st ← st+1 10: end for 11: end for 12: Update value function Vπ(·) parameters θv using {τ0, . . . , τN} 13: Update policy parameters θπ using {τ0, . . . , τN} 14: end while

heading: 8.4 RESULTS DETAILS

heading: 8.4.1 LC HYPERPARAMETER EXPLORATION

text: The LC hyperparameter exploration learning curves in Figure 11 illustrate the process of finding optimal training parameters. It was found that annealing the policy variance over time did not increase the learning efficiency. As well, the best settings for the policy variance and kl threshold are 0.2 and 0.25 respectively.

heading: 8.4.2 MULTI-AGENT NAVIGATION UNIT TASKS

text: In this section, we show several multi-agent navigation tasks and their qualitative performance using our method. These are rendered as birds-eye views of scenarios with each agents trajectory over time rendered. Each agents goal is rendered as a red point within a red circle and the shortest linear path to that from the agent’s current position is rendered as a red line. We show that agent’s learn to navigate towards arbitrary goals repeatedly hitting its mark with each new goal in Figure 12. Predictive reciprocal collision avoidance is important in multi-agent navigation and lends naturalness to qualitative results. We show our method learns high value reipcrocal collision avoidane strategies, and we overlay the sample points for the velocity state space to illustrate how this is acheived in Figure 13. We show that this generalizes to the introduction of obstacles in both Figure 14 & 15.

heading: 8.5 ADDITIONAL DISCUSSION

heading: 8.5.1 LC AGILITY

text:
While often in the crowd simulation literature it is common to have agents that can turn on point or produce holonomic motions, this is unrealistic for articulated agents. Interestingly, the agent learned stopping and in-place turning behaviour which was not contained in the imitation data the LC was trained on, indicating the system can generate behaviours beyond its design. However, the agents in this work do not make unrealistic sharp turns. This is in part related to the NC being able to avoid falls, however there are many other factors: The LC motion capture data, gL selection during LC training and that the LC was not trained with other agents. Progress in any of these areas can improve the responsiveness of the LC and is left for future work.
While the LC is goal driven, this goal is only based on foot placement, a fully interactive agent can have many other types of short term goals, including where to put one’s hands to manipulate items in the environment, like doors.