summaries: This is a well-written paper that discusses how to learn disentangled representations for the learning from demonstrations (LfD) task in robotics. It is shown that using weak-supervision on top of unsupervised learning frameworks (that use the variational autoencoder for instance) can work well in this case. These disentangled factors of variation in the data are shown to correspond well to the 'abstract concepts' of the human demonstrations. This is shown in the example of the PR2 robot dabbing demonstrations, including visual data as well robot trajectories.

summaries: Under the context of learning from demonstrations, the paper studies the problem of leaning interpretable low dimensional representations from high dimensional multimodal inputs using weak supervision. Paper argues that since robots and humans have different levels of abstractions and mechanisms, observation+action spaces between them are greatly misaligned which complicates learning by directly observing humans. However, the underlying concepts essential for tasks lie in a much lower-dimensional manifold. Learning this manifold effectively and in an interpretable way, especially using weak supervision, can significantly change how robots can acquire skills from demonstrations and generalize them to new unseen scenarios. Towards this end, the paper proposes to learn probabilistic generative models capturing high-level notions from demonstrations using variational inference. The strength of the paper is in demonstrating that conditional latent variable models can learn disentangled low dimensional represented using weak supervision; which authors effectively demonstrated using real-world experiments. My main reservations are in terms of the technical novelty of the paper and the narrow scope of experimental evaluation.

summaries: This paper presents a way to learn from demonstrations with weak or no labels. The premise behind this paper is that even when humans provide labels during a demonstration, those labels often do not fully describe the data (e.g., the human may say "soft" when "fast" would also apply). This paper presents a technique that uses latent variables to model the uncertainty over a group of class labels that could describe the task (e.g., slow, soft, left-of-object). The variables are modeled such that the observation is conditionally independent of the human provided labels given the latent variables. This allows the human provided labels to be decoupled (or disentangled as the paper calls it) from the observations. By doing so, it is possible to have only partial labels (weak labels). This model was applied to a task where a human would teleoperate a robot arm and apply a dabbing motion in relation to an object in the scene. The operator would provide only one of several possible applicable labels for each demonstration. The results show that the models using the weak labeling out-performed models with no labeling.

abstractText: Robotic manipulation tasks, such as wiping with a soft sponge, require control from multiple rich sensory modalities. Human-robot interaction, aimed at teaching robots, is difficult in this setting as there is potential for mismatch between human and machine comprehension of the rich data streams. We treat the task of interpretable learning from demonstration as an optimisation problem over a probabilistic generative model. To account for the high-dimensionality of the data, a high-capacity neural network is chosen to represent the model. The latent variables in this model are explicitly aligned with high-level notions and concepts that are manifested in a set of demonstrations. We show that such alignment is best achieved through the use of labels from the end user, in an appropriately restricted vocabulary, in contrast to the conventional approach of the designer picking a prior over the latent variables. Our approach is evaluated in the context of two table-top robot manipulation tasks performed by a PR2 robot – that of dabbing liquids with a sponge (forcefully pressing a sponge and moving it along a surface) and pouring between different containers. The robot provides visual information, arm joint positions and arm joint efforts. We have made videos of the tasks and data available see supplementary materials at: https://sites.google.com/view/weak-label-lfd.

authors:
Paszke Adam
Gross Sam
Chintala Soumith
Chanan Gregory
Yang Edward
D Zachary
Lin Zeming
Desmaison Alban
Antiga Luca
Lerer Adam

title: Automatic differentiation in pytorch

venue: In Proceedings of Neural Information Processing Systems,

authors:
Samuel K. Ainsworth
Nicholas J. Foti
Adrian K.C. Lee
Emily B. Fox

title: oi-VAE: Output interpretable VAEs for nonlinear group factor analysis

venue: Proceedings of the 35th International Conference on Machine Learning,

authors:
Brenna D Argall
Sonia Chernova
Manuela Veloso
Brett Browning

title: A survey of robot learning from demonstration

venue: Robotics and autonomous systems,

authors:
Shaojie Bai
J Zico Kolter
Vladlen Koltun

title: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling

venue: arXiv preprint arXiv:1803.01271,

authors:
Apratim Bhattacharyya
Bernt Schiele
Mario Fritz

title: Accurate and diverse sampling of sequences based on a “best of many

venue: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Tian Qi Chen
Xuechen Li
Roger B Grosse
David K Duvenaud

title: Isolating sources of disentanglement in variational autoencoders

venue: In Advances in Neural Information Processing Systems,

authors:
Xi Chen
Yan Duan
Rein Houthooft
John Schulman
Ilya Sutskever
Pieter Abbeel

title: Infogan: Interpretable representation learning by information maximizing generative adversarial nets

authors:
Adrià Colomé
Gerhard Neumann
Jan Peters
Carme Torras

title: Dimensionality reduction for probabilistic movement primitives

venue: In 2014 IEEE-RAS International Conference on Humanoid Robots,

authors:
Charles Fefferman
Sanjoy Mitter
Hariharan Narayanan

title: Testing the manifold hypothesis

venue: Journal of the American Mathematical Society,

title: The symbol grounding problem

venue: Physica D: Nonlinear Phenomena,

authors:
Irina Higgins
Loic Matthey
Arka Pal
Christopher Burgess
Xavier Glorot
Matthew Botvinick
Shakir Mohamed
Alexander Lerchner

title: beta-vae: Learning basic visual concepts with a constrained variational framework

authors:
Yordan Hristov
Daniel Angelov
Michael Burke
Alex Lascarides
Subramanian Ramamoorthy

title: Disentangled relational representations for explaining and learning from demonstration

venue: In Conference on Robot Learning (CoRL),

title: Introduction to learning bayesian networks from data

authors:
Eric Jang
Shixiang Gu
Ben Poole

title: Categorical reparametrization with gumble-softmax

venue: In International Conference on Learning Representations (ICLR 2017). OpenReview

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Diederik P Kingma
Max Welling

title: Auto-encoding variational bayes

venue: arXiv preprint arXiv:1312.6114,

authors:
Durk P Kingma
Shakir Mohamed
Danilo Jimenez Rezende
Max Welling

title: Semi-supervised learning with deep generative models

venue: In Advances in neural information processing systems,

authors:
Jens Kober
Betty Mohler
Jan Peters

title: Learning perceptual coupling for motor primitives

venue: In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems,

authors:
John E Laird
Kevin Gluck
John Anderson
Kenneth D Forbus
Odest Chadwicke Jenkins
Christian Lebiere
Dario Salvucci
Matthias Scheutz
Andrea Thomaz
Greg Trafton

title: Interactive task learning

venue: IEEE Intelligent Systems,

authors:
Francesco Locatello
Stefan Bauer
Mario Lucic
Gunnar Raetsch
Sylvain Gelly
Bernhard Schölkopf
Olivier Bachem

title: Challenging common assumptions in the unsupervised learning of disentangled representations

authors:
Michael D. Noseworthy
Rohan Paul
Subhro Roy
Daehyung Park
Nicholas Roy

title: Taskconditioned variational autoencoders for learning movement

authors:
Alexandros Paraschos
Christian Daniel
Jan R Peters
Gerhard Neumann

title: Probabilistic movement primitives

authors:
Antti Rasmus
Mathias Berglund
Mikko Honkala
Harri Valpola
Tapani Raiko

title: Semisupervised learning with ladder networks

title: Hybrid position/force control of manipulators

venue: ASME, J. of Dynamic Systems, Measurement, and Control,

authors:
Stéphane Ross
Geoffrey Gordon
Drew Bagnell

title: A reduction of imitation learning and structured prediction to no-regret online learning

venue: In Proceedings of the fourteenth international conference on artificial intelligence and statistics,

authors:
Elmar Rueckert
Jan Mundo
Alexandros Paraschos
Jan Peters
Gerhard Neumann

title: Extracting low-dimensional control variables for movement primitives

venue: In 2015 IEEE International Conference on Robotics and Automation (ICRA),

title: Dynamic movement primitives-a framework for motor control in humans and humanoid robotics

venue: In Adaptive motion of animals and machines,

authors:
Kihyuk Sohn
Honglak Lee
Xinchen Yan

title: Learning structured output representation using deep conditional generative models

title: The physical symbol grounding problem

venue: Cognitive Systems Research,

authors:
Jacob Walker
Carl Doersch
Abhinav Gupta
Martial Hebert

title: An uncertain future: Forecasting from static images using variational autoencoders

venue: In European Conference on Computer Vision,

authors:
Nick Watters
Loic Matthey
Chris P Burgess
Alexander Lerchner

title: Spatial broadcast decoder: A simple architecture for disentangled representations in vaes

heading: 1 INTRODUCTION

text:
Learning from Demonstration (LfD) (Argall et al., 2009) is a commonly used paradigm where a potentially imperfect demonstrator desires to teach a robot how to perform a particular task in its environment. Most often this is achieved through a combination of kinaesthetic teaching and supervised learning—imitation learning (Ross et al., 2011). However, such approaches do not allow for elaborations and corrections from the demonstrator to be seamlessly incorporated. As a result, new demonstrations are required when either the demonstrator changes the task specification or the agent changes its context—typical scenarios in the context of interactive task learning (Laird et al., 2017). Such problems mainly arise because the demonstrator and the agent reason about the world by using notions and mechanisms at different levels of abstraction. An LfD setup, which can accommodate abstract user specifications, requires establishing a mapping from the high-level notions humans use—e.g. spatial concepts, different ways of applying force—to the low-level perceptive and control signals robot agents utilise—e.g. joint angles, efforts and camera images. With this in place, any constraints or elaborations from the human operator must be mapped to behaviour on the agent’s side that is consistent with the semantics of the operator’s desires.
Concretely, we need to be able to ground (Vogt, 2002; Harnad, 1990) the specifications and symbols used by the operator in the actions and observations of the agent. Often the actions and the observations of a robot agent can be high-dimensional—high DoF kinematic chains, high image resolution, etc.—making the symbol grounding problem non-trivial. However, the concepts we need to be able to ground lie on a much-lower-dimensional manifold, embedded in the high-dimensional data space (Fefferman et al., 2016). For example, the concept of pressing softly against a surface manifests itself in a data stream associated with the 7 DoF real-valued space of joint efforts, spread across multiple time steps. However, the essence of what differentiates one type of soft press from another nearby concept can be summarised conceptually using a lower-dimensional abstract space. The focus of this work is finding a nonlinear mapping (represented as a high-capacity neural model) between such a
low-dimensional manifold and the high-dimensional ambient space of cross-modal data. Moreover, we show that, apart from finding such a mapping, we can also shape and refine the low-dimensional manifold by imposing specific biases and structures on the neural model’s architecture and training regime.
In this paper, we propose a framework that allows human operators to teach a PR2 robot about different spatial, temporal and force-related aspects of a robotic manipulation task, using tabletop dabbing and pouring as our main examples. They serve as concrete representative tasks that incorporate key issues specific to robotics (e.g. continuous actions, conditional switching dynamics, forceful interactions, and discrete categorizations of these). Numerous other applications require the very same capability. Our main contributions are:
• A learning method which incorporates information from multiple high-dimensional modalities—vision, joint angles, joint efforts—to instill a disentangled low-dimensional manifold (Locatello et al., 2019). By using weak expert labels during the optimisation process, the manifold eventually aligns with the human demonstrators’ ‘common sense’ notions in a natural and controlled way without the need for separate post-hoc interpretation. • We release a dataset of subjective concepts grounded in multi-modal demonstrations. Us-
ing this, we evaluate whether discrete latent variables or continuous latent variables, both shaped by the discrete user labels, better capture the demonstrated continuous notions.

heading: 2 IMAGE-CONDITIONED TRAJECTORY & LABEL MODEL

text:
In the task we use to demonstrate our ideas, which is representative of a broad class of robotic manipulation tasks, we want to control where and how a robot performs an action through the use of a user specification defined by a set of coarse labels—e.g. “press softly and slowly behind the cube in the image”. In this context, let, x denote a K × T dimensional trajectory for K robot joints and a fixed time length T , y denote a set of discrete labels semantically grouped in N label groups G = {g1, . . . , gN} (equivalent to multi-label classification problem) and i denote an RGB image1. The labels y describe qualitative properties of x and x with respect to i—e.g. left dab, right dab, hard dab, soft dab, etc. We aim to model the distribution of demonstrated robot-arm trajectories x and corresponding user labels y, conditioned on a visual environment context i. This problem is equivalent to that of structured output representation (Walker et al., 2016; Sohn et al., 2015; Bhattacharyya et al., 2018)—finding a one-to-many mapping from i to {x,y} (one image can be part of the generation of many robot trajectories and labels). For this we use a conditional generative model, whose latent variables c = {cs, ce, cu} can accommodate the aforementioned mapping—see Figure 2. The meaning behind the different types of latent variables—cs, ce and cu—is elaborated in section 3.
1What i actually represents is a lower-dimensional version of the original RGB image I. The parameters of the image encoder are jointly optimised with the parameters of the recognition and decoder networks
We want the variability in the trajectory and label data to be concisely captured by c. We choose by design the dimensionality of the latent variables to be much lower than the dimensionality of the data. Therefore, c is forced to represent abstract properties of the robot behaviour which still carry enough information for the behaviour to be usefully generated—absolute notions like speed, effort, length, and relative spatial notions which are grounded with respect to the visual context. Another way to think of c is as a continuous version of y which can therefore incorporate nuances of the same label. In the existing literature discrete labels are usually represented as discrete latent variables—e.g. digit classes (Kingma et al., 2014).
However, the discrete labels humans use are a rather crude approximation to the underlying continuous concepts—e.g. we may have the notion for a soft and a softer dab even though both would be alternatively labelled as soft. For this reason, we use continuous latent variables, shaped by discrete labels, to represent these subjective concepts.
The joint distribution over x and y, conditioned on i, is modelled according to Eq. 1. A prior of choice—isotropic Gaussian, Uniform, Categorical—is placed over the latent variables c which are independent of the image.
p(x,y|i) = ∫ p(x,y|i, c)p(c)dc = ∫ p(x|c, i)p(y|c)p(c)dc (1)
We choose to represent the distribution over x and y as Gaussian and Multinomial distributions respectively. The parameters of these distributions are defined as nonlinear functions (of the image context and latent variables) which are represented as the weights θ of a neural network pθ—µθ(·) is a vector of means, σθ(·) is a vector of log variances and πθ(·) is a vector of probabilities.
pθ(x|c, i) = N (x|µθ(c, i),σθ(c, i)) (2) pθ(y|c) = Cat(y|πθ(c)) (3)
The parameters of pθ are optimised by maximising the variational lower bound (VLB) of the data distribution—see Eq. 4. We additionally optimise a recognition network qφ, parametrised by φ, which acts as an approximation to the posterior of c, also modelled as a Gaussian. The recognition network qφ is used to find good candidates of c, for given x and i, thus making the optimisation of θ, tractable (Kingma & Welling, 2013). The posterior is conditioned on the environment context as well, since i and c are conditionally dependent, once x is observed (Husmeier, 2005). Intuitively, we want some of the latent variables to represent relative spatial concepts—e.g. left of the cube in the image. For such concepts, under the same robot trajectory x we should get different latent codes, given different contexts i. The omission of y from qφ means that we might not be able to fully close the gap between the data distribution and its VLB, leading to a less efficient optimisation procedure (due to higher variance). That is mainly owing to the fact that we use weak supervision—only for some demonstrations do we have labels y. The derivation of and commentary on the optimised VLB are provided in the Supplementary Materials.
log p(x,y|i) ≥ Eqφ(c|x,i)(log pθ(x,y|c, i))− DKL(qφ(c|x, i)||p(c)) (4) qφ(c|x, i) = N (c|µφ(x, i),σφ(x, i)) (5)

heading: 3 WEAK LABELLING AND USER SPECIFICATION

text:
Interpretability through Weak Labels Even though the latent variables are constrained to be useful both for the tasks of trajectory and label generation, nothing forces them to carry the intuitive absolute and relative notions described in Section 2 in a factorised fashion—e.g. for dabbing, c1 to correspond to the effort applied when pressing against the table, c2 to correspond to how quickly the robot presses against the table, etc. Under an optimised model, those notions can be captured in c but with no guarantee that they will be represented in an interpretable way— i.e. c needs to be
disentangled. We achieve disentanglement in the latent space by establishing a one-to-one correspondence between a subset of the latent axes—cs, ce—and the concept groups in G (Hristov et al., 2018). cs is optimised to represent spatial notions, ce—effort-related and temporal notions. The rest of the latent dimensions—cu—encode features in the data which don’t align with the semantics of the labels and their concept groups but are still necessary for good trajectory reconstruction (Figure 2). Under these assumptions the label likelihood from Eq. 3 becomes:
pθ(y|{cs, ce}) = |G|∏ j 1{yj 6=∅}Cat(yj |πj(cj)) (6)
Labelling is weak/restricted since each demonstration is given just a single label from a single concept group. This is meant to represent a more natural LfD scenario, where the main task of the expert is still to perform a number of demonstrations, rather than to exhaustively annotate each demonstration with relevant labels from all groups. For example, a demonstrator might say “this is how you press softly” and proceed to perform the demonstration (as is common, say, in an instructional video). The demonstrated behaviour might also be slow and short but the demonstrator may not have been explicitly thinking about these notions, hence may not have labelled accordingly. The missing label values are incorporated with an indicator function in Eq. 6.
Condition on User Specification Apart from being used in the optimisation objective, the weak labels are also used in an additional conditioning step at test time. The generative process we have consists of first sampling values for c from its prior. These are then passed together with i through pθ in order to get distributions over x and y, from which we can sample. However, we are really interested in being able to incorporate the provided information about the labels y into the sampling of semantically aligned parts of c. Specifically, we are interested in being able to generate robot behaviours which are consistent with chosen values for y (what we call a user specification). Postoptimising the θ and φ parameters, we choose to approximate the posterior over c for each label l in each concept group j with a set of Gaussian distributions N (µjl,Σjl). We use Maximum Likelihood Estimation for µjl and Σjl over a set of samples from c. The samples are weighted with the corresponding label likelihood pθ(yj = l|πj(cj)). As a result, the process of generating a trajectory x ∼ pθ(x|i, c) can be additionally conditioned on an optional user specification y = {y1, . . . , y|G|}, through c, in a single-label fashion (Eq. 7) or a compositional fashion (Eq. 8). Pseudo-code for the generation of trajectories for both types of conditioning is provided in the Supplementary Materials.
c ∼ p(c|yj = l) = N (c|µjl,Σjl) (7) ∀j ∈ {1, . . . , |c|} : cj ∼ p(cj |yj = l) = {
prior, if yj = ∅ or j > |G| N (c|µjl,Σjl)(j), otherwise
(8)

heading: 4 METHODOLOGY

text:
In terms of a concrete model architecture, our model is a Conditional Variational Autoencoder (CVAE) (Sohn et al., 2015) with an encoding network qφ and a decoding network pθ. The parameters of both networks are optimised jointly using methods for amortised variational inference and a stochastic gradient variational bayes estimation (Kingma & Welling, 2013).2
Encoding & Decoding Networks Due to the diversity of modalities that we want to use, qφ is implemented as a combination of 2D convolutions (for the image input), 1D convolutions (for the trajectory input) and an MLP that brings the output of the previous two modules to the common concept manifold c. For the decoding network, we implement a Temporal Convolutional Network (TCN) (Bai et al., 2018) which operates over the concatenation h of a single concept embedding c and a single image encoding i. However, the TCN takes a sequence of length T and transforms it to another sequence of length T , by passing it through a series of dilated convolutions, while h is equivalent to a sequence of length 1. Thus, we tile the concatenated vector T times and to each
2The models are implemented in PyTorch (Adam et al., 2017) and optimised using the Adam optimiser (Kingma & Ba, 2014)
instance of that vector hi, i ∈ {1, . . . , T}, we attach a time dimension ti = iT . This broadcasting technique has previously shown to achieve better disentanglement in the latent space both on visual (Watters et al., 2019) and time-dependent data (Noseworthy et al., 2019). We only set up the mean µθ of the predicted distribution over x to be a non-linear function of [h; t] while σ is fixed, for better numerical stability during training. This allows us to use the L2 distance—Mean Squared Error (MSE)—between the inferred trajectories µθ, acting as a reconstruction, and the ground truth ones.
Label Predictors For each concept group gj we take values from the corresponding latent axis cj and pass it through a single linear layer, with a softmax activation function, to predict a probability vector πj . Maximising the label likelihood is realised as a Softmax-Cross Entropy (SCE) term in the loss function. Optimising this term gives us a better guarantee that some of the latent axes will be semantically aligned with the notions in the label groups. The fact that some labels might be missing is accounted for with an indicator function which calculates ∇θ,∇φ with respect to the SCE, only if yj is present.
Weighted Loss The full loss function that we optimise is presented in Eq. 9. It is composed of three main terms—the trajectory MSE, equivalent to the negative trajectory log-likelihood, the label SCE, equivalent to the negative weak label log-likelihood and the KL divergence between the fit amortised posterior over c and its prior. The concrete values of the three coefficients—α, β, γ—are discussed in the Supplementary Materials.
min θ,φ,w L(x,y, I) = βDKL(qφ(c|x, I)︸ ︷︷ ︸ amortised posterior || pθ(c)︸ ︷︷ ︸ prior )+αEqφ(c|x,I)(log pθ(x|c, I)︸ ︷︷ ︸ MSE )+γ
|G|∑ i 1{yi 6=∅}H(ciw T i , yi)︸ ︷︷ ︸
SCE
(9)

heading: 5 EXPERIMENTS

text:
The models we experiment with include a Variational Autoencoder (Kingma & Welling, 2013) with a Gaussian prior (VAE), an Adversarial Autoencoder (Makhzani et al., 2015) with a Uniform prior (AAE) and a VAE with mixed continuous and categorical latent variables (GS). For the last model, the continuous variables have a Gaussian prior and the categorical variables utilise the GumbelSoftmax reparametrisation (Jang et al., 2017). Each of the three models, is trained while utilising the weak labels—VAE-weak, AAE-weak, GS-weak—and with no label information—VAE, AAE, GS. Whether or not labels are used during training is controlled by setting γ = 0 in Eq. 9.
The setup used in the experiments consists of a PR2 robot, an HTC Vive controller and headset, and a tabletop with a single object on it—a red cube. Multiple pressing motions on the surface of the table are demonstrated by teleoperating the end effector of the robot. The pose of the end effector, with respect to the robot’s torso, is mapped to be the same as the pose of the controller (held by the demonstrator) with respect to the headset (behind the demonstrator in Figure 4).
Data A total of 100 demonstrations were captured from the PR2, consisting of: • Initial frame of each demonstration:
128x128-pixel Kinect2 image (RGB); • 7-DoF joint angle positions;
• 7-DoF joint efforts; • 5 groups of discrete weak labels;
Each demonstration has a single label attached to it. In total, we have 4 spatial symbols—where in the image, with respect to the red cube, do we dab—and 6 force-related symbols—how do we dab (Table 1). Figure 3 provides a graphical depiction demonstrating the differences between the different force-relate groupings qualitatively.
All trajectories are standardised to be of fixed length—the length of the longest trajectory (240 timesteps)—by padding them with the last value for each channel/joint. Additionally, both the joint posi-
tions and efforts are augmented with Gaussian noise or by randomly sliding them leftwards (simulating an earlier start) and padding accordingly. The size of the total dataset after augmentation is 1000 demonstrations which are split according to a 90-10 training-validation split. The size of the latent space is chosen to be |c| = 8. In the context of the problem formulation in Section 2, cs = {c0, c1}, ce = {c2, c3, c4} and cu = {c5, c6, c7}.
plementary Materials. Average accuracy across all images and all samples is reported for each label. For the labels of a spatial nature, for each 7-dimensional (7 DoFs) robot state xi,x = {x1, . . . , xT }, we compute the corresponding end-effector pose pi through the forward kinematics K of the robot. Using the camera model of the Kinect2 sensor, each pose pi is projected in the image we condition on. We report the Mean Absolute Error (MAE) in normalised pixel coordinates between the dab location and the landmark location (in pixel coordinates) for the misclassified trajectories. This gives us a sense of how far from the desired ones the generated samples are. The dab location is inferred as the point in the end-effector pose trajectory with the lowest z coordinate. For example, if we have conditioned on the left label but some of the sampled trajectories result in a dab to the right of the cube in the image, we report how far from the true classification of left was the robot when it touched the table.
Conditioning on a Composition of Labels Since the concepts we encode in the latent space of the model should be independent of each other they can be composed together—e.g. a trajectory can be simultaneously hard, long and fast. We examine how well is this conceptual independence preserved in the latent space by generating trajectories conditioned on all possible combinations of labels regarding effort, speed and length of press—8 possible combinations in total. We still use the Gaussian distributions for each label in each concept group but only take the values along the relevant latent axis—Eq. 8. To fully close the loop, the trajectories which we sample from the model could further be executed on the physical robot through a hybrid position/force controller (Reibert, 1981). However, such an evaluation is beyond the scope of the paper.

heading: 6 RESULTS & DISCUSSION

text:
Our experiments demonstrate that the models which utilise weak user labels within our proposed setup can more reliably generate behaviours consistent with a given spatial or force-related user specification, in contrast to baseline models which do not benefit from such information. Moreover, we find that continuous latent variables better capture the independent continuous notions which underlie the discrete user labels in comparison to discrete latent variables. Below we present quantitative results from the experiments with further qualitative results in the Supplementary Materials.
When conditioning on a single force label—Table 3—the best-performing models are VAE-weak and AAE-weak. In the worse-performing cases—VAE, GS—it would appear that the models suffer from partial mode collapse. For instance, in Table 3, penultimate row, the VAE can not represent soft trajectories, most samples are hard. Interestingly AAE and AAE-weak perform similarly well. We attribute that to the fact the Uniform prior used does not have a discrete number of modes to which the optimised posterior might potentially collapse. Simultaneously, regardless of their prior, none of the weak models suffers from mode collapse, since the label information helps for the corresponding notions to be more distinguishable in the latent space. All models have high accuracy, when conditioned on a spatial label—Table 2. However, GS-weak, VAE-weak and AAEweak appear to be closer to the true conditioned-on concept when a bad trajectory is generated, as measured by the lower Mean Absolute Error of the image-projected end-effector positions.
While single-label conditioning is possible under all models, it is only VAE-weak, AAE-weak and GS-weak which provide a reliable mechanism for composing labels from different independent concept groups. That is possible due to the explicit alignment of each concept and its labels with a single latent axis. Trajectory generation is done after conditioning on compositions of labels from the three independent concept groups designating effort—see Table 1 (right column). Specifically, the difference between the sampled latent values for hard long slow and hard long fast, for example, are only along the axis used to predict slow from fast—c4 in this case. If the generated trajectories match their specification (high measured accuracy) then indeed the perturbed axis is aligned with the notion behind the perturbed labels. Results from this experiment are presented in Figure 5. The better-performing models are the ones with continuous latent variables. While the VAE-weak and AAE-weak perform equally well for most label combinations and do manage to
encode independent concepts as such (along specific latent axes), the model with the mix of continuous and categorical latent variables (GS-weak) does not match their results. Taking the second row in Table 3 into account, this means that the categorical model (which has more constrained capacity than the VAE-weak and AAE-weak) can capture the data distribution (shows good singlelabel-conditioning reconstructions) but in an entangled fashion. Namely, going from slow to fast corresponds to perturbing more than a single latent axis simultaneously. This further supports our initial hypothesis that while discrete, the weak labels only provide a mechanism for shaping the latent representation of what are otherwise continuous phenomena.

heading: 7 POURING EXPERIMENT

text:
An additional pouring task is demonstrated where the manner of execution varies spatially and in terms of how it is executed—see Table 4 and Figure 6. 20 demonstrations are captured for each label. The size of the latent space is kept to be |c| = 8. In the context of the problem formulation in Section 2, cs = {c0}, ce = {c1, c2} and cu = {c3, c4, c5, c6, c7}.
When compared to the dabbing setup, this task presents a more significant challenge in identifying the switching latent dynamics for pouring in each cup (compared to the single red cube in the previous task, acting as a reference point for resolving spatial prepositions).
Interpolation experiments for the VAE-weak model are presented in Figure 7 for 4 test cup configurations. The first latent dimension of the model—c0, optimised to predict in which cup to pour—is linearly interpolated in the range [−2, 2]. In all four rows, we can see the end effector projections in the image plane, through the robot kinematics and camera model, showing a gradual shift from initially pouring in the red cup to eventually pouring in the blue cup. The color evolution of each end-effector trajectory from cyan to magenta signifies time progression. Moreover, given the disentangled nature of the VAE-weak model, we also generate robot trajectories conditioned on specific values for the 3 latent axes (c0, c1, c2), consistent with example label compositions—see Figure 8. For picking the right value for each axis we use the post-training-estimated Normal distributions for each label (consult Section 3). Additional qualitative results and sensitivity analysis of the model to the heterogeneous input modalities are presented in Appendix E and the supplementary website 3.
3https://sites.google.com/view/weak-label-lfd

heading: 8 RELATED WORK

text:
Methods that utilise high-parameter neural models to learn disentangled low-dimensional representation achieve that by either tuning the optimised loss function (Higgins et al., 2017; Chen et al., 2018), imposing crafted priors over the latent space (Chen et al., 2016) or using weak supervision (Hristov et al., 2019; Rasmus et al., 2015). While being able to produce interpretable manifolds, most of these approaches focus on modelling visual data and respectively visually-manifested concepts, with minor exceptions—e.g. modelling motion capture data (Ainsworth et al., 2018). In contrast, we solve a problem involving multiple heterogeneous modalities—vision, forces, joint angles. Moreover, in addition to separating factors of variation in the data, to achieve latent space interpretability, we also deal with the problem of entanglement across modalities—i.e. which factors relate to which data modalities.
Dynamic Movement Primitives (Schaal, 2006) and Probabilistic Movement Primitives (Paraschos et al., 2013) are commonly used to model dynamical systems as ‘point-mass attractors with a nonlinear forcing term’. However, the resulting control parameters can be high-dimensional and unintuitive for end-user manipulation. Performing dimensionality reduction (Colomé et al., 2014) or imposing hierarchical priors (Rueckert et al., 2015) are both ideas seen in the literature as a means of making the high-dimensional parameter space more meaningful to the human demonstrator. These have the advantage of yielding a clearer idea of how changes in the optimised parameters result in deterministic changes in the generated behaviours. However, most of these approaches limit themselves to end-effector trajectories, or – at most – making use of visual input (Kober et al., 2008).
Noseworthy et al. (2019) explore disentangling task parameters from the manner of execution parameters, in the context of pouring. They utilise an adversarial training regime, which facilitates better separation of the two types of parameters in the latent space. However, it is assumed that the task parameters are known a priori. Interpretation of the learned codes is achieved post-training by perturbing latent axis values and qualitatively inspecting generated trajectories (standard evaluation technique for models which are trained unsupervised). We argue that through the use of weak supervision through discrete labels we have finer control over the ‘meaning’ of latent dimensions.

heading: 9 CONCLUSION

text: We recontextualise the problem of interpretable multi-modal LfD as a problem of formulating a conditional probabilistic model. In the example task of table-top dabbing, we utilise weak discrete labels from the demonstrator to represent abstract notions, manifested in a captured multi-modal robot dataset. Through the use of high-capacity neural models and methods from deep variational inference, we show how to align some of the latent variables of the formulated probabilistic model with high-level notions implicit in the demonstrations.

heading: ACKNOWLEDGMENTS

text: This work is partly supported by funding from the Turing Institute, as part of the Safe AI for surgical assistance project. Additionally, we thank the members of the Edinburgh Robust Autonomy and Decisions Group for their feedback and support.

heading: B LABEL CONDITIONING & EVALUATION HEURISTICS

text:
Algorithms 1 and 2 provide pseudo-code for the trajectory generation procedures described in Section 3. The main difference can be summarised as following—when the generative process is conditioned on a single label l, all 8 dimensions of the latent samples are drawn from a single 8-dimensional Gaussian distribution associated with that label and its concept group—line 2 in Algorithm 1. However, when the process is conditioned on a composition of labels from different groups—a user specification—the sampling procedure differs as c is iteratively built. For each latent axis cj , if a label is specified for the concept group aligned with cj , only the values along the j-th axis are taken of a sample from the corresponding 8-dimensional Gaussian. If no label is given, cj is sampled from a Standard Normal distributionN (0, 1) (or the respective prior of choice.
Same applies for latent dimensions which have not been aligned with any concept groups, since |c| > |G|—lines 2 to 9 in Algorithm 2.
Algorithm 1: Trajectory generation conditioned on a single label Input: Visual Scene image I Input: Single user label y from concept group g Input: Image encoder, part of pθ Input: Decoder network qφ Input: set of Gaussian distribution for each label in each concept group: K = {{N (µjl,Σjl)}, j ∈ {1, . . . , |G|}, l ∈ {1, . . . , |gj |} Output: Generated trajectory x̂ of time length T
1 encode image input i← pθ(I); 2 sample latent values c ∼ N (c|µjl,Σjl), such that j = g, l = y; 3 concatenate c and i: h← [c; i]; 4 tile h T times; 5 attach a last time dimension h = [h; t], t = {t1, . . . , tT }, ti = iT ; 6 generate trajectory x̂← qφ(x|h); 7 return x̂;
Algorithm 2: Trajectory generation conditioned on a composition of labels Input: Visual Scene image I Input: User label specification y = {y1, . . . , y|G|}, potentially one label for each concept group Input: Image encoder, part of pθ Input: Decoder network qφ Input: set of Gaussian distribution for each label in each concept group: K = {N (c|µjl,Σjl)}, j ∈ {1, . . . , |G|}, l ∈ {1, . . . , |gj |} Output: Generated trajectory x̂ of time length T
1 encode image input i← pθ(I); 2 init c = []; 3 for each latent dimension cj ∈ c do 4 if yj 6= ∅ and j <= |G| then 5 sample w ∼ N (w|µjl,Σjl), such that, l = yj ; 6 only take values along the j-th dimension: append w[j] to c; 7 else 8 sample w ∼ N (w|0, 1); 9 append w to c;
10 concatenate c and i: h← [c; i]; 11 tile h T times; 12 attach a last time dimension h = [h; t], t = {t1, . . . , tT }, ti = iT ; 13 generate trajectory x̂← qφ(x|h); 14 return x̂;
The hand-designed heuristics used for reporting accuracy on the the generated trajectories have concrete semantics and values which have been chosen after closely inspecting the captured dataset, the given user labels and the robot configuration during the demonstrations. :
• spatial labels—left, right, front, behind—the location of the red object in the scene is extracted using standard computer vision techniques based on color segmentation. The position of the cube is inferred as the center of mass of all red pixels detected. From the overall end-effector trajectory, derived through the robot forward kinematics, the pressing location is chosen as the end-effector position with the lowest z coordinate. That position is then projected into the image plane in order to determine where with respect to the detected object position did the robot touch the table.
• soft vs hard trajectories - the generated trajectories have normalised values in the range [−1, 1]. A soft trajectory is considered as such if specific joint efforts are below a particular threshold. More specifically:
– shoulder lift joint efforts are below 0.5. – upper arm roll joint effort are above -0.5.
• short vs long trajectories - the generated trajectories have length of T = 240. A short trajectory is considered as such if efforts in the range of [max effort-0.2, max effort] are maintained for more than 50 time steps. Otherwise the trajectory is deemed long. The heuristic operates over the following joints:
– shoulder lift joint efforts – upper arm roll joint efforts
• slow vs fast trajectories - a slow trajectory is considered as such if the time it takes to reach maximal joint effort is more than 80 time steps. Otherwise the trajectory is deemed fast. The heuristic operates over the following joints:
– shoulder lift joint efforts – upper arm roll joint efforts
For resolving robot joint names, please consult Figure 9 below:

heading: C ARCHITECTURE DETAILS

text:
The model architectures are implemented in the PyTorch framework4 and is visually presented in Figure 10. The image encoder network takes as input a single RGB 128x128 pixel image. The trajectory encoder takes a single 14-dimensional, 240 time-step-long trajectory. Their outputs both feed into an MLP network which, after a series of nonlinear transformations, outputs parameters of a distribution over the latent space c. Through the reparametrisation trick Kingma & Welling (2013) we sample values for c. Through a residual connection, the output of the image encoder is concatenated to the sampled latent values. The resultant vector is tiled 240 times, extended with a time dimension, and fed into a TCN decoder network to produce reconstructions for the original 14-dimensional input trajectories.
Across all experiments, training is performed for a fixed number of 100 epochs using a batch size of 8. The dimensionality of the latent space |c| = 8 across all experiments. The Adam optimizer (Kingma & Ba, 2014) is used through the learning process with the following values for its parameters—(learningrate = 0.001, β1 = 0.9, β2 = 0.999, eps = 1e− 08, weightdecayrate = 0, amsgrad = False).
4https://pytorch.org/docs/stable/index.html

heading: FC (256)

heading: FC (256)

text:
For all experiments, the values (unless when set to 0) for the three coefficients from Equation 9 are:
• α = 1, β = 0.1, γ = 10
The values are chosen empirically in a manner such that all the loss terms have similar magnitude and thus none of them overwhelms the gradient updates while training the full model.

heading: D QUALITATIVE EVALUATION FOR DABBING

text: Figures 11 to 15 present the qualitative results from perturbing in a controlled way the latent space optimised under the VAE-weak model. For figure the first row of plots represents the generated joint angle positions for each of the 7 joints, for each of the 5 drawn samples, the second row—the generated joint efforts—and the third row the corresponding end-effector positions, from the forward robot kinematics, projected in the image plane. As we can see, perturbing c0 and c1 corresponds to clear and consistent movement of the end effector (through the generated joint position profiles) while there is not much change in the generated effort profiles—Figures 11 and 12. Simultaneously, we observe the opposite effect in other 3 Figures. Perturbations in c2 correspond solely changes in the output efforts in the shoulder lift and upper arm rolls joints—Figure 13 (i) and (j). Perturbations in c3 correspond to changes in length for which the joint efforts are exerted but does not change their magnitude or where they are applied on the table—Figure 14. And finally, perturbations in c4 correspond to changes on how quickly the maximum efforts are reached - most noticeable in the changing slope of the shoulder lift and upper arm rolls joints—Figure 15 (i) and (j). As a result, we can conclude that the latent dimensions, which we aligned with the semantically grouped weak labels, have indeed captured the notions and concepts which underlie the labels.

heading: E MODEL SENSITIVITY ANALYSIS TO DIFFERENT DATA MODALITIES

text: As an exploration step towards analysing to what extend the auxiliary loss terms guide the model to pay attention to the different data modalities it is exposed to, we train the VAE and VAE-weak models on the demonstrations for pouring in the red cup and blue cup. The first latent dimension of the VAE-weak model is optimised to represent the conceptual variation of pouring in one of the two cups. Post-training, the values of the latent vectors c for both models are fixed and trajectory samples are drawn as the models are conditioned on four different test scenes, representing different cup configurations. Figure 16 demonstrates that, averaged across the different configurations, the VAE-weak model has greater variations in the sampled trajectories for all 7 joints of the robot arm when compared to the VAE one. This suggests that unless utilised, the input image modality might be partly ignored—the VAE does not have any auxiliary tasks to force it to use the scene image input, which can result in it just auto-encoding the trajectory data through c. In contrast, predicting where the robot pours in the scene at an abstract level is both a function of the joint angle trajectory and the image inputs—utilised through the classification loss of VAE-weak. Thus, for the same specification (fixed c), conditioning on different scenes leads to different joint trajectories.


summaries: This paper studies fine-tuning BERT-like pretrained language models (PLMs) on low resource target tasks. The authors hypothesize that the general-purpose knowledge obtained by the PLMs from pre-training might be irrelevant and redundant for a given target task. When fine-tuned onto a low resource target task, overfitting is likely to happen. To this end, a fine-tuning framework based on variational information bottleneck (VIB) is proposed to address these challenges. Specifically, the sentence representation will be mapped to a latent Gaussian variable  which compresses information in the sentence and also suppress irrelevant and redundant features, and a reconstructed version of the representation is used for task prediction. Empirical evaluations on sever datasets demonstrates the effectiveness of the method over previous research.

summaries: This work applies information bottleneck as a way to compress the pre-trained representation so that only meaningful features are employed for the target task. It is applied for the number of GLUE tasks especially focusing on low resource settings and show consistent gains over previously known strong baselines, e.g., Mixout and L2-of-difference. This work also demonstrates that the learned model has generalization capacity so that the tuned model works on out-of-domain data.

summaries: The paper proposes a method to avoid overfitting while finetuning the large pretrained models for downstream tasks on small scale datasets. It has been shown that many SOTA models usually overfit w.r.t. spurious correlations in the data and as a result fail miserably when tested for generalization on the out of domain datasets. The proposed method tries to maximally filter out task-irrelevant information in the feature vectors by minimizing the mutual information between the original features and the bottleneck features while simultaneously optimizing for performance. Experiments on several datasets show improved performance on both in-domain and out-of-domain datasets.

abstractText: While large-scale pretrained language models have obtained impressive results when fine-tuned on a wide variety of tasks, they still often suffer from overfitting in low-resource scenarios. Since such models are general-purpose feature extractors, many of these features are inevitably irrelevant for a given target task. We propose to use Variational Information Bottleneck (VIB) to suppress irrelevant features when fine-tuning on low-resource target tasks, and show that our method successfully reduces overfitting. Moreover, we show that our VIB model finds sentence representations that are more robust to biases in natural language inference datasets, and thereby obtains better generalization to out-of-domain datasets. Evaluation on seven low-resource datasets in different tasks shows that our method significantly improves transfer learning in low-resource scenarios, surpassing prior work. Moreover, it improves generalization on 13 out of 15 out-of-domain natural language inference benchmarks. Our code is publicly available in https://github.com/rabeehk/vibert.

authors:
Alex Alemi
Ian Fischer
Josh Dillon
Kevin Murphy

title: Deep variational information bottleneck

venue: In ICLR,

authors:
Yuki Arase
Jun’ichi Tsujii

title: Transfer fine-tuning: A bert case study

venue: In EMNLP,

authors:
Yonatan Belinkov
Adam Poliak
Stuart Shieber
Benjamin Van Durme
Alexander Rush

title: Don’t take the premise for granted: Mitigating artifacts in natural language inference

venue: In ACL,

authors:
Yonatan Belinkov
Adam Poliak
Stuart M Shieber
Benjamin Van Durme
Alexander M Rush

title: On adversarial removal of hypothesis-only bias in natural language inference

venue: In SEM,

authors:
Samuel Bowman
Luke Vilnis
Oriol Vinyals
Andrew Dai
Rafal Jozefowicz
Samy Bengio

title: Generating sentences from a continuous space

venue: In CoNLL,

authors:
Samuel R. Bowman
Gabor Angeli
Christopher Potts
Christopher D. Manning

title: A large annotated corpus for learning natural language inference

authors:
Daniel Cer
Mona Diab
Eneko Agirre
Iñigo Lopez-Gazpio
Lucia Specia

title: Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation

venue: SemEval,

authors:
Ciprian Chelba
Alex Acero

title: Adaptation of maximum entropy capitalizer: Little data can help a lot

authors:
Ido Dagan
Oren Glickman
Bernardo Magnini

title: The pascal recognising textual entailment challenge

venue: In MLCW,

title: Frustratingly easy domain adaptation

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: In NAACL,

authors:
Jesse Dodge
Suchin Gururangan
Dallas Card
Roy Schwartz
Noah A Smith

title: Show your work: Improved reporting of experimental results

venue: EMNLP-IJCNLP,

authors:
Jesse Dodge
Gabriel Ilharco
Roy Schwartz
Ali Farhadi
Hannaneh Hajishirzi
Noah Smith

title: Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping

authors:
William B Dolan
Chris Brockett

title: Automatically constructing a corpus of sentential paraphrases

venue: In IWP,

authors:
Yanai Elazar
Yoav Goldberg

title: Adversarial removal of demographic attributes from text data

authors:
Marzieh Fadaee
Arianna Bisazza
Christof Monz

title: Data augmentation for low-resource neural machine translation

authors:
Yichen Gong
Heng Luo
Jian Zhang

title: Natural language inference over interaction space

authors:
Suchin Gururangan
Swabha Swayamdipta
Omer Levy
Roy Schwartz
Samuel Bowman
Noah A. Smith

title: Annotation artifacts in natural language inference data

authors:
Suchin Gururangan
Tam Dang
Dallas Card
Noah A Smith

title: Variational pretraining for semi-supervised text classification

authors:
Peter Harremoës
Naftali Tishby

title: The information bottleneck revisited or how to choose a good distortion measure

venue: In ISIT,

authors:
Maximilian Igl
Kamil Ciosek
Yingzhen Li
Sebastian Tschiatschek
Cheng Zhang
Sam Devlin
Katja Hofmann

title: Generalization in reinforcement learning with selective noise injection and information bottleneck

venue: NeurIPS,

authors:
Divyansh Kaushik
Zachary C Lipton

title: How much reading does reading comprehension require? a critical investigation of popular benchmarks

authors:
Tushar Khot
Ashish Sabharwal
Peter Clark

title: Scitail: A textual entailment dataset from science question answering

venue: In AAAI,

title: Auto-encoding variational bayes

authors:
Anders Krogh
John A Hertz

title: A simple weight decay can improve generalization

venue: In NeurIPS,

authors:
Alice Lai
Yonatan Bisk
Julia Hockenmaier

title: Natural language inference from multiple premises

venue: In IJCNLP,

authors:
Cheolhyoung Lee
Kyunghyun Cho
Wanmo Kang

title: Mixout: Effective regularization to finetune large-scale pretrained language models

authors:
Xiang Lisa Li
Jason Eisner

title: Specializing word embeddings (for parsing) by information bottleneck

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

authors:
Andrew L. Maas
Raymond E. Daly
Peter T. Pham
Dan Huang
Andrew Y. Ng
Christopher Potts

title: Learning word vectors for sentiment analysis

authors:
Karimi Rabeeh Mahabadi
Yonatan Belinkov
James Henderson

title: End-to-end bias mitigation by modelling biases in corpora

authors:
Marco Marelli
Stefano Menini
Marco Baroni
Luisa Bentivogli
Raffaella Bernardi
Roberto Zamparelli

title: A sick cure for the evaluation of compositional distributional semantic models

venue: In LREC,

authors:
Marius Mosbach
Maksym Andriushchenko
Dietrich Klakow

title: On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines

authors:
Ellie Pavlick
Chris Callison-Burch

title: Most “babies” are “little” and most “problems” are “huge”: Compositional entailment in adjective-nouns

authors:
Ellie Pavlick
Travis Wolfe
Pushpendre Rastogi
Chris Callison-Burch
Mark Dredze
Benjamin Van Durme

title: Framenet+: Fast paraphrastic tripling of framenet

authors:
Jason Phang
Thibault Févry
Samuel R Bowman

title: Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks

authors:
Adam Poliak
Jason Naradowsky
Aparajita Haldar
Rachel Rudinger
Benjamin Van Durme

title: Hypothesis only baselines in natural language inference

venue: SEMEVAL,

authors:
Alec Radford
Jeff Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

authors:
Altaf Rahman
Vincent Ng

title: Resolving complex cases of definite pronouns: the winograd schema challenge

venue: In EMNPL,

authors:
Drew Reisinger
Rachel Rudinger
Francis Ferraro
Craig Harman
Kyle Rawlins
Benjamin Van Durme

title: Semantic proto-roles

venue: In TACL,

authors:
Tal Schuster
Darsh J Shah
Yun Jie Serene Yeo
Daniel Filizzola
Enrico Santus
Regina Barzilay

title: Towards debiasing fact verification models

authors:
Ohad Shamir
Sivan Sabato
Naftali Tishby

title: Learning and generalization with the information bottleneck

venue: In TCS,

authors:
Nitish Srivastava
Geoffrey Hinton
Alex Krizhevsky
Ilya Sutskever
Ruslan Salakhutdinov

title: Dropout: a simple way to prevent neural networks from overfitting

venue: In JMLR,

authors:
Naftali Tishby
Fernando C. Pereira
William Bialek

title: The information bottleneck method

venue: In Allerton,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Elena Voita
Rico Sennrich
Ivan Titov

title: The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives

authors:
Alex Wang
Amapreet Singh
Julian Michael
Felix Hill
Omer Levy
Samuel R. Bowman

title: GLUE: A multi-task benchmark and analysis platform for natural language understanding

authors:
Zhiguo Wang
Wael Hamza
Radu Florian

title: Bilateral multi-perspective matching for natural language sentences

venue: In IJCAI,

authors:
Peter West
Ari Holtzman
Jan Buys
Yejin Choi

title: Bottlesum: Unsupervised and self-supervised sentence summarization using the information bottleneck principle

authors:
Aaron Steven White
Pushpendre Rastogi
Kevin Duh
Benjamin Van Durme

title: Inference is everything: Recasting semantic resources into a unified evaluation framework

authors:
Adina Williams
Nikita Nangia
Samuel Bowman

title: A broad-coverage challenge corpus for sentence understanding through inference

authors:
Thomas Wolf
Lysandre Debut
Victor Sanh
Julien Chaumond
Clement Delangue
Anthony Moi
Pierric Cistac
Tim Rault
Rémi Louf
Morgan Funtowicz
Jamie Brew

title: Huggingface’s transformers: State-of-the-art natural language processing

authors:
Zhilin Yang
Zihang Dai
Yiming Yang
Jaime Carbonell
Russ R Salakhutdinov
Quoc V Le

title: Xlnet: Generalized autoregressive pretraining for language understanding

authors:
Sheng Zhang
Rachel Rudinger
Kevin Duh
Benjamin Van Durme

title: Ordinal common-sense inference

authors:
Tianyi Zhang
Felix Wu
Arzoo Katiyar
Kilian Q Weinberger
Yoav Artzi

title: Revisiting few-sample bert fine-tuning

authors:
Xiang Zhang
Junbo Zhao
Yann LeCun

title: Character-level convolutional networks for text classification

heading: 1 INTRODUCTION

text:
Transfer learning has emerged as the de facto standard technique in natural language processing (NLP), where large-scale language models are pretrained on an immense amount of text to learn a general-purpose representation, which is then transferred to the target domain with fine-tuning on target task data. This method has exhibited state-of-the-art results on a wide range of NLP benchmarks (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019). However, such pretrained models have a huge number of parameters, potentially making fine-tuning susceptible to overfitting.
In particular, the task-universal nature of large-scale pretrained sentence representations means that much of the information in these representations is irrelevant to a given target task. If the amount of target task data is small, it can be hard for fine-tuning to distinguish relevant from irrelevant information, leading to overfitting on statistically spurious correlations between the irrelevant information and target labels. Learning low-resource tasks is an important topic in NLP (Cherry et al., 2019) because annotating more data can be very costly and time-consuming, and because in several tasks access to data is limited.
In this paper, we propose to use the Information Bottleneck (IB) principle (Tishby et al., 1999) to address this problem of overfitting. More specifically, we propose a fine-tuning method that uses Variational Information Bottleneck (VIB; Alemi et al. 2017) to improve transfer learning in low-resource scenarios.
VIB addresses the problem of overfitting by adding a regularization term to the training loss that directly suppresses irrelevant information. As illustrated in Figure 1, the VIB component maps the sentence embedding from the pretrained model to a latent representation z, which is the only input to the taskspecific classifier. The information that is represented in z is chosen based on the IB principle, namely that all the information about the input that is represented in z should be necessary for the task. In particular, VIB directly tries to remove the irrelevant information, making it easier for the task classifier to avoid overfitting when trained on a small amount of data. We find that in low-resource scenarios, using VIB to suppress irrelevant features in pretrained sentence representations substantially improves accuracy on the target task.
∗Supported by the Viterbi Fellowship in the Center for Computer Engineering at the Technion.
Removing unnecessary information from the sentence representation also implies removing redundant information. VIB tries to find the most concise representation which can still solve the task, so even if a feature is useful alone, it may be removed if it isn’t useful when added to other features because it is redundant. We hypothesize that this provides a useful inductive bias for some tasks, resulting in better generalization to out-of-domain data. In particular, it has recently been demonstrated that annotation biases and artifacts in several natural language understanding benchmarks (Kaushik & Lipton, 2018; Gururangan et al., 2018; Poliak et al., 2018; Schuster et al., 2019) allow models to exploit superficial shortcuts during training to perform surprisingly well without learning the underlying task. However, models that rely on such superficial features do not generalize well to out-of-domain datasets, which do not share the same shortcuts (Belinkov et al., 2019a). We investigate whether using VIB to suppress redundant features in pretrained sentence embeddings has the effect of removing these superficial shortcuts and keeping the deep semantic features that are truly useful for learning the underlying task. We find that using VIB does reduce the model’s dependence on shortcut features and substantially improves generalization to out-of-domain datasets.
We evaluate the effectiveness of our method on fine-tuning BERT (Devlin et al., 2019), which we call the VIBERT model (Variational Information Bottleneck for Effective Low-Resource Fine-Tuning). On seven different datasets for text classification, natural language inference, similarity, and paraphrase tasks, VIBERT shows greater robustness to overfitting than conventional fine-tuning and other regularization techniques, improving accuracies on low-resource datasets. Moreover, on NLI datasets, VIBERT shows robustness to dataset biases, obtaining substantially better generalization to out-of-domain NLI datasets. Further analysis demonstrates that VIB regularization results in less biased representations. Our approach is highly effective and simple to implement, involving a small additional MLP classifier on top of the sentence embeddings. It is model agnostic and end-to-end trainable.
In summary, we make the following contributions: 1) Proposing VIB for low-resource fine-tuning of large pretrained language models. 2) Showing empirically that VIB reduces overfitting, resulting in substantially improved accuracies on seven low-resource benchmark datasets against conventional fine-tuning and prior regularization techniques. 3) Showing empirically that training with VIB is more robust to dataset biases in NLI, resulting in significantly improved generalization to out-of-domain NLI datasets. To facilitate future work, we will release our code.

heading: 2 FINE-TUNING IN LOW-RESOURCE SETTINGS

text:
The standard fine-tuning paradigm starts with a large-scale pretrained model such as BERT, adds a task-specific output component which uses the pretrained model’s sentence representation, and trains this model end-to-end on the task data, fine-tuning the parameters of the pretrained model. As depicted in Figure 1, we propose to add a VIB component that controls the flow of information from the representations of the pretrained model to the output component. The goal is to address overfitting in resource-limited scenarios by removing irrelevant and redundant information from the pretrained representation.
Problem Formulation We consider a general multi-class classification problem with a low-resource dataset D={xi,yi}Ni=1 consisting of inputs xi∈X , and labels yi∈Y. We assume we are also given a large-scale pretrained encoder fϕ(.) parameterized by ϕ that computes sentence embeddings for the input xi. Our goal is to fine-tune fϕ(.) onD to maximize generalization.
Information Bottleneck To specifically optimize for the removal of irrelevant and redundant information from the input representations, we adopt the Information Bottleneck principle. The objective of IB is to find a maximally compressed representation Z of the input representationX (compression loss) that
maximally preserves information about the output Y (prediction loss),1 by minimizing:
LIB = βI(X, Z)︸ ︷︷ ︸ Compression Loss − I(Z, Y )︸ ︷︷ ︸ Prediction Loss , (1)
where β≥0 controls the balance between compression and prediction, and I(.,.) is the mutual information.
Variational Information Bottleneck Alemi et al. (2017) derive an efficient variational estimate of (1):
LVIB =β E x [KL[pθ(z|x), r(z)]]+ E z∼pθ(z|x) [−logqφ(y|z)], (2)
where qφ(y|z) is a parametric approximation of p(y|z), r(z) is an estimate of the prior probability p(z) of z, and pθ(z|x) is an estimate of the posterior probability of z. During training, the compressed sentence representation z is sampled from the distribution pθ(z|x), meaning that a specific pattern of noise is added to the input of the output classifier qφ(y|z). Increasing this noise decreases the information conveyed by z. In this way, the VIB module can block the output classifier qφ(y|z) from learning to use specific information. At test time, the expected value of z is used for predicting labels with qφ(y|z). We refer to the dimensionality of z as K, which specifies the bottleneck size. Note that there is an interaction between decreasingK and increasing the compression by increasing β (Shamir et al., 2010; Harremoës & Tishby, 2007). K and β are hyper-parameters (Alemi et al., 2017).
We consider parametric Gaussian distributions for prior r(z) and pθ(z|x) to allow an analytic computation for their Kullback-Leibler divergence,2 namely r(z) = N (z|µ0,Σ0) and pθ(z|x) = N (z|µ(x),Σ(x)), where µ and µ0 are K−dimensional mean vectors, and Σ and Σ0 are diagonal covariance matrices. We use the reparameterization trick (Kingma & Welling, 2013) to estimate the gradients, namely z=µ(x)+Σ(x) , where ∼N (0,I). To compute the compressed sentence representations pθ(z|x), as shown in Figure 1, we first feed sentence embeddings fϕ(x) through a shallow MLP. It is then followed by two linear layers, each withK hidden units to compute µ(x) and Σ(x) (after a softplus transform to ensure non-negativity). We also use another linear layer to approximate qφ(y|z).

heading: 3 EXPERIMENTS

text:
Datasets We evaluate the performance on seven different benchmarks for multiple tasks, in particular text classification, natural language inference, similarity, and paraphrase detection. For NLI, we experiment with two well-known NLI benchmarks, namely SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). For text classification, we evaluate on two sentiment analysis datasets, namely IMDB (Maas et al., 2011) and Yelp2013 (YELP) (Zhang et al., 2015). We additionally evaluate on three low-resource datasets in the GLUE benchmark (Wang et al., 2019):3 paraphrase detection using MRPC (Dolan & Brockett, 2005), semantic textual similarity using STS-B (Cer et al., 2017), and textual entailment using RTE (Dagan et al., 2006). For the GLUE benchmark, SNLI, and Yelp, we evaluate on the standard validation and test splits. For MNLI, since the test sets are not available, we tune on the matched dev set and evaluate on the mismatched dev set (MNLI-M) or vice versa. See Appendix A for datasets statistics and Appendix B for hyper-parameters of all methods.
Base Model We use the BERTBase (12 layers, 110M parameters) and BERTLarge (24 layers, 340M parameters) uncased (Devlin et al., 2019) implementation of Wolf et al. (2019) as our base models,4 known to work well for these tasks. We use the default hyper-parameters of BERT, i.e., we use a sequence length of 128, with batch size 32. We use the stable variant of the Adam optimizer (Zhang et al., 2021; Mosbach et al., 2021) with the default learning rate of 2e−5 through all experiments. We do not use warm-up or weight decay.
Baselines We compare against prior regularization techniques, including previous state-of-the-art, Mixout:
1In this work, Z,X, and Y are random variables, and z, x and y are instances of these random variables. 2KL(N (µ0,Σ0)‖N (µ1,Σ1))= 12(tr(Σ −1 1 Σ0)+(µ1−µ0)TΣ −1 1 (µ1−µ0)−K+log( det(Σ1) det(Σ0)
)). 3We did not evaluate on WNLI and CoLA due to the irregularities in these datasets and the reported instability during the fine-tuning https://gluebenchmark.com/faq. 4To have a controlled comparison, all results are computed with this PyTorch implementation, which might slightly differ from the TensorFlow variant (Devlin et al., 2019).
• Dropout (Srivastava et al., 2014), a widely used stochastic regularization techniques used in multiple large-scale language models (Devlin et al., 2019; Yang et al., 2019; Vaswani et al., 2017) to mitigate overfitting. Following Devlin et al. (2019), we apply dropout on all layers of BERT.
• Mixout (Lee et al., 2019) is a stochastic regularization technique inspired by Dropout with the goal of preventing catastrophic forgetting during fine-tuning. Mixout regularizes the learning to minimize the deviation of a fine-tuned model from the pretrained initialization. It replaces the model parameters with the corresponding value from the pretrained model with probability p.
• Weight Decay (WD) is a common regularization technique to improve generalization (Krogh & Hertz, 1992). It regularizes the large weights w by adding a penalization term λ2‖w‖ to the loss, where λ is a hyperparameter specifying the strength of regularization. Chelba & Acero (2004) and Daumé III (2007) adapt WD for fine-tuning of the pretrained models, and propose to replace this regularization term with λ‖w−w0‖, where w0 are the weights of the pretrained models. Recently, Lee et al. (2019) demonstrated that the latter formulation of WD works better for finetuning of BERT than conventional WD and can improve generalization on small training sets.

heading: 3.1 RESULTS ON THE GLUE BENCHMARK

text:
Table 1 shows results on the low-resource datasets in GLUE.5 We find that a) Our VIBERT model substantially outperforms the baselines on all the datasets, demonstrating the effectiveness of the proposed method. b) Dropout decreases the performance on low-resource datasets. We conjecture that regularization techniques relying on stochasticity without considering the relevance to the output, in contrast to VIB, can make it more difficult for learning to extract relevant information from a small amount of data. Igl et al. (2019) observe similar effects in another application. c) Similar to the results of Zhang et al. (2021), we find less pronounced benefits of the previously suggested methods than the results originally published. This can be explained by using a more stable version of Adam (Zhang et al., 2021) suggested by the very recent work in our experiments, which decreases the added benefits of previously suggested regularization techniques on top of a stable optimizer. In contrast, our VIBERT model still substantially improves the results and surpasses the prior work in all settings for both BERTBase and BERTLarge models. Due to the computational overhead of BERTLarge, for the rest of this work, we stick to BERTBase.
Impact of Random Seeds: Following Dodge et al. (2020), we examine the choice of random seed and evaluate the performance of VIBERT and BERT by fine-tuning them across 50 random seeds on GLUE. To comply with the limited access to the GLUE benchmark online system, we split the original validation sets into half and consider one half as the validation set and use the other half as the test set. We first perform model selection on the validation set to fix the hyper-parameters and then fine-tune the selected models for 50 different seeds. Figure 2 shows the expected test performance (Dodge et al., 2019) as the function
5Note that the test sets are not publicly available and the prior work reports the results on the validation set of the GLUE benchmark (Lee et al., 2019; Dodge et al., 2020). We, however, report the results of their methods and ours on the original test sets by submitting to an online system.
of random trials. The results demonstrate that our VIBERT model consistently obtains better performance than BERT on all datasets. As anticipated, the expected test performance monotonically increases with more random trials (Dodge et al., 2020) till it reaches a plateau, such as after 30 trials on STS-B.

heading: 3.2 VARYING-RESOURCE RESULTS

text: To analyze the performance of our method as a function of dataset size, we use four large-resource NLI and sentiment analysis datasets, namely SNLI, MNLI, IMDB, and YELP to be able to subsample the training data with varying sizes. Table 2 shows the obtained results. VIBERT consistently outperforms all the baselines on low-resource scenarios, but the advantages are reduced or eliminated as we approach a medium-resource scenario. Also, the improvements are generally larger when the datasets are smaller, showing that our method successfully addresses low-resource scenarios.

heading: 3.3 OUT-OF-DOMAIN GENERALIZATION

text:
Besides improving fine-tuning on low-resource data by removing irrelevant features, we expect VIB to improve on out-of-domain data because it removes redundant features. In particular, annotation artifacts create shortcut features, which are superficial cues correlated with a label (Gururangan et al., 2018; Poliak et al., 2018) that do not generalize well to out-of-domain datasets (Belinkov et al., 2019a). Since solving the real underlying task can be done without these superficial shortcuts, they must be redundant with the deep semantic features that are truly needed. We hypothesize that many more superficial shortcut features are needed to reach the same level of performance as a few deep semantic features. If so, then VIB should prefer to keep the concise deep features and remove the abundant superficial features, thus encouraging the classifier to rely on the deep semantic features, and therefore resulting in better generalization to outof-domain data. To evaluate out-of-domain generalization, we take NLI models trained on medium-sized 6K subsampled SNLI and MNLI in Section 3.2 and evaluate their generalization on several NLI datasets.
Datasets: We consider a total of 15 different NLI datasets used in Mahabadi et al. (2020), including SICK (Marelli et al., 2014), ADD1 (Pavlick & Callison-Burch, 2016), JOCI (Zhang et al., 2017), MPE (Lai et al., 2017), MNLI, SNLI, SciTail (Khot et al., 2018), and three datasets from White et al. (2017) namely DPR (Rahman & Ng, 2012), FN+ (Pavlick et al., 2015), SPR (Reisinger et al., 2015), and Quora Question Pairs (QQP) interpreted as an NLI task as by Gong et al. (2017). We use the same split used in Wang et al. (2017). We also consider SNLI hard and MNLI(-M) Hard sets (Gururangan et al., 2018), a subset of SNLI/MNLI(-M) where a hypothesis-only model cannot correctly predict the labels and the known biases are avoided. Since the target datasets have different label spaces, during the evaluation, we map predictions to each target dataset’s space (Appendix C). Following prior work (Belinkov et al., 2019a; Mahabadi et al., 2020), we select hyper-parameters based on the development set of each target dataset and report the results on the test set.
Results: Table 3 shows the results of VIBERT and BERT. We additionally include WD, the baseline that performed the best on average on SNLI and MNLI in Table 2. On models trained on SNLI, VIBERT improves the transfer on 13 out of 15 datasets, obtaining a substantial average improvement of 5.51 points. The amount of improvement on different datasets varies, with the largest improvement on SPR and SciTail with +15.5, and +12.5 points respectively, while WD on average obtains only 0.99 points improvement. On models trained on MNLI, VIBERT improves the transfer on 13 datasets, obtaining an average improvement of 3.83 points. The improvement varies across the datasets, with the largest on ADD1 and JOCI with 16.8 and 8.3 points respectively, substantially surpassing WD. Interestingly, VIBERT improves the results on the SNLI and MNLI(-M) hard sets, resulting in models that are more robust to known biases. These results support our claim that VIBERT motivates learning more general features, rather than redundant superficial features, leading to an improved generalization to datasets without these superficial biases. In the next section, we analyze this phenomenon more.

heading: 4 ANALYSIS

text:
Analysis of the Removed Features Elazar & Goldberg (2018) propose a challenging framework to evaluate if debiasing methods have succeeded in removing biases from the sentence representation. After debiasing, the trained encoder is frozen and the classifier is retrained to try to extract the biases. If the classifier reaches high accuracy given only bias features, then the encoder’s representation has not been successfully debiased. We follow the framework of Elazar & Goldberg (2018) to analyze whether known biases in NLI data have been removed in the trained sentence representations. In particular, following Belinkov et al. (2019b), we train a classifier which only sees the representation of the hypothesis sentence and see if it can predict the class of the sentence pair, which is an established criterion to measure known biases in NLI datasets (Gururangan et al., 2018). Thus, we freeze the trained encoders from our model and the BERT baseline and retrain a hypothesis-only classifier on hypotheses from the SNLI and MNLI datasets.6 For reference, we compare to a hypothesis-only model with a BERT encoder trained end-to-end. Table 4 shows the results. With the baseline (BERT), the retrained classifier is not able to recapture all the biases (H-only), but it captures much more than with our method (VIBERT). VIBERT is so successful at reducing biases that performance of the hypothesis-only classifier is close to chance (33%).
Impact of VIB on Overfitting To analyze the effect of VIB on reducing overfitting, we analyze the effect of the β parameter on training and validation error since β controls the trade-off between removing information from the sentence embedding (high β) and keeping information that is predictive of the output (low β). We fix the bottleneck size (K) based on the models selected in Section 3.1, and we train VIBERT on the GLUE benchmark for varying values of β and plot the validation and training loss in Figure 3.
For small values of β, where VIB has little effect, the validation loss is substantially higher than the training loss, indicating overfitting. This is because the network learns to be more deterministic (Σ≈0), thereby retaining too much irrelevant information. As we increaseβ, where VIB has an effect, we observe better generalization performance with less overfitting. As β becomes too large, both the training and validation losses shoot up because the amount of preserved information is insufficient to differentiate between the classes. This pattern is observable in the MRPC and RTE datasets, with a similar pattern in the STS-B dataset.
Efficiency Evaluation Table 5 presents the efficiency evaluation in terms of memory, number of parameters, and time for all the methods measured on RTE. Our approach has several attractive properties. First, while our method is slightly larger in terms of parameters compared to the other standard regularization approaches due to an additional MLP layer (Figure 1), the difference is still marginal, and for BERTBase model with 109.48M trainable parameters, that is less than 1.22% more parameters. Second,
6Note that with VIBERT, the frozen encoder pθ(z|x) outputs a distribution, and the hypothesis-only classifier is trained on samples from this distribution.
our approach presents a much better memory usage with low-overhead, close to Dropout, while WD and especially Mixout cause substantial memory overhead. In dealing with large-scale transformer models like BERT, efficient memory usage is of paramount importance. Third, in terms of training time, our method is similar to Dropout and much faster than the other two baselines. Relative to BERT, VIBERT increases the training time by 3.77%, while WD and Mixout cause the substantial training overhead of 9.11% and 14.44%. Note that our method and other baselines require hyper-parameter tuning.
Ablation Study As an ablation, Table 6 shows results for our model without the compression loss (VIBERT (β=0)), in which case there is no incentive to introduce noise, and the VIB layer reduces to deterministic dimensionality reduction with an MLP. We optimize the dimensionality of the MLP layer (K) as a hyper-parameter for both methods. This ablation does reduce performance on all considered datasets, demonstrating the added benefit of the compression loss of VIBERT.

heading: 5 RELATED WORK

text:
Low-resource Setting Recently, developing methods for low-resource NLP has gained attention (Cherry et al., 2019). Prior work has investigated improving on low-resource datasets by injecting large unlabeled in-domain data and pretraining a unigram document model using a variational autoencoder and use its internal representations as features for downstream tasks (Gururangan et al., 2019). Other approaches propose injecting a million-scale previously collected phrasal paraphrase relations (Arase & Tsujii, 2019) and data augmentation for translation task (Fadaee et al., 2017). Due to relying on the additional source and in-domain corpus, such techniques are not directly comparable to our model.
Information Bottleneck IB has recently been adopted in NLP in applications such as parsing (Li & Eisner, 2019), and summarization (West et al., 2019). Voita et al. (2019) use the mutual information to study how token representations evolve across layers of a Transformer model (Vaswani et al., 2017). This paper – to the best of our knowledge – is the first attempt to study VIB as a regularization technique to improve the fine-tuning of large-scale language models on low-resource scenarios.
Regularization Techniques for Fine-tuning Language models In addition to references given throughout, Phang et al. (2018) proposed to perform an extra data-rich intermediate supervised task pretraining followed by fine-tuning on the target task. They showed that their method leads to improved fine-tuning performance on the GLUE benchmark. However, their method requires pretraining with a large intermediate task. In contrast, our goal is to use only the provided low-resource target datasets.

heading: 6 CONCLUSION AND FUTURE DIRECTIONS

text: We propose VIBERT, an effective model to reduce overfitting when fine-tuning large-scale pretrained language models on low-resource datasets. By leveraging a VIB objective, VIBERT finds the simplest sentence embedding, predictive of the target labels, while removing task-irrelevant and redundant information. Our approach is model agnostic, simple to implement, and highly effective. Extensive experiments and analyses show that our method substantially improves transfer performance in low-resource scenarios. We demonstrate our obtained sentence embeddings are robust to biases and our model results in a substantially better generalization to out-of-domain NLI datasets. Future work includes exploring incorporating VIB on multiple layers of pretrained language models and using it to jointly learn relevant features and relevant layers.

heading: ACKNOWLEDGEMENTS

text: We would like to thank Maksym Andriushchenko for his helpful comments. Rabeeh Karimi was supported by the Swiss National Science Foundation under the project Learning Representations of Abstraction for Opinion Summarisation (LAOS), grant number “FNS-30216”. Yonatan Belinkov was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20).

heading: A EXPERIMENTAL DETAILS

text:
Datasets Statistics Table 7 shows the statistics of the datasets used in our experiments.
Computing Infrastructure We run all experiments on one GTX1080Ti GPU with 11 GB of RAM.
VIBERT Architecture The MLP module used to compute the compressed sentence representations (Figure 1) is a shallow MLP with 768, 2304+K4 , 768+K 2 hidden units with a ReLU non-linearity, where K is the bottleneck size. Following Alemi et al. (2017), we average over 5 posterior samples, i.e., we compute p(y|x)= 15Σ 5 i=1qφ(y|zi), where zi∼pθ(z|x). Similar to Bowman et al. (2016), we use a linear annealing schedule for β and set it as min(1, epoch×β0) in each epoch, where β0 is the initial value.

heading: B HYPER-PARAMETERS

text:
The GLUE Benchmark Experiment Results on GLUE benchmark are reported in Table 1. We fine-tune all the models for 6 epochs to allow them to converge. We use early stopping for all models by choosing the model performing the best on the validation set with the evaluation criterion of average F1 and accuracy for MRPC, accuracy for RTE, and average Pearson and Spearman correlations for STS-B. For VIBERT, we sweep β over {10−4, 10−5, 10−6} and K over {144, 192, 288, 384}. For dropout, we use dropping probabilities of {0.25, 0.45, 0.65, 0.85}. For Mixout, we consider mixout probability of {0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9}. For WD, we consider weight decay of {10−6,10−5,10−4,10−3,10−2,10−1,1}.
Varying-resource Experiment Results on varying sizes of training data are reported in Table 2. We fine-tune all models for 25 epochs to allow them to converge. We use early stopping for all models based on the performance on the validation set. We also perform hyper-parameter tuning on the validation set. Since we consider datasets of a different number of training samples, we need to account for a suitable range of bottleneck size and we sweep K over {12,18,24,36,48,72,96,144,192,288,384} and β over {10−4,10−5}. For dropout, we consider dropping probabilities of {0.25,0.45,0.65,0.85}. For Mixout, we consider mixout probability of {0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9}. For WD, we consider weight decay of {10−6,10−5,10−4,10−3,10−2,10−1,1}.
Ablation Experiment Ablation results are shown in Table 6. For VIBERT (β=0), we sweep K over the same range of values as VIBERT, i.e., {144,192,288,384}

heading: C MAPPING

text:
We train all models on SNLI or MNLI datasets and evaluate their performance on other target datasets. The SNLI and MNLI datasets contain three labels of contradiction, neutral, and entailment. However,
some of the considered target datasets have only two labels, such as DPR or SciTail. When the target dataset has two labels of entailed and not-entailed, as in DPR, we consider the predicted contradiction and neutral labels as the not-entailed label. In the case the target dataset has two labels of entailment and neutral, as in SciTail, we consider the predicted contradiction label as neutral.