summaries: This paper proposes to improve upon unsupervised representation learning for various downstream vision tasks by leveraging human motion and attention (gaze) information. The authors collect a large spatio-temporal dataset with gaze and body motion labels for this task. They train a network to jointly predict the visual focus of attention in scenes and body motion besides visual instance recognition via an NCE loss to learn good visual representations. They show large improvements in accuracy of many different visual recognition downstream tasks with their approach versus the SOTA MOCO approach, which uses visual information only.

summaries: The main aim of the paper is to make use of human interaction/motion to learn a visual  representation that can be re-used for classic visual tasks such as depth estimation. The authors claim that by encoding interaction and attention cues in the self-supervised representation, the method can outperform visual-only state-of-the-art methods. To study the interaction element, the authors attach sensors like Inertial Movement Units (IMUs) to the limbs of subjects and monitor their reaction to visual events in daily life. The paper also introduces a new dataset of 4260 minutes of human interactions by 35 participants which include synchronized streams of images, body part movements, and gaze information.

summaries: The paper uses a combination of visual, human gaze and human motion sensors to build representations that perform better on downstream tasks such as action recognition, physics prediction and depth estimation than representations extracted from solely visual input. The paper announces the release a new data set of aligned visual images, eye gaze fixations and IMU motion readings from test subjects walking around an environment. Representations are computed using three different forms of information simultaneously. Given a visual input, the system tries to predict the location of eye gaze in image frame coordinates, whether each of 6 groups of motion detectors are active or not (head, torso, legs, etc.) and the result of a more traditional auxiliary visual pretext task. In this work, the paper uses “instance discrimination” where representations of augmented versions of a specific image are pushed close together in latent space and far away from augmentations of other images. Tests on diverse benchmarks show that the gaze and motion prediction improve over visual pretext tasks alone and that there is a small benefit to using both together, but it is not additive. The paper also shows the benefit of gaze and motion is present for two different visual auxiliary tasks.

abstractText: Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely on visual data such as images or videos. In this paper, we explore a novel approach, where we use human interaction and attention cues to investigate whether we can learn better representations compared to visualonly representations. For this study, we collect a dataset of human interactions capturing body part movements and gaze in their daily lives. Our experiments show that our “muscly-supervised” representation that encodes interaction and attention cues outperforms a visual-only state-of-the-art method MoCo (He et al., 2020), on a variety of target tasks: scene classification (semantic), action recognition (temporal), depth estimation (geometric), dynamics prediction (physics) and walkable surface estimation (affordance). Our code and dataset are available at: https://github.com/ehsanik/muscleTorch. Figure 1: We propose to use human’s interactions with their visual surrounding as a training signal for representation learning. We record first person observations as well as the movements and gaze of people living their daily routines and use these cues to learn a visual embedding. We use the learned representation on a variety of diverse tasks and show consistent improvements compared to state-of-the-art self-supervised vision-only techniques.

authors:
Pulkit Agrawal
Joao Carreira
Jitendra Malik

title: Learning to see by moving

venue: In ICCV,

authors:
Philip Bachman
R Devon Hjelm
William Buchwalter

title: Learning representations by maximizing mutual information across views

venue: In NeurIPS,

title: Social learning theory

venue: Prentice-hall,

authors:
Liang-Chieh Chen
George Papandreou
Iasonas Kokkinos
Kevin Murphy
Alan L. Yuille

title: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations. arXiv, 2020a

authors:
Ting Chen
Simon Kornblith
Kevin Swersky
Mohammad Norouzi
Geoffrey Hinton

title: Big selfsupervised models are strong semi-supervised learners

venue: arXiv preprint arXiv:2006.10029,

authors:
Xinlei Chen
Haoqi Fan
Ross Girshick
Kaiming He

title: Improved baselines with momentum contrastive learning. arXiv, 2020c

authors:
Dima Damen
Hazel Doughty
Giovanni Maria Farinella
Sanja Fidler
Antonino Furnari
Evangelos Kazakos
Davide Moltisanti
Jonathan Munro
Toby Perrett
Will Price
Michael Wray

title: Scaling egocentric vision: The epic-kitchens dataset

authors:
Dima Damen
Will Price
Evangelos Kazakos
Antonino Furnari
Giovanni Maria Farinella

title: Epic-kitchens - 2019 challenges report

venue: Technical report,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Fei-Fei Li

title: Imagenet: A large-scale hierarchical image database

venue: In CVPR,

authors:
R Devon Hjelm
Philip Bachman

title: Representation learning with video deep infomax

authors:
Carl Doersch
Abhinav Gupta
Alexei A. Efros

title: Unsupervised visual representation learning by context prediction

authors:
Jeff Donahue
Philipp Krähenbühl
Trevor Darrell

title: Adversarial feature learning

venue: In ICLR,

authors:
Kiana Ehsani
Hessam Bagherinezhad
Joseph Redmon
Roozbeh Mottaghi
Ali Farhadi

title: Who let the dogs out? modeling dog behavior from visual data

authors:
Alireza Fathi
Yin Li
James M Rehg

title: Learning to recognize daily actions using gaze

venue: In ECCV,

authors:
Spyros Gidaris
Praveer Singh
Nikos Komodakis

title: Unsupervised representation learning by predicting image rotations

authors:
Ross Girshick
Jeff Donahue
Trevor Darrell
Jitendra Malik

title: Rich feature hierarchies for accurate object detection and semantic segmentation

authors:
Daniel Gordon
Kiana Ehsani
Dieter Fox
Ali Farhadi

title: Watching the world go by: Representation learning from unlabeled videos

authors:
Priya Goyal
Dhruv Mahajan
Abhinav Gupta
Ishan Misra

title: Scaling and benchmarking selfsupervised visual representation learning

authors:
Michael Gutmann
Aapo Hyvärinen

title: Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

venue: In AISTATS,

authors:
Raia Hadsell
Sumit Chopra
Yann LeCun

title: Dimensionality reduction by learning an invariant mapping

authors:
Mohamed Hassan
Vasileios Choutas
Dimitrios Tzionas
Michael J. Black

title: Resolving 3d human pose ambiguities with 3d scene constraints

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross Girshick

title: Momentum contrast for unsupervised visual representation learning

authors:
Olivier J Hénaff
Aravind Srinivas
Jeffrey De Fauw
Ali Razavi
Carl Doersch
SM Eslami
Aaron van den Oord

title: Data-efficient image recognition with contrastive predictive coding

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

authors:
Drew A. Hudson
Christopher D. Manning

title: Gqa: A new dataset for real-world visual reasoning and compositional question answering

authors:
Catalin Ionescu
Dragos Papava
V. Olaru
C. Sminchisescu

title: Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments

authors:
Dinesh Jayaraman
Kristen Grauman

title: Learning image representations tied to ego-motion

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Ivan Krasin
Tom Duerig
Neil Alldrin
Vittorio Ferrari
Sami Abu-El-Haija
Alina Kuznetsova
Hassan Rom
Jasper Uijlings
Stefan Popov
Andreas Veit
Serge Belongie
Victor Gomes
Abhinav Gupta
Chen Sun
Gal Chechik
David Cai
Zheyun Feng
Dhyanesh Narayanan
Kevin Murphy

title: Openimages: A public dataset for large-scale multi-label and multi-class image classification

venue: Dataset available from https://github.com/openimages,

authors:
Tsung-Yi Lin
Piotr Dollár
Ross Girshick
Kaiming He
Bharath Hariharan
Serge Belongie

title: Feature pyramid networks for object detection

title: Distinctive image features from scale-invariant keypoints

venue: International journal of computer vision,

authors:
Dhruv Mahajan
Ross Girshick
Vignesh Ramanathan
Kaiming He
Manohar Paluri
Yixuan Li
Ashwin Bharambe
Laurens van der Maaten

title: Exploring the limits of weakly supervised pretraining

authors:
Ishan Misra
Laurens van der Maaten

title: Self-supervised learning of pretext-invariant representations

authors:
Roozbeh Mottaghi
Hessam Bagherinezhad
Mohammad Rastegari
Ali Farhadi

title: Newtonian scene understanding: Unfolding the dynamics of objects in static images

authors:
Roozbeh Mottaghi
Hannaneh Hajishirzi
Ali Farhadi

title: A task-oriented approach for costsensitive recognition

authors:
Pushmeet Kohli Nathan Silberman
Derek Hoiem
Rob Fergus

title: Indoor segmentation and support inference from rgbd images

authors:
Mehdi Noroozi
Paolo Favaro

title: Unsupervised learning of visual representations by solving jigsaw puzzles

authors:
Mehdi Noroozi
Hamed Pirsiavash
Paolo Favaro

title: Representation learning by learning to count

authors:
Aaron van den Oord
Yazhe Li
Oriol Vinyals

title: Representation learning with contrastive predictive coding

authors:
Deepak Pathak
Philipp Krähenbühl
Jeff Donahue
Trevor Darrell
Alexei Efros

title: Context encoders: Feature learning by inpainting

authors:
Deepak Pathak
Ross Girshick
Piotr Dollár
Trevor Darrell
Bharath Hariharan

title: Learning features by watching objects move

authors:
Lerrel Pinto
Dhiraj Gandhi
Yuanfeng Han
Yong-Lae Park
Abhinav Gupta

title: The curious robot: Learning visual representations via physical interactions

authors:
Shaoqing Ren
Kaiming He
Ross Girshick
Jian Sun

title: Faster R-CNN: Towards real-time object detection with region proposal networks

venue: NeurIPS,

authors:
Evan Shelhamer
Jonathan Long
Trevor Darrell

title: Fully convolutional networks for semantic segmentation

authors:
Wenzhe Shi
Jose Caballero
Ferenc Huszár
Johannes Totz
Andrew P Aitken
Rob Bishop
Daniel Rueckert
Zehan Wang

title: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

authors:
Gunnar A Sigurdsson
Abhinav Gupta
Cordelia Schmid
Ali Farhadi
Karteek Alahari

title: Charades-ego: A large-scale dataset of paired third and first person

venue: videos. arXiv,

authors:
Carl Vondrick
Hamed Pirsiavash
Antonio Torralba

title: Anticipating visual representations from unlabeled video

authors:
Xiaolong Wang
Abhinav Gupta

title: Unsupervised learning of visual representations using videos

authors:
Luca Weihs
Aniruddha Kembhavi
Winson Han
Alvaro Herrasti
Eric Kolve
Dustin Schwenk
Roozbeh Mottaghi
Ali Farhadi

title: Artificial agents learn flexible visual representations by playing a hiding

venue: game. arXiv,

authors:
Zhirong Wu
Yuanjun Xiong
Stella X Yu
Dahua Lin

title: Unsupervised feature learning via nonparametric instance discrimination

authors:
Jianxiong Xiao
James Hays
Krista A Ehinger
Aude Oliva
Antonio Torralba

title: Sun database: Large-scale scene recognition from abbey to zoo

authors:
Yanyu Xu
Yanbing Dong
Junru Wu
Zhengzhong Sun
Zhiru Shi
Jingyi Yu
Shenghua Gao

title: Gaze prediction in dynamic 360◦ immersive videos

authors:
Ting Yao
Yiheng Zhang
Zhaofan Qiu
Yingwei Pan
Tao Mei. Seco

title: Exploring sequence supervision for unsupervised representation learning

authors:
Amir R. Zamir
Alexander Sax
William Shen
Leonidas J. Guibas
Jitendra Malik
Silvio Savarese

title: Taskonomy: Disentangling task transfer learning

authors:
Richard Zhang
Phillip Isola
Alexei A Efros

title: Colorful image colorization

authors:
Bolei Zhou
Agata Lapedriza
Aditya Khosla
Aude Oliva
Antonio Torralba

title: Places: A 10 million image database for scene recognition

authors:
Chengxu Zhuang
Alex Lin Zhai
Daniel Yamins

title: Local aggregation for unsupervised learning of visual embeddings

heading: 1 INTRODUCTION

text:
Encoding visual information from pixel space to a lower-dimensional vector is the core element of most modern deep learning-based solutions to computer vision. A rich set of algorithms and architectures have been developed to enable learning these encodings. A common practice in computer vision is to explicitly train the networks to map visual inputs to a curated label space. For example, a neural network is pre-trained using a large-scale annotated classification dataset (Deng et al., 2009; Krasin et al., 2017) and the entire network or part of it is fine-tuned to a new target task (Goyal et al., 2019; Zamir et al., 2018).
In recent years, weakly supervised and self-supervised representation learning approaches (e.g., Mahajan et al. (2018); He et al. (2020); Chen et al. (2020a)) have been proposed to mitigate the need for supervision. The most successful ones are contrastive learning-based approaches such as (Chen et al., 2020c;b) and they have shown remarkable results on target tasks such as image
classification and object detection. Despite their success, there are two primary caveats: (1) These self-supervised methods are still trained on ImageNet or similar datasets, which are fairly cleaned up and/or include a pre-specified set of object categories. (2) This method of training is a passive approach in that it does not encode interactions. On the contrary, for humans, a vast majority of our visual understanding is shaped by our interactions and our observations of others interacting with their environments. We are not limited to learning from visual cues alone, and there are various other supervisory signals such as body movements and attention cues available to us. It is shown that by learning how to move the joints to walk and crawl, infants can significantly enhance their perception and cognition (Adolph & Robinson, 2015). Moreover, by observing another person interact with the environment humans obtain a visual and physical perception of the world (Bandura, 1977).
The question we investigate in this paper is, “can we learn a rich generalizable visual representation by encoding human interactions into our visual features?”. In this work, we consider the movement of human body parts and the center of attention (gaze) as an indicator of their interactions with the environment and propose an approach for incorporating interaction information into the musclysupervisedrepresentation learning process.
To study what we can learn from interaction, we attach sensors to humans’ limbs and see how they react to visual events in their daily lives. More specifically, we record the movements of the body parts by Inertial Movement Units (IMUs) and also the gaze to monitor the center of attention. We introduce a new dataset of more than 4,500 minutes of interaction by 35 participants engaging in everyday scenarios with their corresponding body part movements and center of attention. There are no constraints on the actions, and no manual annotations or labels are provided.
Our experiments show that the representation we learn by predicting gaze and body movements in addition to the visual cues outperforms the visual-only baseline on a diverse set of target tasks (Figure 1): semantic (scene classification), temporal (action recognition), geometric (depth estimation), physics (dynamics prediction) and affordance-based (walkable surface estimation). This shows that movement and gaze information can help to learn a more informative representation compared to a visual-only model.

heading: 2 RELATED WORK

text:
Visual representations can be learned using many different techniques from full supervision to no supervision at all. We outline the most common paradigms of representation learning, namely supervised, self-supervised, and interaction-based representation learning.
Supervised Representation Learning. Supervised representation learning in computer vision is typically performed by pre-training neural networks on large-scale datasets with full supervision (e.g., ImageNet (Deng et al., 2009)) or weak supervision (e.g., Instagram-1B (Mahajan et al., 2018)). These models are fine-tuned for a variety of tasks including object detection (Girshick et al., 2014; Ren et al., 2015), semantic segmentation (Shelhamer et al., 2015; Chen et al., 2017), and visual question answering (Agrawal et al., 2015a; Hudson & Manning, 2019). However, collecting a manually annotated large-scale dataset such as ImageNet requires extensive resources in terms of cost and time. In contrast, in this paper, we only use human interaction data, which does not require any manual annotation.
Self-supervised Representation Learning. There has been a wide range of research on selfsupervised learning of visual representations in which properties of the images themselves act as supervision. The objectives for these methods cover a variety of tasks such as solving jigsaw puzzles (Noroozi & Favaro, 2016), colorizing grayscale images (Zhang et al., 2016), learning to count (Noroozi et al., 2017), predicting context (Doersch et al., 2015), inpainting (Pathak et al., 2016), adversarial training (Donahue et al., 2017) and predicting image rotations (Gidaris et al., 2018). This type of representation learning is not limited to learning from single frames. Agrawal et al. (2015b) and Jayaraman & Grauman (2015) both use egomotion, Wang & Gupta (2015) cyclically track patches in videos, Pathak et al. (2017) use low-level non-semantic motion-based cues, and Vondrick et al. (2016) predict the representation of future frames.
Inspired by contrastive learning (Hadsell et al., 2006), recent methods have used “instance discrimination” in which the network uniquely identifies each image. A network is trained to produce a non-linear mapping that projects multiple variations of an image closer to each other than to all other
images. Using Noise Contrastive Estimation (Gutmann & Hyvärinen, 2010), networks are trained to differentiate between similar images under complex noise models (such as non-overlapping crops and heavy color jittering) and dissimilar images. Oord et al. (2018) and Hénaff et al. (2019) introduce and investigate the Contrastive Predictive Coding (CPC) method, which encodes the shared information between different crops of an image to predict the features from masked regions of the image. Wu et al. (2018); Misra & van der Maaten (2020) use a memory bank, which enables contrasting features of the current image against a large set of negative samples, increasing the likelihood of finding a nearby negative. The MoCo technique (He et al., 2020; Chen et al., 2020c) encodes the positive samples with a momentum encoder to avoid the rapid changes in the original feature extractor. They achieve comparable results with supervised learning representations. Chen et al. (2020a;b) show that by using a trainable non-linear transformation between the representation and contrastive latent space and larger batch sizes, they can omit memory banks entirely, allowing for full backpropagation through both positive and negative samples, and achieve better results. Bachman et al. (2019); Tian et al. (2019) maximize the mutual information between different extracted features of the same image from multiple views. Zhuang et al. (2019) enforce the extracted features of similar images to move towards the same part of the embedding space. Gordon et al. (2020), Yao et al. (2020), and Devon Hjelm & Bachman (2020) apply contrastive method to videos and leverage spatio-temporal cues to learn visual representations. Similar to these approaches, we do not rely on any manual annotation, but in contrast to them, we utilize human interactions along with their visual observation for representation learning.
Interaction-Based Representation Learning. The third class of learning representations relies on cues obtained by interacting with a dynamic environment. Pinto et al. (2016) learn a representation from interactions of a robotic arm (e.g., grasping and pushing) with different objects. Chen et al. (2019) and Weihs et al. (2019) both tackle the representation learning problem by training an agent to play a game in an interactive environment. Ehsani et al. (2018) learn a representation by modeling the non-semantic movements of a dog. Our work falls in this category since we use human interactions for learning the representation. We differ from these approaches in that we use lowlevel observations of human interaction such as body part movements and gaze to show significant improvement over a state-of-the-art baseline across multiple low-level and high-level target tasks.

heading: 3 HUMAN INTERACTION DATASET

text:
We introduce a new dataset of human interactions for our representation learning framework. In this section, we describe the data collection. Our goal is to capture how humans react to the visual
world by recording their movements and focus of attention. Previous datasets of human actions and gaze include only gaze information (Fathi et al., 2012; Xu et al., 2018), part movements from a third-person view (Ionescu et al., 2014; Hassan et al., 2019), or only action or hand labels in an egocentric setting (Damen et al., 2018; Sigurdsson et al., 2018). In contrast, our new dataset includes ego-centric observations along with the corresponding gaze and body movement information during their daily activities ranging from walking and cycling to driving and shopping.
To collect the dataset, we record egocentric videos from a GoPro camera attached to the subjects’ forehead. We simultaneously capture body movements, as well as the gaze. We use Tobii Pro2 eyetracking to track the center of the gaze in the camera frame. We record the body part movements using BNO055 Inertial Measurement Units (IMUs) in 10 different locations (torso, neck, 2 triceps, 2 forearms, 2 thighs, and 2 legs). Figure 2 shows the data collection setup along with two clips of the captured sequences. In total, we collected 4,260 minutes of videos with their corresponding body part movement and gaze from 35 people. Unlike the common large-scale datasets used for representation learning such as ImageNet, there is no restriction on the categories observed in the images, and no manual annotation is provided. Although the supervision from movement and gaze does not come for free and still requires the participants wearing the sensors, no manual or semantic annotation is needed for acquiring this dataset. Statistical analysis of the dataset is provided in Appendix A.3. Moreover, we provide details of aligning the video with the motion sensors and synchronization of the sensors in Appendix A.2. The supplementary video shows a few examples of the video clips.

heading: 4 INTERACTION-BASED REPRESENTATION LEARNING

text: Visual representation learning is typically performed using visual cues from single images or videos (He et al., 2020; Gordon et al., 2020). Our goal in this paper is to incorporate human interactions into our representations to move beyond a purely visually-trained feature representation. Below, we describe our approach for integrating movement and gaze information in the representation learning pipeline. Intuitively, body part movements should encode the temporal changes in the image based on the underlying cause of those changes (e.g., moving legs results in walking which makes distant objects move closer). Additionally, gaze grounds the visual features with the location in the image where the person pays the most attention. This should correlate well with semantic concepts such as objects, or affordances such as walkable surfaces.

heading: 4.1 LEARNING FEATURES

text:
Our goal is to learn visual representations by simultaneous learning of a visual encoding for each frame and predicting body part movements and gaze attention from the sequence of observations. Formally, given an ego-centric video as a sequence of images V = (It, . . . , It+k), the goal is to 1) estimate the gaze G = (Gt, . . . , Gt+k), where Gt is the person’s center of focus in 2D camera coordinates, and 2) predict the body part movements P = (Pt, . . . , Pt+k), where Pt is a binary vector of length equal to the number of body parts, indicating whether a part is moved at time t.
We optimize three objectives: (1) gaze prediction, (2) body part movement prediction, and (3) auxiliary visual prediction. The visual features obtained from a CNN backbone are combined with a sequence-to-sequence model in order to predict gaze and movement. Note that the weights of the backbone are randomly initialized, i.e. we train the model from scratch. Figure 3 shows an overview of the architecture. The objectives are jointly optimized. In the following, we explain each of them in more detail.
Gaze: We predict the person’s focus of attention by modeling their gaze in the camera reference frame. We use the Huber loss to train the center of attention Lattention(Ĝ,G),
Lattention(Ĝ,G) =  12 ∥∥∥Ĝ−G∥∥∥2 2 |Ĝ−G| < δ
|Ĝ−G| − 12 otherwise (1)
Movement: We find the task of predicting body part movement direction and magnitude to be highly ambiguous. For example, when walking, the visual information may not show the legs, so we cannot know how high the legs were lifted. Instead, for each body part, we predict whether it is moving
at all which is less ambiguous and reduces the problem to a binary classification task. Rather than predicting the movements for lower and upper parts of the joint separately (leg and thigh, forearm and tricep), we combine the movements into 6 categories of torso, neck, right arm, left arm, right leg, and left leg. To estimate this movement, we use binary cross-entropy loss and denote it as Lmovement(P̂ , P ).
Auxiliary Visual Prediction: We also use a visual objective, Lvisual(Ît, It). For this objective, similar to (He et al., 2020), we use instance discrimination. Any alternative visual encoding objective can be used instead. In section 5.3.1, we provide results for another type of visual encoding as well.
Instance discrimination’s objective is to force the visual features of different augmentations of the same image to be as close as possible in the latent space (Wu et al., 2018; Chen et al., 2020a; Zhuang et al., 2019) while pushing apart all other image embeddings. By learning to extract what makes each image unique, the network focuses on semantically meaningful features of the image. This enables the feature extractor to embed a more invariant representation of the image, which is especially important when transferring to different tasks and domains. To contrast the positive samples (the augmentations of the image), with a large set of negative samples, we maintain a memory bank of embedded features from different images in the data. The final objective can be formalized as Lvisual (which is also known as the InfoNCE (Oord et al., 2018) loss),
Lvisual(Ît, It) = − log exp(f(It) · f(Ît)/τ)∑N i=0 exp(f(It) ·Mi/τ) , (2)
where It, Ît are two different random augmentations of the first image of the sequence V , f is the image feature extractor (ResNet backbone), M = (M0, . . . ,MN ) is the bank of negative samples and τ is a parameter that controls the concentration level of the distribution (Hinton et al., 2015). We also use a momentum-updated encoder as in (He et al., 2020). We apply the visual loss only to the first image of the sequence, as images within a sequence tend to be visually similar to each other.
The overall objective is a weighted sum of the described loss functions. More details on the architecture are provided in Appendix A.4.1.
Linteraction = αLattention(Ĝ,G) + βLmovement(P̂ , P ) + γLvisual(Ît, It) (3)

heading: 4.2 ADAPTING THE REPRESENTATION TO NEW TASKS

text:
After training the model using Linteraction objective, we use the trained weights of our feature extraction network (i.e., only the ResNet part) as the initialization for our target tasks. Our goal in this paper is to evaluate the visual representation on its own rather than using it as initialization for end-to-end training. Hence, during training for the target tasks, the weights of the feature extraction backbone are frozen. We have a diverse set of target tasks, where each requires a specific network
architecture (for example, depth estimation requires up-convolutional layers, while action recognition requires a temporal architecture). Below, we describe the result of the transfer to the target tasks. We explain the details of the architectures for each target task in Appendix A.4.2.

heading: 5 EXPERIMENTS

text: To evaluate our representation learning approach, we consider five different types of target tasks. The tasks are chosen such that they cover a wide range of domains: semantic (scene classification), temporal (action recognition), geometric (depth estimation), physical (dynamics prediction), and affordance (walkable surface estimation). We show that our learned representation, which encodes body part movement and gaze and does not rely on any manual annotation, outperforms a strong self-supervised baseline which relies on purely visual cues. Furthermore, we provide ablations of our model by using an alternative visual loss and using a subset of body parts for representation learning. For implementation details, refer to Appendix A.4.

heading: 5.1 SELF-SUPERVISED BASELINE

text: We compare our method with the recently introduced self-supervised representation learning technique, Momentum Contrast network (MoCo) (He et al., 2020), which is a state-of-the-art representation learning approach and achieves strong performance on a variety of target tasks such as image classification and object detection. The original work was trained on images from the ImageNet dataset. To ensure the comparison between our method and the baseline is fair, we train MoCo on the images from our dataset. Note that this baseline relies on visual cues only. Our goal is to show whether we can learn better representations when we use movement and gaze information in addition to the visual information.

heading: 5.2 EVALUATION OF THE LEARNED REPRESENTATION

text:
We evaluate the learned representation on five different target tasks. The weights for feature extraction backbone are frozen, and only the task-specific layers are trained. We show that the representation trained using the movement and attention (gaze) supervision in addition to the visual cues outperforms MoCo (He et al., 2020) baseline (trained on our data) across the board. For each target task, we report the results in four settings, each using a different combination of visual, movement, and gaze (attention) cues for representation learning.
Scene Classification. For the task of scene classification, a network receives a single image as input and predicts the scene category of the image. We use SUN397 (Xiao et al., 2010) dataset for this task, as it provides a large-scale dataset of 130k images of 397 different scene categories (e.g., park, restaurant, kitchen). The results are shown in Table 1-column (a). The representation that encodes both movement and attention cues performs the best on the semantic task of scene classification. We achieve nearly a 7% improvement compared to fine-tuning the MoCo (He et al., 2020) baseline.
Action Recognition. The task is to predict the category of action from ego-centric videos. We use the EPIC-KITCHENS dataset (Damen et al., 2018) for this task, which is a large-scale dataset of 11M images from different action categories that are performed in various kitchens.
As shown in Table 1-column (b), our method outperforms the strong baseline representation learning method by 3.5%. This again shows that incorporating additional cues such as part movements and gaze in the representation learning is beneficial for downstream tasks. It seems that both movement and attention cues are helpful for action recognition. This is aligned with our intuition that predicting the gaze of a person and how they move their body parts may be beneficial to recognizing the actions they perform.
Future Prediction of Dynamics. The goal of this task is to predict the future dynamics of an object in an image. We use the VIND (Mottaghi et al., 2016a) dataset for this task. It includes 150K images with corresponding object bounding boxes. The dataset categorizes physical dynamics into Newtonian scenarios such as sliding, projectile motion, and bouncing. The goal is to predict these Newtonian scenarios and the camera viewpoint for a query object that is specified by a bounding box and physical motion labels. There are 66 classes in total. The input to the network is a single RGB image and the bounding box for the query object. Table 1-column (c) includes the results for this task. We outperform the baseline by 1.3%. The representation that is learned by using both attention and movement provides the best performance for this task, which involves predicting the future trajectory of objects.
Walkable Surface Estimation. The goal of this task is to segment the pixels in an image that a person can walk on. We use the data from (Mottaghi et al., 2016b), which provides annotation for 1449 images of the NYU DepthV2 (Nathan Silberman & Fergus, 2012) dataset. The results are shown in Table 1-column (d). The variation of our method that uses only the gaze information achieves the highest accuracy. This might be due to the fact that, during walking, human attention is focused on the places that they can walk on. Therefore, the gaze provides sufficient information to perform this task.
Depth Estimation. For depth estimation, the task is to regress the values of the depth for a single monocular RGB image. We use NYU DepthV2 (Nathan Silberman & Fergus, 2012) dataset for this task, which provides 1449 densely labeled pairs of RGB and depth images. The results are shown in Table 1-column (e). Our learned representation outperforms the baseline for this task as well. Movement cues seem more aligned with the task of depth estimation, and the representation embedding with this information performs better. Note that the metrics for depth and walkable surface estimation are global metrics i.e. they are computed for the entire image. Therefore, typically a small improvement in those metrics has a significant effect on the qualitative results.

heading: 5.3 ABLATIVE ANALYSES

text: We ablate our results by replacing the InfoNCE visual loss with an autoencoder loss. Additionally, we show how gaze information affects the prediction of the body part movements. Finally, we evaluate which movements serve as an important supervision by masking out subsets of the body parts and retraining the representation.

heading: 5.3.1 VISUAL LOSS

text:
As discussed in Section 4.1, we use the InfoNCE loss (Oord et al., 2018) while learning the representation. In order to investigate the impact of using this objective, we learn the representation using an autoencoder loss for our visual objective and evaluate the learned representation on all five target tasks after re-training using the new backbone. The autoencoder loss, Lae is defined as Lae(d(f(It)), It) = ‖d(f(It))− It‖2, where f is the feature extractor backbone and d is a decoder network of five up-convolution layers, which receives the 512 × 7 × 7 feature as input and reconstructs a 3× 56× 56 image.
Table 2 shows that gaze and movement information still provide a strong signal compared to the visual-only case. However, the results are worse than the case that we use the InfoNCE loss for learning the representation.

heading: 5.3.2 MOVEMENT ESTIMATION

text:
To better understand the effect of gaze, we predict body part movements with and without gaze information. This experiment is not part of the representation learning experiments. It is just to evaluate whether using gaze provides any additional cue for prediction of the movements.
In this experiment, we predict which subset of the six groups of body parts (neck, torso, left arm, right arm, left leg, right leg) have moved. The overall architecture for this experiment is the same as our representation learning model, except for the inputs to the LSTM modules, which is instead the concatenated image features from ResNet and the embedded input gaze. The input gaze embedding is a two-layer network encoding the gaze into a feature vector of size 512.
Table 3 shows the results for this experiment. The network achieves an improvement in predicting the body parts movements by having the additional information of the person’s center of attention, which can intuitively serve as a proper indicator of their “intentions”.

heading: 5.3.3 EFFECTS OF THE BODY PARTS

text: To evaluate how each body part affects the learned representation, we perform an experiment where we ignore a subset of body parts, re-train the representation learning (from scratch) and evaluate the features on target tasks. Table 4 summarizes the results. The performance on the target tasks (except depth estimation) drops when we ignore a body part during representation learning. For depth estimation, removing torso results in a slightly lower error, which might indicate that the torso movement is not as helpful for estimating the depth.

heading: 6 CONCLUSION

text:
Representations that encode movements and actions become a necessity as we move deeper towards embodied visual understanding. In this paper, we investigate the idea of using human interactions to learn visual representations. To enable this research, we introduce a new dataset of human interactions which includes hours of synchronized streams of image frames, body part movements, and gaze information across different subjects and activities. We show that representations trained to predict body movements and gaze encode additional information compared to their purely visual counterparts. More specifically, we show our representation outperforms a state-of-the-art selfsupervised representation learning baseline for a variety of target tasks. Our main point in this paper
is to show there are signals other than visual information that can be used for improving visual representation. However, we require additional sensors to capture data from other modalities and we acknowledge that scalability of our dataset requires additional data collection.

heading: ACKNOWLEDGMENTS

text: This work is in part supported by NSF IIS 1652052, IIS 17303166, DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543 and gifts from Allen Institute for Artificial Intelligence. We thank Winson Han for helping us with making the figures for this paper.

heading: APPENDIX

heading: A.1 DATASET EXAMPLES

text: The supplementary video provides examples of our dataset and also qualitative results of our representation learning model.

heading: A.2 DATA COLLECTION DETAILS

text: We describe the details of our hardware setup and the alignment method used to synchronize the recordings between our camera, gaze tracker, and movement sensors.

heading: A.2.1 ALIGNMENT AND SYNCHRONIZATION OF DEVICES

text:
There are three different devices in our setup, 1) Tobii Pro eye-tracking glasses to record gaze, 2) BNO055 IMU sensors to record movements, and 3) GoPro camera attached to the forehead to capture ego-centric videos. These different devices record data independently. Therefore, it is necessary to synchronize all recordings.
Gaze Tracker and GoPro Alignment. Tobii Pro2 eye-tracking captures a video and the center of the gaze in the camera frame. Due to the low quality of the video captured by the eye-tracking glasses, we use an additional high-quality GoPro hero 6 camera (with resolution 1920× 1080 and 60 frames per second). To synchronize the videos from the gaze tracker and GoPro, we extract SIFT (Lowe, 2004) features and use a brute force algorithm for feature matching and the RANSAC method to find a homography that maps the gaze from Tobii’s camera frame to GoPro’s camera coordinate. Note that the gaze might be missing for some frames due to the device noise.
IMU Sensors and GoPro Synchronization. The outputs of the IMU sensors are recorded on a Raspberry Pi board. There is also a microphone on the Raspberry Pi that records the audio. We synchronize the IMU and video recordings using two methods, 1) synchronize the audio from the GoPro video and the voice recording on the Raspberry Pi board, and 2) repeat a specific pattern of body movements in front of a mirror, so it can be uniquely identified in both the movements depicted by the IMU sensors and the GoPro camera (which recorded the participant’s body pose in the mirror).

heading: A.2.2 MOVEMENT CALCULATIONS

text:
We record the body part movements using BNO055 Inertial Measurement Units (IMUs) in 10 different locations (torso, neck, 2 triceps, 2 forearms, 2 thighs, and 2 legs). The body parts may not appear in ego-centric video frames, therefore, the task of predicting the exact orientation and location of a body part (e.g., arm) using ego-centric videos can be very challenging. We train the model using the simpler task of predicting whether a part has moved or not. This still contains rich information about the action that is happening in the video, for example, walking can be defined as periodic movements of the left and right legs.
To compute the loss function, we need to distinguish between movement and no movement in the dataset. One way is finding a threshold in the domain of the angles of the part movements, and label all the moves smaller than the threshold as no movement and the rest as movement. However, this might result in ambiguities for the movements close to the threshold, and the network might over penalize the wrong predictions in the neighborhood close to the threshold. Therefore, we add a third gray area label, where the network is not penalized for wrong predictions. We divide the range of movements for each sensor into three equal ranges, the first 33% is labeled as no movement, the last 33% is labeled as movement and the remaining interval is the gray area, for which there is no penalty for misprediction.

heading: A.3 DATASET ANALYSIS

text:
To ensure that the videos in the dataset consist of a wide range of activities, we do not provide any specific instructions to the subjects, and we ask them to perform their daily routine activities. Hence, the dataset includes a variety of different situations including but not limited to driving,
cycling, playing pool, cooking, cleaning, walking in the streets, and shoveling snow. Purely for dataset analysis purposes, we gather proxy scene labels for each image. We use an off-the-shelf scene classification model trained on the Places dataset (Zhou et al., 2017) and record the top-1 prediction for each frame of our dataset. Many scene categories in our data are not present in Places, so we see a moderate amount of misclassification. However, the classifier confidently (more than 70%) predicts 101 of the 365 classes exist somewhere within our dataset, showing the diversity of our data. Figure 4 Left shows the 20 most frequent predictions. Even though there are some mispredictions among them (such as jail cell which is frequently mistaken for dark rooms), we observe that our data is fairly diverse.
Figure 4 Middle shows the distribution of the gaze in the images. As expected, the focus is mostly in the center of the image. In Figure 4 Right, we show the change in the orientation of the body parts between two consecutive frames. We observe more movements in the limbs compared to the torso and neck. We additionally notice more right arm movement than the left which is likely caused by more of our participants being right-handed.

heading: A.4 ARCHITECTURE & TRAINING DETAILS

text: In this section, we describe the details of our network architectures as well as the hyperparameters and optimization methods that we used, for reproducibility purposes. Also, the code and data will be made publicly available for further research.

heading: A.4.1 BACKBONE NETWORK

text: We train the feature extractor network by using a sequence of images of length k = 5, which are 1 6 th of a second apart, as input. We use the ResNet18 (He et al., 2016) convolution layers as the feature extraction backbone. To preserve spatial information, which is essential for gaze prediction, we use the 512 × 7 × 7 features before average pooling. We then add a 1 × 1 convolutional layer on top to reduce the feature size to 64 × 7 × 7. The flattened feature is then input to a 3 layer LSTM with hidden size 512, which encodes the input video into a hidden feature vector. Next, the embedded video feature vector is decoded using a 3 layer LSTM, to predict the binary movement vector and gaze. For Lvisual, we use the feature size 128 (obtained by a fully connected layer on top of the ResNet18 features) and a memory bank of size 16384. We choose δ = 1 in Equation 1, α = 0.09, β = 0.01, γ = 0.9 in Equation 3, and τ = 0.07 in Equation 2. We use a dropout of 0.5 and weight decay of 0.1. For data augmentation, we only use color jitter and random flip. We flip the entire sequence of images, swap the part movements for right and left arms and legs, and calculate the updated gaze in the flipped images.

heading: A.4.2 TARGET TASK NETWORKS

text:
Shared Implementation Details. During the target task training, the weights for the backbone are frozen. For all of our experiments, we use the Adam optimizer (Kingma & Ba, 2015) and images
are reshaped to 224× 224. The size of the hidden layer in our LSTM in all temporal experiments is 512. We use leaky-ReLU non-linearities between all network layers except for LSTMs.
Self-supervised Baseline Details. When training the MoCo encoder network, we use the SGD optimizer with 0.03 learning rate for the MoCo baseline since it performs best in this setting. For data augmentation, we use random cropping, horizontal flipping, gray scaling, and color jittering with the same parameters used in that work. We train the baseline with batch size 256 on 8 GPUs for 200 epochs with the same training regime as the original work.
Scene Classification. We use a decoder network of a single 1× 1 convolution layer that reduces the feature size from 512 × 7 × 7 to 64 × 7 × 7, followed by two fully connected layers, that convert these features to a vector of size 512 and then 397. We use the cross-entropy loss for training, and evaluate using mean per class top-1 accuracy.
Action Recognition. For this task, we train a single 1× 1 convolution layer that reduces the feature size from 512× 7× 7 to 64× 7× 7, followed by an LSTM to embed the video in one hidden vector of size 512, and two fully connected layers that convert the features to a vector of size 200 and then to the number of actions. As before, we use the cross-entropy loss as the objective and evaluate with mean per class top-1 accuracy. Since some of the action classes in this dataset appear in a limited set of videos, following one of the EPIC challenge finalists (Damen et al., 2019), we choose 9 verbs that result in a state transition, namely, take, put, open, close, wash, cut, mix, pour and peel and ignore verbs that do not cause a state transition (e.g., check).
Dynamic Prediction. We create a binary rectangular mask using the object bounding box. We use this mask image as the input to a two-layer convolutional network. As the result, we obtain a feature of size 64× 7× 7. These two convolutional layers are trained for both our method and the baseline. We concatenate the mask feature with the feature vector obtained from the image and add two fully convolutional layers on top, to obtain the class labels. Again, we optimize the network using the cross-entropy loss and use mean per class top-1 accuracy as the evaluation metric.
Depth Estimation. The ResNet backbone is connected to a Feature Pyramid Network (FPN) (Lin et al., 2017). We use Pixel Shuffle layers (Shi et al., 2016) for up-scaling the lower level features. The learned ResNet backbone is frozen; the 5 up-convolution layers are the only layers trained for the target task. We use the Huber loss as the objective.
Walkable Surface Estimation. The architecture of this network is the same as the depth estimation network. The ResNet backbone is frozen and five up-convolution layers are the only layers that are trained for the target task. We use the binary cross-entropy loss as the objective, where the goal is segmenting walkable and non-walkable pixels. For evaluation, we use the standard Intersection over Union (IOU) metric for segmentation tasks.

heading: A.5 RESULT OF FULL SUPERVISION

text: As a point of reference, we also provide the results using a fully supervised backbone that is trained using ImageNet. Neither our method nor the purely visual baselines use any supervision for representation learning. Therefore, a direct comparison is not fair. The results are shown in Table 5. The corresponding self-supervised results are shown in Table 1.

title: WHAT CAN YOU LEARN FROM YOUR MUSCLES? LEARNING VISUAL REPRESENTATION FROM HUMAN INTERACTIONS


summaries: This work investigates the intriguing phenomenon where pretraining on one task hurts the finetuning performance of another. Besides being interesting in general, this phenomenon has practical relevance as pretraining becomes increasingly popular with large-scale models. Here, the authors present a clean case for “negative pretraining effect” on images, and propose three ways to mitigate it.

summaries: This paper conducts an empirical study to examine the well-known negative transfer phenomenon (termed as a negative pretraining effect in this work) in neural networks. In particular, a network trained on a sequence of tasks performs inferior to a network trained from scratch on the intended target task. The main idea of the paper is to study this phenomenon by formulating and intervening on different constituents of the sequential learning process - (1) changing the learning rate across tasks, (2) number and type of tasks encountered in the learning process, and (3) resetting the model biases when going from one task to another. The paper conducts experiments on four visual classification datasets (CIFAR-10, FashionMNIST, MNIST, SVHN) and report their findings for sequential training of ResNet-18 architecture. They show that increasing the learning rate after training on the first task can alleviate the negative pretraining effect. They further showcase how different task discretization and resetting model biases help to reduce the effect.

summaries: The authors study what they call a negative pretraining effect = models pretrained on task 1 and tuned on task 2 sometimes underperform compared to just training on task 2 from scratch. This is an important factor in many forms of life long learning, multi-task learning and curriculum learning. They investigate 3 potential remedies / setups: a) using different learning rates for task 1 and task 2, b) changing from task 1 to task 2 more smoothly, and c) resetting network biases at different stages of the process. The perform with a single ResNet18 architecture on MNIST, Fashion MNIST, SVHN and CIFAR-10.

abstractText: Negative pretraining is a prominent sequential learning effect of neural networks where a pretrained model obtains a worse generalization performance than a model that is trained from scratch when either are trained on a target task. We conceptualize the ingredients of this problem setting and examine the negative pretraining effect experimentally by providing three interventions to remove and fix it. First, acting on the learning process, altering the learning rate after pretraining can yield even better results than training directly on the target task. Second, on the learning task-level, we intervene by increasing the discretization of data distribution changes from start to target task instead of “jumping” to a target task. Finally at the model-level, resetting network biases to larger values likewise removes negative pretraining effects, albeit to a smaller degree. With these intervention experiments, we aim to provide new evidence to help understand the subtle influences that neural network training and pretraining can have on final generalization performance on a target task in the context of negative pretraining.

authors:
Alessandro Achille
Matteo Rovere
Stefano Soatto

title: Critical learning periods in deep neural networks

venue: arXiv preprint arXiv:1711.08856,

authors:
Alessandro Achille
Michael Lam
Rahul Tewari
Avinash Ravichandran
Subhransu Maji
Charless C Fowlkes
Stefano Soatto
Pietro Perona

title: Task2vec: Task embedding for meta-learning

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Haitham Bou Ammar
Eric Eaton
Matthew E Taylor
Decebal Constantin Mocanu
Kurt Driessens
Gerhard Weiss
Karl Tuyls

title: An automated measure of mdp similarity for transfer in reinforcement learning

venue: In Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence,

authors:
Yoshua Bengio
Jérôme Louradour
Ronan Collobert
Jason Weston

title: Curriculum learning

venue: In Proceedings of the 26th annual international conference on machine learning,

authors:
Tom B Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell

title: Language models are few-shot learners

authors:
Marie Cheour
Rita Ceponiene
Anne Lehtokoski
Aavo Luuk
Jüri Allik
Kimmo Alho
Risto Näätänen

title: Development of language-specific phoneme representations in the infant brain

venue: Nature neuroscience,

authors:
Djork-Arné Clevert
Thomas Unterthiner
Sepp Hochreiter

title: Fast and accurate deep network learning by exponential linear units (elus)

venue: arXiv preprint arXiv:1511.07289,

authors:
Yann Le Cun
Ido Kanter
Sara A. Solla

title: Eigenvalues of covariance matrices: Application to neural-network learning

venue: Phys. Rev. Lett.,

authors:
Christian Darken
John E Moody

title: Note on learning rate schedules for stochastic optimization

venue: In Advances in neural information processing systems,

authors:
Eran Dayan
Leonardo G Cohen

title: Neuroplasticity subserving motor skill

venue: learning. Neuron,

authors:
Ethan Dyer
Guy Gur-Ari

title: Asymptotics of wide networks from feynman diagrams

venue: arXiv preprint arXiv:1909.11304,

title: Learning and development in neural networks: The importance of starting small

authors:
Dumitru Erhan
Pierre-Antoine Manzagol
Yoshua Bengio
Samy Bengio
Pascal Vincent

title: The difficulty of training deep architectures and the effect of unsupervised pre-training

venue: In Artificial Intelligence and Statistics,

authors:
Sebastian Farquhar
Yarin Gal

title: Towards robust evaluations of continual learning

venue: arXiv preprint arXiv:1805.09733,

title: Catastrophic forgetting in connectionist networks

venue: Trends in cognitive sciences,

authors:
F.A. Gers
J. Schmidhuber
F. Cummins

title: Learning to forget: continual prediction with lstm

venue: Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470),

authors:
Aditya Sharad Golatkar
Alessandro Achille
Stefano Soatto

title: Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence

venue: In Advances in Neural Information Processing Systems,

authors:
Alex Graves
Marc G Bellemare
Jacob Menick
Remi Munos
Koray Kavukcuoglu

title: Automated curriculum learning for neural networks

venue: In Proceedings of the 34th International Conference on Machine Learning-Volume

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

venue: In Proceedings of the IEEE international conference on computer vision,

title: Identity mappings in deep residual networks

venue: In European conference on computer vision,

authors:
Nicolas Heess
Dhruva TB
Srinivasan Sriram
Jay Lemmon
Josh Merel
Greg Wayne
Yuval Tassa
Tom Erez
Ziyu Wang
S.M. Ali Eslami
Martin A. Riedmiller
David Silver

title: Emergence of locomotion behaviours in rich environments

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Simplifying neural nets by discovering flat minima

venue: In Advances in Neural Information Processing Systems

authors:
Sepp Hochreiter
A Steven Younger
Peter R Conwell

title: Learning to learn using gradient descent

venue: In International Conference on Artificial Neural Networks,

authors:
Sergey Ioffe
Christian Szegedy

title: Batch normalization: Accelerating deep network training by reducing internal covariate shift

venue: arXiv preprint arXiv:1502.03167,

authors:
Lu Jiang
Deyu Meng
Qian Zhao
Shiguang Shan
Alexander G Hauptmann

title: Self-paced curriculum learning

venue: In Twenty-Ninth AAAI Conference on Artificial Intelligence,

authors:
Yiding Jiang
Behnam Neyshabur
Hossein Mobahi
Dilip Krishnan
Samy Bengio

title: Fantastic generalization measures and where to find them

authors:
Robert Kleinberg
Yuanzhi Li
Yang Yuan

title: An alternative view: When does sgd escape local minima

venue: arXiv preprint arXiv:1802.06175,

authors:
I Kohler
T Pissarek

title: Brillenversuche zur vertikalentäuschung [spectacle experiments on horizontal-vertical illusion

venue: Psychol Beiträge,

title: Learning multiple layers of features from tiny images

venue: Master’s thesis,

authors:
Patricia K Kuhl
Karen A Williams
Francisco Lacerda
Kenneth N Stevens
Björn Lindblom

title: Linguistic experience alters phonetic perception in infants by 6 months of age

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Yann A LeCun
Léon Bottou
Genevieve B Orr
Klaus-Robert Müller

title: Efficient backprop

venue: In Neural networks: Tricks of the trade,

authors:
Jaehoon Lee
Lechao Xiao
Samuel Schoenholz
Yasaman Bahri
Roman Novak
Jascha SohlDickstein
Jeffrey Pennington

title: Wide neural networks of any depth evolve as linear models under gradient descent

authors:
Timothée Lesort
Vincenzo Lomonaco
Andrei Stoian
Davide Maltoni
David Filliat
Natalia Dı́az-Rodrı́guez

title: Continual learning for robotics

venue: arXiv preprint arXiv:1907.00182,

authors:
Aitor Lewkowycz
Yasaman Bahri
Ethan Dyer
Jascha Sohl-Dickstein
Guy Gur-Ari

title: The large learning rate phase of deep learning: the catapult mechanism

venue: arXiv preprint arXiv:2003.02218,

authors:
Yanghao Li
Naiyan Wang
Jianping Shi
Jiaying Liu
Xiaodi Hou

title: Revisiting batch normalization for practical domain adaptation

venue: arXiv preprint arXiv:1603.04779,

authors:
Yuanzhi Li
Colin Wei
Tengyu Ma

title: Towards explaining the regularization effect of initial large learning rate in training neural networks, 2019

authors:
Shengchao Liu
Dimitris S. Papailiopoulos
Dimitris Achlioptas

title: Bad global minima exist and SGD can reach

venue: them. CoRR,

authors:
Hartmut Maennel
Ibrahim Alabdulmohsin
Ilya Tolstikhin
Robert JN Baldock
Olivier Bousquet
Sylvain Gelly
Daniel Keysers

title: What do neural networks learn when trained with random labels

authors:
Martial Mermillod
Aurélia Bugaiska
Patrick Bonin

title: The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects

venue: Frontiers in psychology,

title: Learning rate annealing can provably help generalization, even for convex problems

venue: arXiv preprint arXiv:2005.07360,

authors:
Cuong V Nguyen
Alessandro Achille
Michael Lam
Tal Hassner
Vijay Mahadevan
Stefano Soatto

title: Toward understanding catastrophic forgetting in continual learning

authors:
Samet Oymak
Mahdi Soltanolkotabi

title: Overparameterized nonlinear learning: Gradient descent takes the shortest path

venue: In International Conference on Machine Learning,

authors:
Olivier Pascalis
Michelle de Haan
Charles A Nelson

title: Is face processing species-specific during the first year of life

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga

title: Pytorch: An imperative style, high-performance deep learning library

authors:
Steven J Rennie
Vaibhava Goel
Samuel Thomas

title: Annealed dropout training of deep networks

venue: In 2014 IEEE Spoken Language Technology Workshop (SLT),

authors:
Hippolyt Ritter
Aleksandar Botev
David Barber

title: Online structured laplace approximations for overcoming catastrophic forgetting

authors:
Pierre Sachse
Ursula Beermann
Markus Martini
Thomas Maran
Markus Domeier
Marco R. Furtner

title: The world is upside down

venue: The Innsbruck Goggle Experiments of Theodor Erismann (1883–1961) and Ivo Kohler (1915–1985). Cortex,

title: Centering neural network gradient factors

venue: In Neural Networks: Tricks of the Trade,

authors:
Sihyeon Seong
Yegang Lee
Youngwook Kee
Dongyoon Han
Junmo Kim

title: Towards flatter loss surface via nonmonotonic learning rate scheduling

venue: In UAI,

authors:
Pierre Sermanet
Soumith Chintala
Yann LeCun

title: Convolutional neural networks applied to house numbers digit classification

venue: In Proceedings of the 21st International Conference on Pattern Recognition

authors:
Or Sharir
Barak Peleg
Yoav Shoham

title: The cost of training nlp models: A concise overview

venue: arXiv preprint arXiv:2004.08900,

authors:
Ravid Shwartz-Ziv
Naftali Tishby

title: Opening the black box of deep neural networks via information

venue: arXiv preprint arXiv:1703.00810,

authors:
Daniel L Silver
Qiang Yang
Lianghao Li

title: Lifelong machine learning systems: Beyond learning algorithms

venue: AAAI spring symposium series,

title: Cyclical learning rates for training neural networks

venue: IEEE Winter Conference on Applications of Computer Vision (WACV),

authors:
Rupesh K Srivastava
Klaus Greff
Jürgen Schmidhuber

title: Training very deep networks

authors:
Sebastian Thrun
Tom M Mitchell

title: Lifelong robot learning

venue: Robotics and autonomous systems,

authors:
Sebastian Thrun
Lorien Pratt

title: Learning to learn

venue: Springer Science & Business Media,

authors:
Jos Van Der Westhuizen
Joan Lasenby

title: The unreasonable effectiveness of the forget gate

venue: arXiv preprint arXiv:1804.04849,

authors:
Renato Vicente
Nestor Caticha

title: Functional optimization of online algorithms in multilayer neural networks

venue: Journal of Physics A: Mathematical and General,

authors:
Ricardo Vilalta
Youssef Drissi

title: A perspective view and survey of meta-learning

venue: Artificial intelligence review,

authors:
Zirui Wang
Zihang Dai
Barnabás Póczos
Jaime Carbonell

title: Characterizing and avoiding negative transfer

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Han Xiao
Kashif Rasul
Roland Vollgraf

title: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

venue: arXiv preprint arXiv:1708.07747,

authors:
Amir R Zamir
Alexander Sax
William Shen
Leonidas J Guibas
Jitendra Malik
Silvio Savarese

authors:
Zhanxing Zhu
Jingfeng Wu
Bing Yu
Lei Wu
Jinwen Ma

title: The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects

venue: arXiv preprint arXiv:1803.00195,

authors:
Julian Georg Zilly
Rupesh Kumar Srivastava
Jan Koutnı́k
Jürgen Schmidhuber

title: Recurrent highway networks

venue: In Proceedings of the 34th International Conference on Machine LearningVolume

heading: 1 INTRODUCTION

text:
Lifelong learning (Thrun & Mitchell, 1995; Silver et al., 2013) holds both the promise of benefitting from past experiences and the challenge of having to continually adapt to new problem settings. The characteristics of this intriguing sequential learning problem make it a balancing act between retaining knowledge and adapting to new experiences referred to as the stability-plasticity dilemma (Mermillod et al., 2013). Similar to a coach or teacher, we want to understand when a sequence of tasks helps or hinders further learning and what the underlying mechanism are that influence this process given that the cost of training very large models from scratch such as GPT-3 (Brown et al., 2020) has increased considerably (Sharir et al., 2020). This setting of training a model on a sequence of tasks to maximize its performance on a target task we coin the sequential learning problem.
The existing literature on sequential learning sheds some light on the characteristics of this problem setting. In some cases, following a sequence of learning tasks can lead to better results than simply training on the target task from scratch (Bengio et al., 2009). Achille and Soatto (Achille et al., 2017) and Wang et al. (2019) contrast this picture by highlighting that neural networks can suffer from pretraining on some tasks. When learning data is corrupted for a sufficiently long period, even switching back to clean data does not allow the model to recover its original performance (Maennel et al., 2020). Until compared, it may unforuntely be unclear that a model is corrupted.
Contribution: We formalize the basic ingredients of sequential learning as following a path on a learning manifold as depicted in Fig. 1 and provide three ways to remove the negative pretraining effect that can occur for certain task changes in neural network training as depicted in Fig. 2. We establish experimentally for a variety of task changes and datasets that this final performance can be highly dependent on the way the chosen model is trained, the sequence of training tasks, and how the model is perturbed during the training process. First, we demonstrate that increasing learning rates after pretraining can remove negative pretraining effects. Secondly, we show that continuously changing from source to target task instead of “jumping” can reap significant benefits in performance and does not incur a deficit penalty. Thirdly, we display that resetting only the biases for each task in the training process appears to decrease negative pretraining effects. Finally we speculate on plausible reasons such as qualitatively different training behavior during the “high learning rate phase” (Lewkowycz et al., 2020) that may explain the observed effects.

heading: 2 RELATED WORK

text:
Sequential learning involves choosing a sequence of tasks and learning processes and following this learning path to improve performance on a single or set of target tasks. This strongly resembles the objectives of lifelong learning (Thrun & Mitchell, 1995), continual learning (Lesort et al., 2019) and meta-learning (Thrun & Pratt, 2012; Schaul & Schmidhuber, 2010; Hochreiter et al., 2001) and connects to many time-dependent learning aspects in relation to neural networks.
Time-dependent learning: We are aware of several time-dependent phenomena in training neural networks such as catastrophic forgetting (French, 1999), critical learning periods (Achille et al., 2017), and time sensitivity of regularization and data augmentation (Golatkar et al., 2019). Achille et al. (2017) show that by dropping out certain frequencies during the beginning of training, networks are unable to recover to full performance, even when those frequencies are reintroduced. Similarly, Liu et al. (2019) show that first optimizing on random labels ruins performance on clean data shown indefinitely to the same network afterwards. Recent information-theoretic analysis (Shwartz-Ziv & Tishby, 2017) suggests that neural network have distinct learning phases: First, a phase where network parameters grow in information content, before, in a second phase, self-regularizing and pruning away irrelevant information. More commonly known time-dependent or sequential heuristics are learning rate schedules (Darken & Moody, 1991), cyclical learning rates (Smith, 2017), and regularization annealing such as dropout annealing (Rennie et al., 2014), student-teacher model transfer (Vicente & Caticha, 1997), and pretraining (Erhan et al., 2009).
Curriculum learning and generation: Curriculum learning (Elman, 1993; Bengio et al., 2009) is the technique of training agents by showing them a progression of tasks from easier variants to harder examples to attempt to solve the final target task. Both the utility of transferring between similar or dissimilar tasks is debated, with some results pointing towards similarity (Achille et al., 2019; Ammar et al., 2014; Ritter et al., 2018) and others towards dissimilarity (Farquhar & Gal, 2018; Nguyen et al., 2019). Self-paced learning (Jiang et al., 2015) extends curriculum learning by letting the model itself influence the sequence of tasks (Graves et al., 2017). The order of tasks, the curriculum, is generally an input that is hand-designed by human experimenters and often based on heuristics to guide the scheduling (i.e. it may be easier to make an agent jump over small gaps before moving to larger ones (Heess et al., 2017)). Meta-learning (Schaul & Schmidhuber, 2010; Vilalta & Drissi, 2002) is similarly related by focusing on speeding up or enhancing performance on a target task by pretraining on other tasks to first learn to learn.
Biological analogies: It is through neuroplasticity (Dayan & Cohen, 2011) that animals and humans learn new skills and recover from changes or damage to their sensory organs or brain. Famously, Erismann and Kohler (Kohler & Pissarek, 1960) asked study participants to wear special mirror goggles which flipped the participants’ visual field upside down as summarized in (Sachse et al., 2017). Remarkably, the participants visual perception adjusted after a few days by returning the visual field back to their “upright” standard perception. In contrast to this, there are other stimuli people learn to discard in childhood if they are not needed. Young children can differentiate phonemes of both their own and foreign languages; however, older children are better at distinguishing native phonemes and have more difficulty distinguishing foreign phonemes (Cheour et al., 1998; Kuhl et al., 1992). A similar effect of “perceptual narrowing” is found for 6-month-old and 9-month-old children when comparing the ability to discriminate between human faces to the ability to discriminate between primate faces (Pascalis et al., 2002). Younger children are again found to be less specialized to human faces than slightly older ones. Summarizing many results (Costandi, 2016), we currently know that neuroplasticity is “largest” at an early age yet generally does not end in adulthood. It is instead always present in reorganization processes involved in new experiences and when adjusting to injury.

heading: 3 SEQUENTIAL LEARNING SETTING

text:
We are interested in better understanding what influences the outcome of a sequential learning problem. In the following, we say that a pretrained model has plasticity if it can generalize similarly well when trained on a target task as an uncorrupted model trained directly on the target task. If it cannot generlize similarly well, we refer to this as a negative pretraining effect. To facilitate further reading, we summarize essential elements such as the definition of a learning task ⌧ , constrained to supervised tasks in this work, the learning mapping !, the learning path as an ordered sequence of task-learning process pairs T , and the setup of its repeated application in a sequential learning process ⌦ in Tab. 1 and visualize the overall setting in Fig. 1.
A task ⌧ consists of a probability distribution p(x, y) of data points and desired outputs as well as a loss function L that maps an input-output behavior of a model to a cost score. The learning mapping ! uses the information in task ⌧ and initial model finit to train a model optimized for the task flearn until convergence. Applying the learning mappings and tasks (⌧i,!i) 2 T sequentially starting from an initial model finit is summarized via the symbol ⌦. Based on these elements, we ask what influences the final generalization outcome on a target task when following a learning path of task-learn process pairs T . Specifically, we examine these question via three interventions aimed at distinct ingredients of the learning process as visualized in Fig. 2.

heading: 3.1 GENERAL EXPERIMENTAL BENCHMARK SETUP

text:
We will focus on stochastic gradient descent (SGD) learning applied to convolutional neural networks. As a stopping criterion, we consider the scenario where training has already converged, where the validation loss cannot be decreased further by changing the model f via SGD. Empirical task transfer often deals with tasks that are separate (Zamir et al., 2018), and while fruitful empirically, it becomes difficult to understand conceptually. Instead we employ controlled customized task changes, allowing us to better gauge the effect of task changes. With the basic building blocks in place, we ask in which cases, even with an infinite number of data points, previous tasks can hinder learning and generalization on new tasks. We examine three intervention scenarios acting on various levels of abstraction as visualized in Fig. 2:
A) Learning rate changes: Applying changes to the learning process, we observe how employing different learning rates both on the first and second task can influence pretraining effects.
B) Task discretization: Intervening on the task-level, we consider the influence of inserting more tasks between first and last task by discretizing the changes from first to last task.
C) Model biases resetting: Finally, we act on the model and examine how final performance is affected when resetting biases before tasks.
We deal with four visual supervised classification datasets of which we use three in every experimental section below thereby significantly extending smaller scale experiments by Achille et al. (2017):
• CIFAR-10 (Krizhevsky, 2009): A 32x32 full-color dataset with 10 classes. • FashionMNIST (Xiao et al., 2017): A 28x28, black-and-white dataset with 10 classes. • MNIST (LeCun et al., 1998): A 28x28, black-and-white dataset with 10 classes. • SVHN (Sermanet et al., 2012): A 32x32 full-color dataset with 10 classes.
For all experiments if not mentioned otherwise, we use Pytorch (Paszke et al., 2019) and run a ResNet-18 (He et al., 2016) with a fixed initial learning rate of 10e-4 with momentum 0.9 and weight decay 1e-3. The networks are initialized using the standard Pytorch layer intialization discussed in Appendix 8.2. Each task is trained until convergence, for a maximum of 200 epochs per task, using early stopping on a validation set with patience 50. As baseline, the model was trained only on the target task - the standard uncorrupted dataset.

heading: 4 HIGHER LEARNING RATES PREVENT A LOSS OF PLASTICITY

text:
The learning rate schedule is arguably one of the most important hyperparameters in neural network training as attested to by many deep learning practitioners (Li et al., 2019; Nakkiran, 2020; Smith, 2017). In several works (Seong et al., 2018; Zhu et al., 2018; Kleinberg et al., 2018; Lewkowycz et al., 2020), a higher learning rate has been linked to better generalization due to its increased probability to escape local minima during optimization. By escaping sharp minima, the network is deemed more likely to converge to a flat minimum (Hochreiter & Schmidhuber, 1995), i.e. a minimum with a
Hessian with lower maximal eigenvalue. Accordingly, we present a number of experiments detailing the effect of the learning rate on outcomes of neural network training after negative pretraining.
We keep the number of tasks, i.e. two, fixed and focus on the blurring transformation for all three datasets. For the first task, all images are downscaled to 1/4 their size before rescaling to their original size. The second task has standard unblurred images. The default learning rate ⌘ of 1e-3 is varied either for both tasks, meaning a constant learning rate over the learning path, or kept for the first and changed only for the second task. We picked 2e-3 as a higher and 5e-4 as a lower learning rate. Each experiment was performed for 10 random seeds. The baseline, trained directly on the standard uncorrupted data, is averaged across 3 random seeds. Results are shown in Fig. 3.
We observe the overall trend that higher learning rates appear to decrease the difference between models pretrained on blurred images and the baseline models trained on sharp images. Remarkably and consistently across datasets, training first with a learning rate 1e-3 and then 2e-3 is better in expectation than only training at the higher learning rate 2e-3. We conclude that in sequential learning, the precise order of learning rates across tasks can be crucial to obtain the best final learning loss.

heading: 5 SMOOTHLY CHANGING THE TASK CAN PREVENT A LOSS OF PLASTICITY

text:
Reasoning that a more “continuous” change from source to target task provides the model with more opportunity to adapt, we provide an intervention that changes the discretization of “jumping” from source to target task. To create each learning path, we focus on effects of three common image transformations: blurring, contrast, and random shifts, as visually depicted in the appendix in Fig. 7. Blurring applies a resizing operation to each image, before resizing it back to the original size (as done in (Achille et al., 2017)). Contrast changes an image’s channel attributes: A contrast of 0% results in a fully gray image1, whereas a contrast of 100% results in the standard image. Finally, random shift changes the maximum width of the random affine transformation applied to the image: Each component of the shift is sampled from [ a, a] and then applied to the image, where a depends on the task parameterization. For reproducibility further details are provided in appendix 8.1.
Discretization of paths: Traditionally, curriculum learning is concerned only with the sequence of tasks. In contrast, to test plasticity, we are additionally concerned with how discretely we traverse the space from source to target tasks. To characterize this, we keep the learning method fixed and segment the one-dimensional task path space into N tasks as shown in Fig. 4 and 5 and focus on how N affects target task performance. We continuously transform tasks from source to target task, without violating the definition that all tasks lie on a task manifold that can be derived from the definition of a task2. All discretization experiments in this section were run with a maximum of 100 epochs and patience of 20 epochs across 3 random seeds. As visualized in Fig. 4, increasing discretization removes most
1Using 10% minimum contrast to ensure meaningful gradients. 2This rules out naive interpolation between two standard computer vision datasets.
of negative pretraining of blurring and contrast on CIFAR-10. Remarkably, the already improved results of random shift are further improved via discretization. On the other hand shows no clear trend on the FashionMNIST and SVHN datasets is visible. Possibly on these datasets, the “nudge” that increased discretization provides is not enough to overcome entrenched model characteristics.
Multi-dim. paths from start to target task: We extend the sequential learning setup to multiple dimensions as visualized in Fig. 5. Previously we only varied tasks along one transformation “dimension”. Unlike the one-dimensional case, we now vary both contrast or random shift and blurring levels (see Fig. 5, left) at the same time. Each path starts with blurred images and either low contrast or high random shift transformation. Then we reduce transformations in the order mentioned in the title of each subplot.3 We present results of these paths on CIFAR-10 in Fig. 5. Across most experiments and transformations, we see that performance on the target task increases as we approach a continuous density transformation between source and target tasks (i.e. as N increases). We observe that continuous changes in tasks can help networks adapt to a new task ; in contrast, jumpy paths induce a distributional shift too large to overcome, leading to poor performance on the target task.
3Example: Blur then Contrast means that we start at blurred and low contrast images. The next tasks will first move to unblur the images, but keep them at low contrast. Once unblurred, task move to increase the contrast. Diagonal: Refers to lowering both transformations equally for each discretization step.
CIFAR-10
FashionMNIST
SVHN

heading: 6 BIAS RESETTING CAN PREVENT A LOSS OF PLASTICITY

text:
The neural network literature is ripe with examples of initializing a model to be more amenable to optimization. For sequence modeling, the LSTM network is commonly initialized by setting biases such that the forget gate is inactive (Gers et al., 1999; Van Der Westhuizen & Lasenby, 2018). The same method has also found application in the first very deep networks such as Highway Networks (Srivastava et al., 2015) and its recurrent extension Recurrent Highway Networks (Zilly et al., 2017) to ensure that gradients flow through the entire network. Batch normalization (BN) (Ioffe & Szegedy, 2015) aims to normalize the bias and variance of network activations to improve training. Building upon this, Li et al. (2016) adapted to a new domain solely by retraining the BN weights while leaving all other parameters unchanged. Likewise we reason that resetting biases may, albeit shortly, activate many ReLU-activations in the network and thus allow gradients to flow freely.
In this section we focus on the effect of resetting the network biases. Again, we use a task path consisting of two tasks and blurring as transformation for all three datasets. Using bias reset (BR) implies, that before training on a task all biases are reset by adding 0.01 to their value manually chosen to be of a similar order of magnitude as the largest biases in the network. Depending on which task bias resetting is used, we distinguish the following set-ups:
1. No BR - No BR: No bias resetting is used for both task, which is the default set-up. 2. No BR - BR: Only before training on the second task the network biases are reset. 3. BR - No BR: Just right in the beginning, before any training, the biases are reset. 4. BR - BR: The biases are reset before both training on the first task and the second task.
Again we used 10 different random seeds for the experiments and provide as baseline which is averaged across three random seeds. Viewing results in Fig. 6, we note that the removal of the negative pretraining effect is most pronounced on the CIFAR-10 dataset. Remarkable for this intervention is that resetting biases even before the blurred image task and not before training on sharp images ([Yes, No]) can remove the negative pretraining effect. On FashionMNIST and SVHN, the BR-intervention improves the test loss slightly ([Yes, Yes] vs. [No, No]). For these two datasets, the negative pretraining effect is however also less substantial. Notable however is that with BR intervention some seeds now are better than the baseline whereas before [No,No] this was not the case. One observed difference between CIFAR-10 and the two other datasets is the amount of training until best validation epoch. Bias reset leads to an increase in the number of epochs (about +44% for [Yes, Yes] compared to [No, No]) for CIFAR-10, while for FashionMNIST and SVHN the number of epochs decreases when using bias reset ( 44%).To corroborate our understanding of these results we provide norms of biases, weight as well activation statistics in appendix 8.3.

heading: 7 SPECULATION ON CAUSES OF THE NEGATIVE PRETRAINING EFFECT

text:
What we have shown are three markedly distinct ways of removing negative pretraining effects. What remains unclear is why we observe such a learning and generalization behavior. In the following, we will attempt an explanation or speculation of the underlying causes of the observed learning phenomena. An important aspect to note before diving in deeper is that the negative pretraining effect is not an optimization issue. In all cases the training loss is reduced either to zero or to the loss it generally reaches upon convergence even when generalizing well. Negative pretraining causes a change in generalization on the test set but not a drop in performance on the training set.
Large update rate phase: A plausible explanation for the observed behavior is that the phase with larger updates similar to the large learning rate phase for neural networks is qualitatively different to updates in the low-loss setting. All interventions achieve settings where either A) a higher learning rate makes all updates larger, B) a higher loss setting is encountered more frequently, or C) resetting slows down convergence. Such a high learning rate phase is associated with better generalization.
More specifically, Zhu et al. (2018) claim that the anisotropic noise of SGD helps the network escape sharp minima and thus generalize better. The escape probability can be higher, if the training loss is still large compared to the loss at the minimum of the SGD optimization. Seong et al. (2018) evoke the same image, plausibly showing that large learning rates make it more likely to escape sharp minima. In the same spirit Nakkiran (2020) provides an example of how a mismatch between training and test loss landscape can favor optimization with an initially large learning rate. Lewkowycz et al. (2020) argue that in the large learning rate phase the curvature of the loss landscape is lowered thus yielding a tangible difference in optimization behavior. Focusing on bias resetting, centered biases are known to accelerate convergence (Clevert et al., 2015; Cun et al., 1991; LeCun et al., 2012; Schraudolph, 1998). Ensuring that biases are not centered is thus likely to have the opposite effect of slowing convergence which may ensure training remains in the large update rate phase longer.
Preshaped learning landscape: Taking the opposite viewpoint when gradient updates are not large, optimization is likely to occur within the “optimization valleys” in which the optimization started due to pretraining. In the limit case of a converged network, such a scenario is described for very wide neural networks (Lee et al., 2019). These wide networks are characterized by the neural tangent kernel (NTK). The largest eigenvalue of the NTK is in turn tied to generalization behavior (Lewkowycz et al., 2020; Jiang et al., 2019; Dyer & Gur-Ari, 2019). Possibly, the tangent kernel is not updated enough to overcome the negative pretraining effect when retraining on the target task. Corroborating this point, Oymak & Soltanolkotabi (2019) argue that SGD, under certain conditions, takes a direct path to the minimum such that final parameters may change little from initial parameters. Thus all interventions may serve to increase the training time, given the network more time and opportunity to adapt not only its output but also its tangent kernel to the target task.

heading: 8 DISCUSSION AND CONCLUSION

text:
The proposed three interventions provide us with a host of insights on what determines plasticity as the ability to adapt in sequential learning. We learned that increasing the learning rate after pretraining can consistently benefit learning to the point of surpassing the performance of only training on standard data. In the discretization experiment setting, we learn that pretraining on another task can have a significant biasing effect on final results. By providing a finer discretization from source to target task, possible retraining issues are alleviated thus yielding a better target performance. Finally, perturbing the model after pretraining by resetting biases can similarly diminish the negative pretraining effect albeit not consistently. Resetting the biases of a network can increase the initial training loss for a new task and slow down convergence.
Several new questions arise from the experiments. What is the best task to start from? What is the best path of tasks and learning processes to follow thereafter? Which kind of model is best suited to make use of the advantages of a carefully-chosen path?
In this work, we presented a novel perspective on the sequential learning behavior of neural networks through the lens of negative pretraining. While casting light on issues a learning model may face in a sequential learning setup, we also discover a variety of new questions. The picture we are left with is an intriguing sequential learning problem setting offering a fruitful ground for further inquiry.