summaries: This paper presents a HyperGrid Transformer approach to fine-tuning, where one takes a pre-trained transformer model and then modifies it by introducing hypernetworks that modify the 2nd FFN in each transformer block by generating additional weights conditioned on input. These hyper-networks are trained on all tasks in GLUE/SuperGLUE datasets simultaneously and are task aware through prefixing of a task specific token to input. This allows one to fine-tune only a small number of parameters and end up with a model that performs quite well on all tasks at the same time, not much worse than fine-tuning the entire transformer model on all of these tasks.

summaries: The authors propose HyperGrid Transformers with a decomposable hypernet-work that learns grid-wise projections to specialize regions in weight matrices for different tasks. Usually, people would use different models to solve different tasks respectively. In this paper, the authors focus on using a single model to solve all tasks and it will save a lot of model parameters for natural language understanding. And the authors have done comprehensive experiments on GLUE and SuperGLUE, and prove that the proposed single model can achieve much better performance than baseline and competitive performance with multiple task-specific models.

summaries: This manuscript presents a HyperGrid Transformer, which is engaged in learning a single model to account for multi-tasks in NLP. The core idea of HyperGrid Transformer is to learn task-conditional dynamic weights in a grid-wise manner in the feed-forward layers, where the weights are factorized in local and global components. This idea is simple, materializing the goal of reducing the parameter cost for the used multi-task network. However, the conducted experiments look nice, showing promising performance on GLUE/SuperGLUE. Therefore, from my point of view, this work is worthy of a publication at ICLR.

abstractText: Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning a single multi-task model that is able to do well for all the tasks has been a challenging and yet attractive proposition. In this paper, we propose HyperGrid Transformers, a new Transformer architecture that leverages task-conditioned hyper networks for controlling its feed-forward layers. Specifically, we propose a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks. In order to construct the proposed hypernetwork, our method learns the interactions and composition between a global (task-agnostic) state and a local task-specific state. We conduct an extensive set of experiments on GLUE/SuperGLUE. On the SuperGLUE test set, we match the performance of the state-of-the-art while being 16 times more parameter efficient. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.

authors:
Roy Bar-Haim
Ido Dagan
Bill Dolan
Lisa Ferro
Danilo Giampiccolo
Bernardo Magnini
Idan Szpektor

title: The second pascal recognising textual entailment challenge

venue: In Proceedings of the second PASCAL challenges workshop on recognising textual entailment,

authors:
Luisa Bentivogli
Peter Clark
Ido Dagan
Danilo Giampiccolo

title: The fifth pascal recognizing textual entailment challenge

venue: In TAC,

title: Multitask learning

venue: Machine learning,

authors:
Daniel Cer
Mona Diab
Eneko Agirre
Inigo Lopez-Gazpio
Lucia Specia

title: Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation

venue: arXiv preprint arXiv:1708.00055,

authors:
Christopher Clark
Kenton Lee
Ming-Wei Chang
Tom Kwiatkowski
Michael Collins
Kristina Toutanova

title: Boolq: Exploring the surprising difficulty of natural yes/no questions

venue: In NAACL,

authors:
Kevin Clark
Minh-Thang Luong
Urvashi Khandelwal
Christopher D Manning
Quoc V Le

title: Bam! born-again multi-task networks for natural language understanding

venue: arXiv preprint arXiv:1907.04829,

authors:
Kevin Clark
Minh-Thang Luong
Quoc V Le
Christopher D Manning

title: Electra: Pre-training text encoders as discriminators rather than generators

venue: arXiv preprint arXiv:2003.10555,

authors:
Ido Dagan
Oren Glickman
Bernardo Magnini

title: The pascal recognising textual entailment challenge

venue: In Machine Learning Challenges Workshop,

authors:
Marie-Catherine De Marneff
Mandy Simons
Judith Tonhauser

title: The commitmentbank: Investigating projection in naturally occurring discourse

venue: proceedings of Sinn und Bedeutung

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
William B Dolan
Chris Brockett

title: Automatically constructing a corpus of sentential paraphrases

venue: In Proceedings of the Third International Workshop on Paraphrasing (IWP2005),

authors:
Robert M French
Nick Chater

title: Using noise to compute error surfaces in connectionist networks: A novel means of reducing catastrophic forgetting

venue: Neural computation,

authors:
Danilo Giampiccolo
Bernardo Magnini
Ido Dagan
Bill Dolan

title: The third pascal recognizing textual entailment challenge

venue: In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing,

authors:
Kazuma Hashimoto
Caiming Xiong
Yoshimasa Tsuruoka
Richard Socher

title: A joint many-task model: Growing a neural network for multiple nlp tasks

venue: arXiv preprint arXiv:1611.01587,

authors:
Shankar Iyer
Nikhil Dandekar
Kornel Csernai

title: First quora dataset release: Question pairs, 2017

authors:
Lukasz Kaiser
Aidan N Gomez
Noam Shazeer
Ashish Vaswani
Niki Parmar
Llion Jones
Jakob Uszkoreit

title: One model to learn them all

venue: arXiv preprint arXiv:1706.05137,

authors:
Daniel Khashabi
Snigdha Chaturvedi
Michael Roth
Shyam Upadhyay
Dan Roth

title: Looking beyond the surface:a challenge set for reading comprehension over multiple sentences

venue: In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL),

authors:
James Kirkpatrick
Razvan Pascanu
Neil Rabinowitz
Joel Veness
Guillaume Desjardins
Andrei A Rusu
Kieran Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the national academy of sciences,

authors:
Zhenzhong Lan
Mingda Chen
Sebastian Goodman
Kevin Gimpel
Piyush Sharma
Radu Soricut

title: Albert: A lite bert for self-supervised learning of language representations

authors:
Jason Lee
Kyunghyun Cho
Thomas Hofmann

title: Fully character-level neural machine translation without explicit segmentation

venue: Transactions of the Association for Computational Linguistics,

authors:
Hector Levesque
Ernest Davis
Leora Morgenstern

title: The winograd schema challenge

venue: In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning,

authors:
Xiaodong Liu
Pengcheng He
Weizhu Chen
Jianfeng Gao

title: Multi-task deep neural networks for natural language understanding

venue: arXiv preprint arXiv:1901.11504,

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke Zettlemoyer
Veselin Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

venue: arXiv preprint arXiv:1907.11692,

authors:
Jiaqi Ma
Zhe Zhao
Xinyang Yi
Jilin Chen
Lichan Hong
Ed H Chi

title: Modeling task relationships in multi-task learning with multi-gate mixture-of-experts

venue: In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,

authors:
Jiaqi Ma
Zhe Zhao
Jilin Chen
Ang Li
Lichan Hong
Ed H Chi

title: Snr: Sub-network routing for flexible parameter sharing in multi-task learning

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Bryan McCann
Nitish Shirish Keskar
Caiming Xiong
Richard Socher

title: The natural language decathlon: Multitask learning as question answering

venue: arXiv preprint arXiv:1806.08730,

authors:
James L McClelland
Bruce L McNaughton
Randall C O’Reilly

title: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory

venue: Psychological review,

authors:
Michael McCloskey
Neal J Cohen

title: Catastrophic interference in connectionist networks: The sequential learning problem

venue: In Psychology of learning and motivation,

authors:
Jonas Pfeiffer
Ivan Vulić
Iryna Gurevych
Sebastian Ruder

title: Mad-x: An adapter-based framework for multi-task cross-lingual transfer

venue: arXiv preprint arXiv:2005.00052,

authors:
Mohammad Taher Pilehvar
os’e Camacho-Collados

title: Wic: 10, 000 example pairs for evaluating context-sensitive representations

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer

venue: arXiv preprint arXiv:1910.10683,

authors:
Pranav Rajpurkar
Jian Zhang
Konstantin Lopyrev
Percy Liang

title: Squad: 100,000+ questions for machine comprehension of text

venue: arXiv preprint arXiv:1606.05250,

authors:
Melissa Roemmele
Cosmin Adrian Bejan
Andrew S Gordon

title: Choice of plausible alternatives: An evaluation of commonsense causal reasoning

venue: In 2011 AAAI Spring Symposium Series,

title: An overview of multi-task learning in deep neural networks

venue: arXiv preprint arXiv:1706.05098,

authors:
Victor Sanh
Thomas Wolf
Sebastian Ruder

title: A hierarchical multi-task approach for learning embeddings from semantic tasks

authors:
Abigail See
Peter J Liu
Christopher D Manning

title: Get to the point: Summarization with pointergenerator networks

venue: arXiv preprint arXiv:1704.04368,

authors:
Noam Shazeer
Mitchell Stern

title: Adafactor: Adaptive learning rates with sublinear memory cost

venue: arXiv preprint arXiv:1804.04235,

authors:
Noam Shazeer
Azalia Mirhoseini
Krzysztof Maziarz
Andy Davis
Quoc Le
Geoffrey Hinton
Jeff Dean

title: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

venue: arXiv preprint arXiv:1701.06538,

authors:
Noam Shazeer
Youlong Cheng
Niki Parmar
Dustin Tran
Ashish Vaswani
Penporn Koanantakool
Peter Hawkins
HyoukJoong Lee
Mingsheng Hong
Cliff Young

title: Mesh-tensorflow: Deep learning for supercomputers

venue: In Advances in Neural Information Processing Systems,

authors:
Richard Socher
Alex Perelygin
Jean Wu
Jason Chuang
Christopher D Manning
Andrew Ng
Christopher Potts

title: Recursive deep models for semantic compositionality over a sentiment treebank

venue: In Proceedings of the 2013 conference on empirical methods in natural language processing,

authors:
Asa Cooper Stickland
Iain Murray

title: Bert and pals: Projected attention layers for efficient adaptation in multi-task learning

authors:
Ilya Sutskever
Oriol Vinyals
Quoc V Le

title: Sequence to sequence learning with neural networks

venue: In Advances in neural information processing systems,

authors:
Johannes von Oswald
Christian Henning
João Sacramento
Benjamin F Grewe

title: Continual learning with hypernetworks

venue: arXiv preprint arXiv:1906.00695,

authors:
Alex Wang
Amanpreet Singh
Julian Michael
Felix Hill
Omer Levy
Samuel R Bowman

title: Glue: A multi-task benchmark and analysis platform for natural language understanding

venue: arXiv preprint arXiv:1804.07461,

authors:
Alex Wang
Yada Pruksachatkun
Nikita Nangia
Amanpreet Singh
Julian Michael
Felix Hill
Omer Levy
Samuel Bowman

title: Superglue: A stickier benchmark for general-purpose language understanding systems

authors:
Alex Warstadt
Amanpreet Singh
Samuel R Bowman

title: Neural network acceptability judgments

venue: arXiv preprint arXiv:1805.12471,

authors:
Adina Williams
Nikita Nangia
Samuel Bowman

title: A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122

venue: Association for Computational Linguistics,

authors:
Sen Wu
Hongyang R Zhang
Christopher Ré

title: Understanding and improving information transfer in multi-task learning

venue: arXiv preprint arXiv:2005.00944,

authors:
Zhilin Yang
Zihang Dai
Yiming Yang
Jaime Carbonell
Russ R Salakhutdinov
Quoc V Le

title: Xlnet: Generalized autoregressive pretraining for language understanding

authors:
Jaehong Yoon
Saehoon Kim
Eunho Yang
Sung Ju Hwang

title: Oracle: Order robust adaptive continual learning

venue: arXiv preprint arXiv:1902.09432,

authors:
Poorya Zaremoodi
Wray Buntine
Gholamreza Haffari

title: Adaptive knowledge sharing in multitask learning: Improving low-resource neural machine translation

venue: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),

authors:
Sheng Zhang
Xiaodong Liu
Jingjing Liu
Jianfeng Gao
Kevin Duh
Benjamin Van Durme

title: Record: Bridging the gap between human and machine commonsense reading comprehension

venue: arXiv preprint arXiv:1810.12885,

title: QNLI (Rajpurkar et al., 2016)

venue: RTE (Dagan et al.,

heading: 1 INTRODUCTION

text:
Learning a single multi-task model that performs well across multiple targeted tasks is an attractive proposition for many reasons (Kaiser et al., 2017; Ruder, 2017; Clark et al., 2019b). Although extremely challenging, this paradigm enables a substantial savings in overall parameter costs, along with eliminating the need for maintaining multiple models in production (Stickland and Murray, 2019). However, achieving state-of-the-art performance on natural language understanding benchmarks today (Wang et al., 2018; 2019) still relies on fine-tuning a new model for every single task. This methodology is infeasible in many situations. Moreover, certain tasks rely on an extensive ensemble of models and/or task-specific fine-tuning tricks (Liu et al., 2019b; Devlin et al., 2018; Clark et al., 2020).
The single-task fine-tuning paradigm is well-established to be the dominant approach (Raffel et al., 2019), as training multiple tasks using a single set of parameters can be problematic in many ways, such as catastrophic forgetting (French and Chater, 2002; McCloskey and Cohen, 1989; McClelland et al., 1995; Kirkpatrick et al., 2017) or the inherent difficulty of finding a consistently good model for all tasks (Clark et al., 2019b; Wu et al., 2020). Inevitable task conflicts and difficulty in fitting all models within a set of hard parameters is also a challenging problem for multi-task co-training.
In this paper, we propose a new Transformer architecture, the HyperGrid Transformer for efficient modeling of multiple tasks within a single set of model parameters. HyperGrid Transformers rely on a hypernetwork-based (Ha et al., 2016) module that performs gridwise decomposable hyper projections. This module is task conditioned and dynamically learns to generate weights of the feedforward layers of the Transformer model.
Overall, our eventual goal is to dispense with task specific fine-tuning tricks altogether. While neural networks typically maintain the same consistent set of parameters for all input instances, the proposed HyperGrid Transformers introduces instance-specific parameters by conditioning on the current input. This setup enables our model to learn task-specific reparameterization for each input instance, which mitigates several challenges of multi-task co-training.
Our proposed HyperGrid module belongs to a family of hypernetworks (Ha et al., 2016), in which a side network is responsible for weight generation for the main network. In our case, task-conditioned hypernetworks provide greater flexibility and expressiveness for capturing the dynamics of multiple tasks within a single set of parameters. Specifically, we introduce two novel algorithmic improvements over the existing methods.
First, we introduce the notion of grid-wise projections in which we assume a structural layout in vanilla projection layers. For each input sample, our projections dynamically control the parameters in a grid-wise, region-specific manner. The structural segmentation of feed-forward layers is similar in spirit to mixture-of-experts gating (Shazeer et al., 2017), albeit at a lower-level. Conversely, standard hypernetworks only consider row-wise re-weighting of weight matrices.
Second, we introduce decomposable hyper-projections. The key idea is to learn rich compositional and pairwise interactions between dual hypernetworks. A dual setup is adopted, where we explore different hypernetwork composition variants. We introduce a novel local-global setup, which composes a local instance-specific and task-specific hyper-projection with a task agnostic global state embedding. This is intuitive since this setup is not only highly expressive and flexible but also serves as a factorization of local and global components. To the best of our knowledge, our work is the first to explore this setup with respect to learning conditional parameters.
Finally, we conduct extensive experiments on GLUE/SuperGLUE. Our proposed model is able to match the performance of individually fine-tuned state-of-the-art Text-to-Text Transformers (T5) (Raffel et al., 2019) models with a single model that is learned to fit all GLUE and SuperGLUE tasks at once. Moreover, our single model also outperforms strong baselines that employ ensembling and other task-specific tricks (Liu et al., 2019b; Clark et al., 2020).
Our Contributions The contributions of this paper can be summarized as follows:
• We propose HyperGrid Transformers, a form of hypernetwork-based Transformer that learns task-conditioned dynamic weights for its feed-forward layers.
• The key novelty behind HyperGrid Transformers is the factorization of local and global components for weight generation. Our weight generation is grid-wise and imbues the model with a structural layout.
• We conduct extensive experiments on natural language understanding benchmarks (GLUE/SuperGLUE). With a single model, we match the state-of-the-art T5 model that is finetuned in a per-task fashion (multiple models), resulting in 16x parameter savings.

heading: 2 HYPERGRID TRANSFORMERS

text: This section outlines the key idea of the proposed algorithm.

heading: 2.1 HYPERGRID MODULE

text:
HyperGrid operates on weight matrices (linear transformations), i.e., Y = WX + b. In a hypernetwork formulation, instead of letting W be free weights, we generate W using a parameterized side network H(.).
Y = Wx+ b where W = H(X) (1)
where W ∈ Rdm×df . In the case where X is a single vector ∈ Rdm , we may parameterize H(.) with a simple feed-forward layer.
H(X) = σ(UX)1> W (2)
where 1 is a column vector of ones, σ is the sigmoid activation function and U ∈ Rdm×df . The key idea is that the hypernetwork generates a vector, i.e., UX ∈ Rdf that is broadcast (multiplied by 1) and multiplied by W , acting as a row-wise scaling of W . We are also able to reduce U ∈ Rdm×n where df mod n = 0 and repeat the vector df n times to form the original dimension of df . These methods only consider scaling one dimension of W (e.g., row-wise). We now consider methods beyond simple row-wise weight scaling.

heading: 2.1.1 DECOMPOSABLE GRIDWISE PROJECTIONS

text:
In our method, we propose grid-wise projections that segments W into a grid, i.e., blocks of dmdr × df dc
. We generate blocks by the outer product of Lr ∈ Rdr and Lc ∈ Rdc . Note that dr and dc are user-specified hyperparameters that control the grid-size for the fan-in and fan-out of the output matrix. For simplicity, we consider divisible blocks where dr < dm, dm mod dr = 0, and dc < df , df mod dc = 0. In this case:
H(X) = ψ(σ((LrX)(LcX) >)) W (3)
where (LrX)(LcX)> ∈ Rdr×dc , ψ(.) is a repeat vector function that repeats its input dmdr times on the row axis and dfdc times on the column axis. We name this approach the L2 variant, short for Local-Local Gridwise Projection.
Composition between Local and Global Factors The decomposable grid-wise projections learn Lr and Lc from X , which makes it conditioned on local, instance-wise information. Here, we postulate that it may be beneficial for either Lr or Lc to be a global embedding. By keeping Lc as a global, trainable embedding, this can be formulated as:
H(X) = ψ(σ((LrX)G > c )) W (4)
where Gc ∈ Rdf . In this case, Lr is conditioned from X , the specific input sample. On the other hand, Gc remains consistent across all input samples. Hence, the outer product is essentially a rich dyadic composition between local and global factors.
Local-Global and Global-Local It is easy to see that there are two ways of composing L and G. The above method considers the Local-Global approach where the fan-in uses a local hypernetwork and the global part uses a trainable embedding. An alternative that flips this around to use a GlobalLocal composition is evaluated in our experiments. Namely, this can be expressed as:
H(X) = ψ(σ((Gr(LcX) >)) W (5)

heading: 2.2 DYNAMIC WEIGHT GENERATION WITH HYPERGRID

text:
This section describes how we use task-conditioned hypernetworks to influence and generate the parameters for HyperGrid Transformers.
module.
Weight Gating The HyperGrid module is added at the position-wise feed-forward layers of the Transformer models. More specifically, we equip the second positional FFN after the ReLU activations with HyperGrid. There are several reasons for doing so. In most Transformer implementations, the fan out of this layer is typically scaled up to very large values (Raffel et al., 2019). Hence, the influence on this layer has the greatest potential to benefit the Transformer model. Second, early experiments on both of the positional feed-forward layers yielded no substantial improvements. Hence, we opt to only modify the second positional FFN of the Transformer model.
Initialization In our experiments, we take advantage of existing pretrained model (Raffel et al., 2019) and add additional HyperGrid parameters that are fine-tuned along with the rest of the network. The overall formulation of the HyperGrid-enhanced Transformer can be written as:
Yi = Hi(Xi−1,Wi) +Wi(Xi−1) (7)
where i denotes the layer i. We construct a new HyperGrid (with non-shared parameters) for each layer. Since W has been pretrained, we also add a residual connection of the original Wi(Xi−1) computation to the mixture.
Parameter Costs We note that the parameter counts added by HyperGrid are relatively negligible since dr and dc are small. In the LG setting, the model adds dmdr+dc parameters at each layer. On the GL setting, the parameter cost added is dr + dfdc. The most expensive option is L2 where the added cost is dmdr + dfdc. These added parameter costs are often negligible for large Transformer models.

heading: 3 EXPERIMENTAL RESULTS

text: We conduct experiments on GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) which are consolidated benchmarks of multiple challenging NLP and NLU tasks. While most of the work in this area has been focused on achieving good task-specific performance, our work focuses on trying to get good performance with a single model on all GLUE and SuperGLUE tasks. Therefore, most of our experiments are conducted on a mixture of all GLUE and SuperGLUE tasks.

heading: 3.1 EXPERIMENTAL RESULTS

text: In this section, we discuss the empirical results of our experiments. Further details about the experimental setup can be found in the appendix. Pertaining to parameter counts (reported as θ in our experiments), this is the total parameter cost for serving the entire suite of tasks. If a single model (finetuned on a single task) is X parameters, then the costs of serving N tasks in a multi-model setup would be NX .

heading: 3.1.1 SINGLE MODEL VERSUS MULTIPLE MODELS

text:
These experiments investigate different model settings, namely using (1) multiple models (MM) or a single model (SM) for all tasks. The MM model trains a fresh model on every single task and retains the best checkpoint for all tasks. Meanwhile, the SM model is trained on all tasks at once. We compare them against HGT, our single model HyperGrid Transformer approach.
Details about Baselines For the Single Model Baseline, this is the identical T5 model without our HyperGrid layers which serves as a fair comparison to our model. This enables us to observe the effect of HyperGrid directly. As for the multi-model method, this is done by finetuning T5 directly on each task and reporting results from the best checkpoint (of each task). This is a very strong baseline because the model is also allowed to choose specific best checkpoints for each task which the single model approach is prohibited against. As for the sampling strategy in the singlemodel approach (baseline and our approaches), we use a proportionate mix of tasks according to the number of samples in the each task/dataset.
Results Table 2 reports results of our experiments on the GLUE and SuperGLUE benchmark. The first key observation is that the single model (SM) approach is outperformed by multi model (MM). This is a well known phenomena and therefore multi-model is generally adopted when the absolute best score is desired on every single task. The interesting result is that we are able to come rather close to the performance of single model with our approach. As a result, the multi-model has 16x more parameters. To fit both GLUE and SuperGLUE, this would require 16x the parameters. Given that our goal is to bridge the performance of a single model versus multiple models for multiple tasks, we find that this result is considerably successful. Moreover, we observe that our singlemodel approach outperforms the multi-model baseline by +0.6% on average across 8 tasks. We observe similar trends as on the GLUE benchmark. Naturally, the best model is the multi-model model which involves finetuning a specialized model for each task. The gap between multi-model and single-model is at 74.8 versus 73.6. Our approach bridges this gap, improving the single-model score to 74.5, competitive with the multi-model approach.

heading: 3.1.2 PERFORMANCE GAINS ACROSS MODEL SIZES

text:
We investigate the gains of the proposed HyperGrid over the base model on various sizes of the T5 model. For models larger than Base, we train with 64 TPU V3 chips for 200K steps and select the best checkpoint for all tasks based on the benchmark score.
Model GLUE SuperGLUE AVG Base 84.99 73.55 79.27 Ours 85.22 (+0.27%) 75.30 (+2.7%) 80.26 (+1.3%) Large 88.22 80.04 84.13 Ours 88.31 (+0.1%) 81.56 (+1.9%) 84.94 (+1.0%) 3B 89.53 84.22 86.87 Ours 89.67 (+0.2%) 85.75 (+1.8%) 87.71 (+1.0%)

heading: 3.1.3 EFFECT OF MODELING CHOICES

text:
To ascertain the effectiveness of our approach, we test different architectural variants of HyperGrid Transformers, along with other architectural variants considered during model development.
Setup We evaluate all four model variants of HyperGrid Transformers (L, L2, GL and LG). For the other architectural variants, we were mainly interested to know if a hypernetwork setup (weight gating) is better than gating on the output representations (details can be found in the supplementary material). For the base setting, we ran the baseline T5 model (single-model) four times and reported the mean and standard deviation of the runs. When comparing the performance gain of our method, we compare against the max run of the baseline runs. We report relative performance gains/loss against this max baseline score. We conduct ablation studies on the four composition types on the large models1.
Is Output Gating Better? The other architectural variants (OutGate) do not perform well and generally perform with a net loss in performance as compared to the baseline. As such, we ascertain that gating on weights is more effective than gating on the output representations. This verifies that our hypernetwork-based approach is indeed effective as opposed to simple task-conditioned output gating.

heading: 3.1.4 EFFECT OF GRID SIZE ON PERFORMANCE

text:
We investigate the effect of Grid size (fan-in and fan-out) of our proposed HyperGrid method. The purpose of this experiment is to discover how fine-grained or coarse-grained the hypernetwork should be. Notably, smaller values of dr, dc signify a more coarse-grained control of the Transformer weights.
Setup We searched dr (fan-in) and dc (fan-out) in the ranges of {4, 8, 16, 32, 128, 256} and {8, 16, 32, 128, 256} respectively and report the results on GLUE + SuperGLUE (macro-average) by varying a single value. When varying dr, we took the average of all dc runs and plot the max, mean and min. Likewise, when varying dc, we took the average of all dr runs and plot max, mean and average. We report scores across the L2, LG, and GL variants of HyperGrid.
Findings pertaining to Grid Size Figures 3 to Figures 8 illustrates performance across varied grid sizes. We observe that a clear trend exists. For most settings, a small fan-out (dc) works well (e.g., 32) as noted by many spikes around this region. For fan-in (dr) a smaller value also works well. However, performance gets better at higher fan-out dc values again (e.g., > 128). Trends are quite consistent across all three variations that we considered. These results suggest that a more coarse grid may be more effective, as the regions within the grid become larger.
1Due to the relative increased cost of searching large models, we performed a sparingly low number of ablations on large models.
Figure 4: fan-in on LG setting. 8 16 32 64 128 Dimensions (Fanin)
4816 32 64 128 256 Dimensions (Fanout)
78.5
79.0
79.5
80.0
80.5
GL UE
+ S
up er
GL UE
S co
re
mean max min
Figure 6: fan-out on L2 setting. 4816 32 64 128 256 Dimensions (Fanout)
78.6
78.8
79.0
79.2
79.4
79.6
79.8
80.0
80.2
GL UE
+ S
up er
GL UE
S co
re
mean max min
Figure 7: fan-out on LG setting. 4816 32 64 128 256 Dimensions (Fanout)
78.8
79.0
79.2
79.4
79.6
79.8
80.0
80.2
GL UE
+ S
up er
GL UE
S co
re
mean max min
Figure 8: fan-out on GL setting.

heading: 3.1.5 PERFORMANCE ON TEST SET

text:
For our final runs, we submit our model predictions to the GLUE and SuperGLUE test servers.
Setup We run experiments with a 3B and 11B HyperGrid Transformer model in multi-task2 setup (GLUE + SuperGLUE). We initialize with the T5 pre-trained checkpoints. Since this is a relatively expensive run, we only train the single model HyperGrid once using a 32 × 128 grid with the LG (local-global) setting. For GLUE, we compare against baselines reported in (Clark et al., 2020) which includes models such as BERT (Devlin et al., 2018), ALBERT Lan et al. (2019), RoBERTa (Liu et al., 2019b), and XLNet (Yang et al., 2019). Note that all these models are ensembles and heavily rely on task-specific fine-tunining strategies. More details can be found in the supplementary material.
Results on Test Set We find that our single model approach can achieve highly competitive results on both GLUE and SuperGLUE. Our model achieves a strong performance of 88.9 on SuperGLUE, matching the reported T5 results while having 16 times fewer total parameters. On GLUE, the performance gap is also small, almost matching the T5 model at 89.4 versus 89.7. The gap on the base model remains similar at 88.2 versus 88.5. On SuperGLUE, our 3B model achieves 84.7, a respectable score that matches the performance of RoBERTa ensembles fine-tuned individually with task specific tricks (Liu et al., 2019b).

heading: 4 RELATED WORK

text:
Multi-task learning (MTL) (Caruana, 1997) is a long standing research problem. Learning a single unified model that does well on multiple tasks is an uphill battle given well-known problems such as catastrophic forgetting (Kirkpatrick et al., 2017). As such, learning a large number of tasks with a single set of model parameters is an extremely challenging endeavour. Moreover, the disproportionate amount of data per task is also potentially problematic (Lee et al., 2017; Pfeiffer et al., 2020), which results in models overfitting on high resource tasks but underfitting on low resource tasks.
Early work in multi-task NLP typically considered a hierarchical taxonomy of tasks (Hashimoto et al., 2016) where a clear hierarchy of tasks exist, such as POS → Chunking → entailment. The Joint Many-Task (JMT) model explores an incremental and hierarchical paradigm for building multitask NLP models. Similarly, (Sanh et al., 2019) proposed a hierarchical multi-task model based on the intuition of low-level and high-level tasks. Another line of recent work explores casting all tasks into a form of question answering problem (McCann et al., 2018) and using an interpolated pointer-generator (See et al., 2017) mechanism for generating ‘answers’.
Exploiting task relatedness as a means for improved model quality has been frequently explored. In relatively recent work, (Liu et al., 2019a) proposed MTDNN, a multi-task deep neural network that shares parameters between several NLP tasks. The model achieves strong performance on the GLUE benchmark. However, MTDNN simply leverages MTL as a form of pretraining and uses task-specific models for final evaluation. The recent T5 (Text-to-Text Transfer Transformers) model (Raffel et al., 2019) frames all NLP problems as a Seq2Seq (Sutskever et al., 2014) problem. However, the best results are again obtained by task-specific fine-tuning.
Orthogonal to other research efforts, (Clark et al., 2019b) proposed Born Again Neural Networks (BAM), a clever way to obtain a single multi-task network by knowledge distillation. (Stickland and Murray, 2019) proposed Projected Attention Layers for task-specific fine-tuning of BERT (Devlin et al., 2018). (Zaremoodi et al., 2018) proposed Adaptive Knowledge Sharing3 for low-resource neural machine translation. Our work is related to the literature surrounding hypernetworks (Ha et al., 2016) which have been found to useful in areas such as continual learning (von Oswald et al., 2019). Learning task-adaptive parameters to avoid catastrophic forgetting has also been a go-to strategy for continual learning (Yoon et al., 2019). Outside of the NLP domain, flexible parameter sharing approaches are also dominant strategies for learning multi-task models (Ma et al., 2018; 2019).
2Since we did not co-train with the WNLI dataset due to issues stated in (Raffel et al., 2019), we simply report T5 results on WNLI. To be fair, we ignore WNLI parameter counts for all baseline models.
3The authors of (Raffel et al., 2019) explored this approach but did not find it to be satisfactory.

heading: 5 CONCLUSION

text: We proposed Hypergrid Transformers, a new Transformer architecture that leverages Grid-wise Decomposable Hyper Projections (HyperGrid), a hypernetwork-based projection layer for task conditioned weight generation. We learn and fit all GLUE and SuperGLUE tasks within the same set of model parameters and achieve competitive results to the same state-of-the-art model that is specially and individually fine-tuned on each and every task. On GLUE/SuperGLUE, this efficient single-model method results in 16x fewer parameters.

heading: 6 SUPPLEMENTARY MATERIAL

heading: 6.1 DATASETS

heading: 6.1.1 GLUE

text: The datasets in GLUE are CoLA (Corpus of Linguistic Acceptability) (Warstadt et al., 2018), Sentiment Treebank SST-2 Socher et al. (2013), Microsoft Research Paraphrase Corpus (MRPC) (Dolan and Brockett, 2005), QQP (Quora Question Pairs) (Iyer et al., 2017), Semantic Textual Similarity Benchmark (STSB) (Cer et al., 2017), MNLI (Multi-Genre Natural Language Inference) Williams et al. (2018), QNLI (Rajpurkar et al., 2016), RTE (Dagan et al., 2005), Winograd Schema Challenge WNLI (Levesque et al., 2012). More details can be found at https: //github.com/tensorflow/datasets/blob/master/docs/catalog/glue.md.

heading: 6.1.2 SUPERGLUE

text: The datasets in SuperGLUE (Wang et al., 2019) are BoolQ (Boolean Questions) (Clark et al., 2019a), CB (Commitment Bank) (De Marneff et al., 2019), CoPA (Roemmele et al., 2011) (Choice of Plausible Alternatives), MultiRC (Multi-Sentence Reading Comprehension Dataset) (Khashabi et al., 2018), Record (Reading Comprehension with Commonsense Reasoning) (Zhang et al., 2018), RTE (Recognizing Textual Entailment) (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), Word-in-Context (WiC) (Pilehvar and os’e Camacho-Collados, 2018), and WSC (Winograd Schema Challenge) (Levesque et al., 2012). We use Tensorflow datasets for loading and preprocessing these datasets. More details can be found at https://github.com/ tensorflow/datasets/blob/master/docs/catalog/super_glue.md.

heading: 6.2 EXPERIMENT SETTINGS

text: This section describes most of the hyperparameter settings for our experiments.

heading: 6.3 DATASETS AND EXPERIMENTAL SETUP

text:
Our experiments are built upon the existing state-of-the-art model, T5. We run most of our experiments using the base T5 setting, which is comprised of 220M parameters. We fine-tune for a maximum of 100K steps. We initialize our models with the released pretrained checkpoints4. Our implementation is in Mesh Tensorflow (Shazeer et al., 2018). We consider the following setups for the baseline T5 model. First, we compare with the T5 results reported in the original5 paper (Raffel et al., 2019). These results are denoted with T5†. Second, we compare with T5 (PTFT), which stands for pretrain-finetune. In this setup, we fine-tune a T5 model for each task individually following common practice. Finally, we compare with T5 (MTL) which is a fair comparison of T5 without HyperGrid. In this setting, T5 is co-trained and results are reported from a single model checkpoint selected from the best overall GLUE dev score. Note that in the MTL setting, we co-train GLUE and SuperGLUE within the same model.
Experiments for Base Models For all experiments with base models, we train models for 100K steps with a batch size of 128. We use the en mix mixture which samples each task proportionately to the number of examples in the dataset. Learning rate is a constant 0.001 with Adafactor (Shazeer and Stern, 2018). All results for baselines are reported with scores at the last checkpoint. During fine-tuning, the embeddings are not fine-tuned. Experiments are run with 16 TPU V3 chips and are typically completed in about 8 to 10 hours.
Experiments with Large Models We increased the search for large models to 200K steps pick the best checkpoint for all models based on the best GLUE score. Experiment and hyperparameter settings remain identical although we use 64 TPU V3 chips for finetuning which typically take about 12 hours to complete.
4https://github.com/google-research/text-to-text-transfer-transformer. 5This model is not directly comparable as they used less pretraining steps. No dev score results on a
comparable setup is reported. We report this score for the sake of completeness.
Experiments with 3B and 11B Models For the large models, we only use 1 − 2 HyperGrid configurations 32x128 or 32x256 in LGmode for finetuning the model. We submit each model only once to the leaderboard6. Finetuning hyperparameters remain identical. We pick a single checkpoint based on the best GLUE score. Finetuning for the 3B model is using 64 TPU V3 chips and the 11B model is fine-tuned with 128 TPU V3 chips.

heading: 6.4 COMPARING WITH OUTPUT GATING

text:
One of the model architecture variants we compared with is Output Gating. It can be formulated as:
Y = max(Wx+ b, 0) (σ(UX)1>) (8)
Comparing to the HyperGrid, which gates the weights in the Relu layer, output gating directly gates the Relu layer outputs. We can apply either the basic projection method (Equation (2)), or the grid-wise projection method with block-wise projection on layer outputs.
There are two key differences: (1) Output Gating applies sigmoid gating on Relu layer outputs, while HyperGrid applies sigmoid gating on weights before the Relu function. Output gating is similar to the Mixture-of-Expert architecture while concatenating the expert outputs. (2) Based on this formulation, the full grid-based projection cannot be applied to output gating.

heading: 6.5 FURTHER ARCHITECTURAL ABLATIONS

text:
We include more architectural ablations to supplement the results and findings of the paper.
Ablation Architectures We run experiments for 6 different ablations. The first (1) - (4) is concerning with where to apply HyperGrid. (1) applies on the entire network (all QKV + both FFNs), (2) applies it to both positional FFNs, (3) applies it on the 1st FFN only and (4) applies it on the 2nd FFN only. For (5) and (6), we evaluate on different weighting schemes for HyperGrid. (5) projects HyperGrid to a scalar value by pooling (in essense, instead of upsampling the grid, we downsample the grid into a scalar value). As fo (6), this is the setting where the the row and column size of the HyperGrid is essentially the same as the FFN.
Results Our ablation studies show that the best place to place the FFN is only at the 2nd FFN. Using HyperGrid on all the layers, apart from slowing down the network, would also degrade performance. This can be observed because the difference between (1) and (2) is the addition of QKV HyperGrids to the model. In (5) and (6), we also note that Scalar HyperGrid performs poorly and worse than the Baseline model. Finally, the Max-Grid HyperGrid performs reasonably well. However, this incurs a huge cost because the hyper network is now larger (as composed to learning small grids and upsampling).
6Discounting submissions that turn out to be incomplete or error submissions.

title: HYPERGRID TRANSFORMERS: TOWARDS A SINGLE MODEL FOR MULTIPLE TASKS


summaries: This paper proposed a novel adaptive data augmentation algorithm that produces random perturbations on the training dataset to train an imitation learning-based self-driving network. It starts with a sensitivity analysis of network performance under different types and levels of perturbations. And a novel automated perturbed training dataset selection mechanism is then proposed to improve the performance. Validation has been conducted over simulated data with both seen and unseen perturbation types.

summaries: This work proposes a new method to improve the generalization of ML models for the task of vehicle steering using a hybrid of data augmentation and adversarial examples. In a nutshell, the proposed method attempts to increase the accuracy of the model by dynamically adding a selection of candidate datasets during training. Each of these “candidates” is created offline applying a transform (e.g. blur, distortion, and changes in color representation) to the original (base) dataset. During training, the method chooses among the K transformed-datasets those who minimize the mean validation accuracy and based on this selection the steering model is retrained. The approach is evaluated on a driving dataset.

summaries: This paper presents an algorithm to improve the model generalization of the task of "learning to steer". First, the sensitivity of a baseline learning algorithm to degraded images in varying qualities caused by different factors is carried out. Some empirical insights are gained. Then, a new training algorithm is proposed to solve a min-max optimization problem, where the most difficult datasets are chosen and used for training at each iteration. Experiments are conducted to validate the effectiveness of the proposed method.

abstractText: To ensure the wide adoption and safety of autonomous driving, vehicles need to be able to drive under various lighting, weather, and visibility conditions in different environments. These external and environmental factors, along with internal factors associated with sensors, can pose significant challenges to perceptual data processing, hence affecting the decision-making of the vehicle. In this work, we address this critical issue by analyzing the sensitivity of the learning algorithm with respect to varying quality in the image input for autonomous driving. Using the results of sensitivity analysis, we further propose an algorithm to improve the overall performance of the task of “learning to steer”. The results show that our approach is able to enhance the learning outcomes up to 48%. A comparative study drawn between our approach and other related techniques, such as data augmentation and adversarial training, confirms the effectiveness of our algorithm as a way to improve the robustness and generalization of neural network training for self-driving cars.

authors:
Isabelle Bégin
Frank Ferrie

title: Blind super-resolution using a learning-based approach

venue: In Proceedings of the 17th International Conference on Pattern Recognition (ICPR),

authors:
Mariusz Bojarski
Davide Del Testa
Daniel Dworakowski
Bernhard Firner
Beat Flepp
Prasoon Goyal
Lawrence D Jackel
Mathew Monfort
Urs Muller
Jiakai Zhang

title: End to end learning for self-driving cars

venue: arXiv preprint arXiv:1604.07316,

authors:
Holger Caesar
Varun Bankiti
Alex H. Lang
Sourabh Vora
Venice Erin Liong
Qiang Xu
Anush Krishnan
Yu Pan
Giancarlo Baldan
Oscar Beijbom

title: nuscenes: A multimodal dataset for autonomous driving

venue: arXiv preprint arXiv:1903.11027,

authors:
Qianwen Chao
Huikun Bi
Weizi Li
Tianlu Mao
Zhaoqi Wang
Ming C. Lin
Zhigang Deng

title: A survey on visual traffic simulation: Models, evaluations, and applications in autonomous driving

venue: Computer Graphics Forum,

title: A collection of labeled car driving datasets, https://github.com/sullychen/driving-datasets, 2018

authors:
Marius Cordts
Mohamed Omran
Sebastian Ramos
Timo Rehfeld
Markus Enzweiler
Rodrigo Benenson
Uwe Franke
Stefan Roth
Bernt Schiele

title: The cityscapes dataset for semantic urban scene understanding

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Ekin D Cubuk
Barret Zoph
Dandelion Mane
Vijay Vasudevan
Quoc V Le

title: Autoaugment: Learning augmentation policies from data

venue: arXiv preprint arXiv:1805.09501,

authors:
Samuel Dodge
Lina Karam

title: A study and comparison of human and deep learning recognition performance under visual distortions

venue: 26th international conference on computer communication and networks (ICCCN),

title: Quality resilient deep neural networks

venue: arXiv preprint arXiv:1703.08119,

authors:
Alexey Dosovitskiy
German Ros
Felipe Codevilla
Antonio Lopez
Vladlen Koltun

title: CARLA: An open urban driving simulator

venue: In Proceedings of the 1st Annual Conference on Robot Learning,

authors:
Xiang Gao
Ripon K Saha
Mukul R Prasad
Abhik Roychoudhury

title: Fuzz testing based data augmentation to improve robustness of deep neural networks

venue: In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering,

authors:
Andreas Geiger
Philip Lenz
Christoph Stiller
Raquel Urtasun

title: Vision meets robotics: The kitti dataset

venue: International Journal of Robotics Research (IJRR),

authors:
S. Ghosh
R. Shet
P. Amon
A. Hutter
A. Kaup

title: Robustness of deep convolutional neural networks for image degradations

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Chengyue Gong
Tongzheng Ren
Mao Ye
Qiang Liu

title: MaxUp: A Simple Way to Improve Generalization of Neural Network Training

venue: arXiv e-prints, art

authors:
Dan Hendrycks
Thomas Dietterich

title: Benchmarking neural network robustness to common corruptions and perturbations

venue: arXiv preprint arXiv:1903.12261,

authors:
Dan Hendrycks
Mantas Mazeika
Saurav Kadavath
Dawn Song

title: Using self-supervised learning can improve model robustness and uncertainty

venue: Advances in Neural Information Processing Systems

authors:
Dan Hendrycks
Norman Mu
Ekin Dogus Cubuk
Barret Zoph
Justin Gilmer
Balaji Lakshminarayanan

title: Augmix: A simple data processing method to improve robustness and uncertainty

venue: In ICLR,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Weizi Li
David Wolinski
Ming C. Lin

title: ADAPS: Autonomous driving via principled simulations

venue: In IEEE International Conference on Robotics and Automation (ICRA),

authors:
Will Maddern
Geoff Pascoe
Chris Linegar
Paul Newman

title: Year, 1000km: The Oxford RobotCar Dataset

venue: The International Journal of Robotics Research (IJRR),

authors:
Aleksander Madry
Aleksandar Makelov
Ludwig Schmidt
Dimitris Tsipras
Adrian Vladu

title: Towards deep learning models resistant to adversarial attacks

authors:
Camilo Pestana
Naveed Akhtar
Wei Liu
David Glance
Ajmal Mian

title: Adversarial Perturbations Prevail in the Y-Channel of the YCbCr Color Space

authors:
Matthew Pitropov
Danson Garcia
Jason Rebello
Michael Smart
Carlos Wang
Krzysztof Czarnecki
Steven Waslander

title: Canadian adverse driving conditions dataset

venue: arXiv preprint arXiv:2001.10117,

authors:
Vasili Ramanishka
Yi-Ting Chen
Teruhisa Misu
Kate Saenko

title: Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Mengye Ren
Wenyuan Zeng
Bin Yang
Raquel Urtasun

title: Learning to reweight examples for robust deep learning

venue: arXiv preprint arXiv:1803.09050,

authors:
Eder Santana
George Hotz

title: Learning a driving simulator

venue: arXiv preprint arXiv:1608.01230,

authors:
Ali Shafahi
Mahyar Najibi
Mohammad Amin Ghiasi
Zheng Xu
John Dickerson
Christoph Studer
Larry S Davis
Gavin Taylor
Tom Goldstein

title: Adversarial training for free

venue: NeurIPS,

authors:
Manli Shu
Zuxuan Wu
Micah Goldblum
Tom Goldstein

title: Preparing for the worst: Making networks less brittle with adversarial batch normalization

venue: arXiv preprint arXiv:2009.08965,

authors:
Pei Sun
Henrik Kretzschmar
Xerxes Dotiwalla
Aurelien Chouard
Vijaysai Patnaik
Paul Tsui
James Guo
Yin Zhou
Yuning Chai
Benjamin Caine

title: Scalability in perception for autonomous driving: Waymo open dataset

venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,

authors:
Yuchi Tian
Kexin Pei
Suman Jana
Baishakhi Ray

title: Deeptest: Automated testing of deep-neuralnetwork-driven autonomous cars

venue: In Proceedings of the 40th international conference on software engineering,

authors:
Toan Tran
Trung Pham
Gustavo Carneiro
Lyle Palmer
Ian Reid

title: A bayesian data augmentation approach for learning deep models

authors:
Frederick Tung
Jianhui Chen
Lili Meng
James J Little

title: The raincouver scene parsing benchmark for self-driving in adverse weather and at night

venue: IEEE Robotics and Automation Letters,

authors:
Igor Vasiljevic
Ayan Chakrabarti
Gregory Shakhnarovich

title: Examining the impact of blur on recognition by convolutional networks

venue: arXiv preprint arXiv:1611.05760,

authors:
Zhe Wu
Zuxuan Wu
Bharat Singh
Larry Davis

title: Recognizing instagram filtered images with feature de-stylization

venue: Proceedings of the AAAI Conference on Artificial Intelligence,

authors:
Cihang Xie
Mingxing Tan
Boqing Gong
Jiang Wang
Alan L. Yuille
Quoc V. Le

title: Adversarial examples improve image recognition

authors:
Mengshi Zhang
Yuqun Zhang
Lingming Zhang
Cong Liu
Sarfraz Khurshid

title: Deeproad: Ganbased metamorphic testing and input validation framework for autonomous driving systems

venue: In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE),

title: A flexible new technique for camera calibration

venue: IEEE Transactions on pattern analysis and machine intelligence,

authors:
Yiren Zhou
Sibo Song
Ngai-Man Cheung

title: On classification of distorted images with deep convolutional neural networks

heading: 1 INTRODUCTION

text:
Autonomous driving is a complex task that requires many software and hardware components to operate reliably under highly disparate and often unpredictable conditions. While on the road, vehicles are likely to experience day and night, clear and foggy conditions, sunny and rainy days, as well as bright cityscapes and dark tunnels. All these external factors can lead to quality variations in image data, which are then served as inputs to autonomous systems. Adding to the complexity are internal factors of the camera (e.g., those associated with hardware), which can also result in varying-quality images as input to learning algorithms. One can harden machine learning systems to these degradations by simulating them at train time (Chao et al., 2019). However, there currently lacks algorithmic tools for analyzing the sensitivity of real-world neural network performance on the properties of simulated training images and, more importantly, a mechanism to leverage such a sensitivity analysis for improving learning outcomes. In this work, we quantify the influence of varying-quality images on the task of “learning to steer” and provide a systematic approach to improve the performance of the learning algorithm based on quantitative analysis.
Image degradations can often be simulated at training time by adjusting a combinations of image quality attributes, including blur, noise, distortion, color representations (such as RGB or CMY) hues, saturation, and intensity values (HSV), etc. However, identifying the correct combination of simulated attribute parameters to obtain optimal performance on real data during training is a difficult—if not impossible—task, requiring exploration of a high dimensional parameterized space.
The first goal of this work is to design a systematic method for studying, predicting, and quantifying the impact of an image degradation on system performance after training. We do this by measuring the similarity between real-world datasets and simulated datasets with degradations using the well known Fréchet Inception Distance (FID). We find that the FID between simulated and real datasets is a good predictor of whether training on simulated data will produce good performance in the real world. We also use FID between different simulated datasets as a unified metric to parameterize the severity of various image quality degradations on the same FID-based scale (see Section 3).
Our second goal is to borrow concepts from the adversarial training literature (Madry et al., 2018; Shafahi et al., 2019; Xie et al., 2020) to build a scalable training scheme to improve the robustness of autonomous driving systems against multi-faceted image degradations, while increasing the overall accuracy of the steering task for self-driving cars. Our proposed method builds a dataset of adversarially degraded images by apply evolutionary optimization within the space of possible degredations during training. The method begins by training on a combination of real and simulated/degraded data using arbitrary degradation parameters. On each training iteration, the parameters are updated to generate a new degradation combination so that system performance is (approximately) minimized. The network is then trained on these adversarially simulated images to promote robustness. Our proposed algorithm speeds up the combinatorial degradation updates by discretizing the search space using our FID-based parameterization. See details in Section 4.
Experiments show that the algorithm improves the task performance of “learning to steer” up to 48% in mean accuracy over strong baselines. We compare our approach with other related techniques, such as data augmentation and adversarial training, and the results show that our method consistently achieves higher performance. The method also improves the performance on datasets contaminated with complex combinations of perturbations (up to 33%), and additionally boosts performance on degradations that are not seen during training, like simulated snow, fog, and frost (up to 29%), as discussed in Section 4.2.
For evaluation, we propose a more complete robustness evaluation standard under 4 different scenarios: clean data, single perturbation data, multiple perturbation data, and unseen perturbation data. While previous works usually use one or two scenarios, our work is among the first to test results under all 4 meaningful scenarios that are important for evaluating robustness of algorithms.
We also plan to release “learn to steer under perturbations” datasets for benchmarking. These datasets will contain a base dataset; simulated adversarial datasets with multiple levels of image degradation due to either single or multiple image quality attributes; and simulated adversarial datasets with five levels of combinatorial perturbations due to a different set of unseen factors on images with corruptions in ImageNet-C (Hendrycks & Dietterich, 2019), totaling about 1.2M images and 120 datasets in all. (See Section 4.5.)

heading: 2 RELATED WORK

text:
The influence of the noise and distortion effects on real images on learning tasks has been explored in the last five years. For example, researchers have examined the impact of optical blur on convolutional neural networks and present a fine-tuning method for recovering lost accuracy using blurred images (Vasiljevic et al., 2016). This fine-tuning method resolved lost accuracy when images were distorted instead of blurred (Zhou et al., 2017). While these fine tuning methods are promising, (Dodge & Karam, 2017b) find that tuning to one type of image quality reduction effect would cause poor generalization to other types of quality reduction effects. Comparison on image classification performance between deep neural networks and humans has been done by (Dodge & Karam, 2017a), and found to be similar with images of good quality. However, Deep Neural Networks struggle significantly more than humans on low-quality, distorted or noisy images. Color spaces have also been shown to negatively affect performance of learning models. It was observed that perturbations affect learning performance more in the Y channel of YCbCr color space that is analogous to the YUV color space (Pestana et al., 2020). Another work (Wu et al., 2020) studies the effect of Instagram filters, which mostly change the coloring of an image, on learning tasks. In this work, we study nine common factors characterizing image quality, i.e., blur, noise, distortion, three-color (Red-Green-Blue or RGB) channels, and hues, saturation, and intensity values (HSV). Not only does our study analyze a more comprehensive set of image attributes that could influence the learning-based steering task, but we also parameterize these nine factors into one integrated image-quality space using the Fréchet Inception Distance as the unifiying metric, thus enabling us to conduct sensitivity analysis.
Researchers have also explored how to improve the robustness of learning algorithms under various image quality degradations. One recent work (Tran et al., 2017) provides a novel Bayesian formulation for data augmentation. Cubuk et al. (2018) proposes an approach to automatically search for improved data augmentation policies. Ghosh et al. (2018) performs analyses on the performance of Convolutional Neural Networks on quality degradations due to common causes like compression loss, noise, blur, and contrast, etc. and introduces a method to improve the learning outcomes using
a master-slave architecture. Another work (Hendrycks et al., 2019) shows that self-supervision techniques can be used to improve model robustness that exceeds the performance of fully-supervised methods. A new method, also by Hendrycks et al., tackles model robustness from a data augmentation perspective, where compositions of transformations are used to create new data that is visually and semantically close to the original dataset (Hendrycks et al., 2020). Gao et al. (2020) proposes a technique that re-purposes software testing methods, specifically mutation-based fuzzing, to augment the training data of DNNs, with the objective of enhancing their robustness. A more recent work (Gong et al., 2020) introduces a solution to improving model generalization by first augmenting training dataset with random perturbations, and then minimizing worst-case loss over the augmented data. Our work differs from these studies in several regards. First, we simulate adversarial conditions of image factors instead of commonplace image conditions. Second, we conduct a systematic sensitivity analysis for preparing datasets that are representative of image degradations due to multiple factors at different levels. Third, our algorithm for improving the generalization of the learning task can work with the discretized parameter space while generalize well to the continuous parameter space. Another advantage of our approach is that we can augment the training dataset without the derivatives of the factor parameters, which may not exist or are difficult to compute. These differences distinguish our approach from the prior arts and other techniques for improving model robustness, such as data augmentation and adversarial training.

heading: 3 ANALYZING IMAGE QUALITY FACTORS

heading: 3.1 EXPERIMENT SETUP

text:
Base Dataset. We build our base dataset using the real-world driving dataset (Chen, 2018), which contains about 63,000 images at the resolution 455 × 256. There are several good autonomous driving datasets, but not all of them are suitable for the end-to-end learning to steer task. Some datasets (e.g., Waymo (Sun et al., 2020), KITTI (Geiger et al., 2013), Cityscapes (Cordts et al., 2016), OxfordRoboCar (Maddern et al., 2017), Raincouver (Tung et al., 2017), etc) do not contain steering angle labels. Some other datasets contain steering angle labels (e.g., Audi (Geyer et al., 2020), Honda (Ramanishka et al., 2018), nuScenes (Caesar et al., 2019), Ford AV (Agarwal et al., 2020), Canadian Adverse Driving Conditions (Pitropov et al., 2020), etc), but they are not designed for end-to-end learning to steer task specifically, the driving environment may not be suitable for our end-to-end learning to steer task. For example, driving at intersections (lead to confusion for steering), or only a very small subset of data containing image data from turning on a curved road without branches. There are also several well-known simulators like CARLA (Dosovitskiy et al., 2017) that can generate synthetic dataset, but our work focuses on the real-world driving using real images. Chen’s dataset is collected specifically for steering task and has a relatively long continuous driving image sequence on a road without branches and has relatively high turning cases.
Image quality factors. We study nine image attributes in this work, i.e., blur, noise, distortion, three-color (Red-Green-Blue or RGB) channels, and hues, saturation, and intensity values (HSV). Blur, noise, and distortion are among the most commonly used and directly affect image quality. R, G, B, H, S, V channels are chosen because both RGB and HSV spaces are frequently used to represent image color spaces: RGB represent three basic color values of an image, while HSV represent three common metrics of an image. Other color spaces such as HSL or YUV have similar properties, hence are excluded from this study to avoid redundant analysis.
Learning Algorithm. We choose the model by Bojarski et al. (2016) as the learning algorithm. The model contains five convolutional layers followed by three dense layers. We select this model, because it has been used to steer an AV successfully in both real world (Bojarski et al., 2016) and virtual world (Li et al., 2019).
Evaluation Metric. We use mean accuracy (MA) to evaluate our regression task since it can represent the overall performance under different thresholds. We first define the accuracy with respect to a particular threshold τ as accτ = count(|vpredicted − vactual| < τ)/n, where n denotes the number of test cases; vpredicted and vactual indicate the predicted and ground-truth value, respectively. Then, MA is computed as ∑ τ accτ∈T /|T |, where T = {1.5, 3.0, 7.5, 15, 30, 75} contains empirically selected thresholds of steering angles. Based on MA, we use the maximum MA improvement (denoted as MMAI), the average MA improvement (denoted as AMAI), and mean Corruption Errors (mCE) (Hendrycks & Dietterich, 2019) as the evaluation metrics.

heading: 3.2 SENSITIVITY ANALYSIS

text:
In this section, we introduce how we conduct sensitivity analysis on the nine image factors, i.e., blur, noise, distortion and R, G, B, H, S, V channels, and how we use the sensitivity analysis results to prepare datasets that are representative of image qualities at different levels for further model training.
For simulating blur, noise, and distortion effects, we use Gaussian blur (Bégin & Ferrie, 2004), additive white Gaussian noise (AWGN), and radial distortion (Zhang, 2000), respectively. For representing channel-level perturbations, we use a linear combination model: denote the value range of one channel as [a, b], in the darker direction, we set vnew = αa+(1−α)v; in the lighter direction, we set vnew = αb + (1 − α)v. The default values are aC = 0 and bC = 255, where C represents one of the 6 channels. To prevent an image being completely dark, we set aV = 10 and bH = 179.
We adopt the Fréchet Inception Distance (FID) as a unified metric for our sensitivity analysis, instead of using the parameter values of each image factor for three reasons. First, FID can better capture different levels of image qualities than the parameter values of each factor, because the correspondence between the factor parameter values and image quality is not straightforward. For example, if we uniformly sample the parameter space of distortion, most parameter values will result in images with qualities similar to the level 4 and level 5 images of distortion datasets shown in Fig. 1. This is not ideal since we would like to have representative datasets that enable us to study the sensitivity of a learning task with respect to the gradual degradation of each image attribute. Second, by adopting FID as a single metric, we can map the parameter spaces of all factors into one space that better facilitates both the dependency trend and sensitivity analyses. Lastly, FID can naturally serve as a metric to evaluate the distance between two image datasets. It takes image pixels, image features, and correlations between images into account – these are all meaningful factors to interpret the performance of a learning-based task.
Starting from empirically-selected parameters of each factor, we ‘simulate’ perturbed datasets and compute their corresponding mean accuracy (MA) using the trained model on the base dataset (Chen, 2018). We then map these correspondences between MAs and the parameter values into the FID space using the FID between each perturbed dataset and the base dataset. Characterizing the influence of each image quality attribute on the task performance, as captured in MA-FID space, we can then re-define parameter value range of each quality factor so that the newly generated perturbed datasets will have similar ‘distance gaps’ in image quality, as defined by FID. Examples of the resulting images are shown in Fig. 1. More examples can be found in Appendix A.2. The corresponding parameters of the final perturbed datasets can be found in Appendix A.3. The notation for these datasets is summarized as follows: R: the base dataset (Chen, 2018); B1, B2, B3, B4, B5: five levels of blur; N1, N2, N3, N4, N5: five levels of noise; D1, D2, D3, D4, D5: five levels of distortion; RD1, RD2, RD3, RD4, RD5: five dark levels of the red (R) channel; RL1, RL2, RL3, RL4, RL5: five light levels of the red (R) channel; the dark and light levels of the G, B, H, S, V channels have similar naming conventions, i.e., channel label + D/L (dark/light) + level (from 1 to 5).
The analysis results using the final perturbed datasets at different levels of several sample factors are shown in Fig. 2. The complete numeric results and the figure for all factors can be found in Appendix A.6. Note that all factors have some influence on the learning-based steering task. Their
(negative) impact increases, as the perturbation level increases. The performance loss due to image quality degradation can be as high as more than 50% (see lighter Lv5 of the G channel), which can introduce very high risks in autonomous driving.
The final MA differences in the FID space are visualized in Fig. 3 (for blur, noise, distortion, G channel and V channel, see entire figure in Appendix.A.4). Since FID aligns different factors into the same space, we can compare the performance loss due to each factor at various levels. We first observe that the learning-based steering task is more sensitive to the channel-level perturbations (i.e., from R, G, B, H, S, V channels) than the image-level perturbations (i.e., from blur, noise, distortion effects). Second, within the quality factors we analyze, the learning task is least sensitive to blur and noise but most sensitive to the intensity value in V channel. Lastly, for the same color channel, darker and lighter levels appear to have different MA trends, i.e., different accuracy loss at the same FID values can vary.
Compared with other analysis works on learning to steer task, e.g. Tian et al. (2018) and Zhang et al. (2018), our method is the first work that transfers perturbations from multiple parameter spaces into one unified FID space to enable the cross-factor comparison, e.g., to show that our task is more sensitive to V channel perturbation of HSV space than other perturbations.

heading: 4 IMPROVING GENERALIZATION OF LEARNING-BASED STEERING

heading: 4.1 METHODOLOGY

text:
In this section, we introduce our method to improve the generalization of the learning-based steering for autonomous driving using the results from the sensitivity analysis.
Our algorithm uses an iterative min-max training process: at each iteration, we first choose a dataset among all datasets of one factor that can minimize mean accuracy (MA). Then, we combine such datasets of all factors with the base dataset to re-train our model while maximizing MA. The algorithm stops when a pre-specified number of iterations is reached or the MA loss is below a certain threshold. The design rationale of our architecture resembles adversarial training: we train the
model to maximize accuracy using the original dataset plus the perturbed datasets with the minimum accuracy in order to improve the robustness of the model. The loss function is the following:
min p max θ MA(θ, Up), (1)
where p represents a union of the parameter levels of all analyzed image factors; θ denotes the model parameters; Up is the training dataset; and MA() is the function computing MA. The overall description of our method is shown in Algorithm 1. See the overall pipeline figure in Appendix A.1.
Our method has several advantages: 1) the training data is augmented without re-training the model, thus achieving efficiency; 2) it provides the flexibility to generate datasets at various discretized levels of the factor parameters; 3) it does not require the derivatives of factor parameters, unlike other frameworks that optimize factor parameters in the continuous space where derivatives are difficult to compute; and 4) it can generalize over not only the parameters of a single perturbation factor, but also the composition of the parameters of multiple factors.
Algorithm 1: Improve the generalization of learning-based steering Result: a trained model parameterized by θ Pre-processing: Conduct sensitivity analysis and discretize the parameters of n factors into their corresponding li, i = 1, . . . , n levels; Generate new datasets for each factor with the discretized values from base dataset B: {Di,j}, i = 1, 2, .., n, j = 1, 2, ..li; Initialization: Set t = 0, and initialize the maximum of iterations T and the number of epochs k; Initialize model parameters θ(0); while t ≤ T do
For each factor, select the dataset Di,pi that can minimize the validation mean accuracy, where pi = argminjMA(θ
(t), Dij); Merge all selected datasets Up = ( ⋃n i=1Di,pi) ⋃ B; Train the network for k epochs and update the parameters θ(t+1) = train(θ(t), Up, k) to maximize MA(θ(t+1), Up); Break if stop conditions are met; Otherwise set t = t+ 1;
end

heading: 4.2 EXPERIMENTS AND RESULTS

text:
All experiments are conducted using Intel(R) Xeon(TM) W-2123 CPU, Nvidia GTX 1080 GPU, and 32G RAM. We use the Adam optimizer (Kingma & Ba, 2014) with learning rate 0.0001 and batch size 128 for training. The maximum number of epochs is 1000. Images in the base dataset are sampled from videos at 30 frames per second (FPS). For efficiency, we adopt a similar approach as in Bojarski et al. (2016) by further downsampling the dataset to 5 FPS to reduce similarities between adjacent frames. The resulting dataset contains approximately 10,000 images. We randomly split the dataset with about 20:1:2 as training/validation/test data.
We compare our method with three methods: a baseline method (Bojarski et al., 2016), an adversarial training method (Shu et al., 2020), and a naïve data augmentation method. For the naïve data augmentation method, we simply merge all perturbed datasets together to train the model. We test the performance of all methods in four scenarios with increasing complexity:
• Scenario 1: Clean data. Test on the base clean dataset only. • Scenario 2: Single Perturbation. Test on datasets of each factor at its discretized levels.
Specifically, we use five levels for blur, noise, distortion, and ten levels for R, G, B, H, S, V. Hence, we have 75 datasets in total for testing in this scenario.
• Scenario 3: Combined Perturbation. Test on datasets with combinations of all factors at all levels. To be specific, we sample varying levels from each factor, and combine the resulting datasets of all factors into one combined dataset. In total, we have six combined datasets for testing in this scenario. See examples in the second row of Figure 6, and parameters for each dataset in the appendix section A.3.
• Scenario 4: Unseen Perturbation. Test on datasets under previously unseen factors at different levels. The unseen factors are “motion blur”, “zoom blur”, “pixelate”, “jpeg compression”, “snow”, “frost”, “fog” from ImageNet-C (Hendrycks & Dietterich, 2019). We choose these factors because, specifically, “Motion blur” and “zoom blur” can happen during driving; “pixelate” and “jpeg compression” are possible during image processing; and “snow”, “frost”, “fog” are natural weather conditions that can affect driving experiences. In addition, all these factors can be possibly “simulated” using the nine image factors analyzed in this work. See examples in Figure 4.
We use the maximum MA improvement (denoted as MMAI), the average MA improvement (denoted as AMAI) and mean Corruption Errors (mCE) (Hendrycks & Dietterich, 2019) as the evaluation metrics. From Table 1, we observe that our method outperforms other methods under all metrics in all scenarios. Specifically, our algorithm improves the performance of “learning to steer” up to 48% in MMAI, while reducing mCE down by 50% over the baseline. We have compared our approach with other related techniques, namely data augmentation and adversarial training. The results show that our method achieves consistent better performance. Our method also improves the task performance using the combined datasets (Scenario 3) up to 33%. Lastly, when tested on unseen factors (Scenario 4), our algorithm maintains the best performance by 29% in MMAI, while reducing mCE error to 76%.
We also illustrate the detailed MA improvements in Fig. 5, which shows that our method can achieve improvements in some extreme cases of the channel-factor levels and some of the unseen image effects. However, our method fails to improve on “motion blur”, which we plan to study in future. More detailed data, results, and analysis can be found in Appendix A.6.
By visualizing the salience map on several combined samples in Fig. 6, we show that our method can assist the network to focus on more important areas (e.g., the road in front), instead of focusing on random areas on perturbed images.

heading: 4.3 WIDER COMPARISON

text: To show our method can be generalized well in other cases, we show the experiment results on two new datasets (Honda (Ramanishka et al., 2018) and Audi (Geyer et al., 2020), see dataset setup in Appendix A.5), one new method (Augmix (Hendrycks et al., 2020)), and one new backbone network (comma.ai (Santana & Hotz, 2016)) in Table. 2 and Table. 3. As shown in the tables, our method achieves the best performance on most cases. Specifically, our method outperforms on clean and single perturbation data consistently (at least 2.5% better), and does better in most cases of unseen perturbation data. In combined perturbation data, our method performs better when using NVidia backbone network, while AugMix performs better when using comma.ai backbone network.

heading: 4.4 GENERALIZATION

text:
In this work, we introduce an efficient and effective computational framework that incorporates sensitivity analysis and a systematic mechanism to improve the performance of a learning algorithm for autonomous driving on both the original dataset and the simulated adversarial scenarios due to multiple perturbations defined on an influential set of important image attributes. This approach can be easily extended and applied beyond the set of factors and the learning algorithm analyzed in this paper. This method can generalize to analyzing any arbitrarily high number of image/input
factors, other learning algorithms, and multimodal sensor data. Furthermore, other autonomous systems where the perception-to-control functionality plays a key role can possibly benefit from such a technique as well.

heading: 4.5 BENCHMARKING DATASETS

text: We plan to release our “learn to steer under perturbations” datasets for benchmarking. The datasets will contain a base dataset, datasets with five levels of perturbation in blur, noise, and distortion, ten levels of variations in the channels R, G, B, H, S, V, multiple with combined perturbations over all nine factors, and five levels of each unseen simulated factor, including snow, fog, frost, motion blur, zoom blur, pixelate, and jpeg compression. There are 120 datasets in total, where each dataset contains about 10,000 images (totaling ∼ 1,200,000 images in all). The ground-truth steering angles for all images will also be provided for validation, along with the code that generated all simulated perturbations. We detail the parameters used to generate the datasets in Appendix A.3.

heading: 5 CONCLUSION AND FUTURE WORK

text:
In this paper, we first analyze how different attributes of image quality influences the performance of the “learning to steer” task for autonomous vehicles, as quantified by mean accuracy (MA). We study three image-level effects (i.e., blur, noise, and distortion) and six channel-level effects (i.e., R, G, B, H, S, V) on real images. We observe that image degradation due to all these effects can impact the task performance. By utilizing FID as the unifying metric, we conduct sensitivity analysis on FID-MA space. Next, we propose an effective and efficient training method to improve the generalization of the model for learning-based steering under multiple perturbations. Our model not only improves the task performance on the original dataset, but also achieves significant performance improvement on datasets with a mixture of perturbations (up to 48%), as well as unseen adversarial examples including snow, fog, and frost (up to 29%). Theses results showing that our model is one of the most general methods in this particular application domain. We will also release the datasets used in this work for benchmarking the evaluation of the robustness of “learning to steer” for autonomous driving.
The implementation of our method currently uses discretization to achieve efficient training, but further implementation optimization is possible. The efficiency can be further improved by utilizing other works like the reweighting strategy(Ren et al., 2018). In addition, we believe that the computational framework is generalizable to other image factors, learning algorithms, multimodal sensor data, and tasks in other application domains. These are all natural directions worthy of further exploration.