summaries: This paper proposes and analyzes a method for generating trained weights already optimized for a given single task for a large class of diverse architectures using Graph hypernetworks. They introduce a new DEEPNETS-1M benchmark with in-distribution and out of distribution architectures. The work also proposed three main improvements to improve performance of their GHN: normalization of predicted parameters, enhanced long-range interactions in the GHN, and meta-batching of architectures.  They demonstrate that their method is able to produce weights that do surprisingly well on unseen and diverse networks in less than 1 second on CPU or GPU for CIFAR-10 and Imagenet. They demonstrate that even for large Resnet-50s which are out of the training distribution, they could achieve 60% accuracy on CIFAR10. This paper studied the problem of parameter prediction for deep neural networks. Especially, the authors focused on the task of predicting parameters for various network structures with a single hypernetwork. The method is based GHN and introduces several important modifications. To evaluate the performance of the proposed method, a standardized benchmark is introduced.  The improved method showed impressive performance on CIFAR-10 and ImageNet even with large neural networks. Besides, the method is also proved to be useful in architecture representation learning.    In this paper, the authors propose the task of parameter prediction with a new DeepNets-1M dataset. By using a improved Graph HyperNetwork, they can achieve reasonable performance for unseen and diverse networks without iterative optimization. The learned neural architecture representation also performs better than previous works. This paper propose a way to predict/generate the parameters of an image based deep network instead of training it via SGD. It's a thought-provoking paper and the authors appears to be very detailed in their implementation. The results are surprisingly good but not better than standard optimization algorithms.
abstractText: Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures – DEEPNETS-1M– and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.
authors.name: Boris Knyazev
authors.name: Michal Drozdzal
authors.name: Graham W. Taylor
authors.name: Adriana Romero-Soriano
references.authors: Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla Michael Bernstein
references.title: Imagenet large scale visual recognition challenge
references.venue: International journal of computer vision,
references.authors: Sebastian Ruder
references.title: An overview of gradient descent optimization algorithms
references.venue: arXiv preprint arXiv:1609.04747,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Gao Huang Yu Sun Zhuang Liu Daniel Sedra Kilian Q Weinberger
references.title: Deep networks with stochastic depth
references.venue: In European conference on computer vision,
references.authors: Andrew Brock Theodore Lim James M Ritchie Nick Weston
references.title: Freezeout: Accelerate training by progressively freezing layers
references.venue: arXiv preprint arXiv:1706.04983,
references.authors: Dami Choi Alexandre Passos Christopher J Shallue George E Dahl
references.title: Faster neural network training with data echoing
references.authors: Sergey Ioffe Christian Szegedy
references.title: Batch normalization: Accelerating deep network training by reducing internal covariate shift
references.venue: arXiv preprint arXiv:1502.03167,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Tom B Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell
references.title: Language models are few-shot learners
references.authors: Emma Strubell Ananya Ganesh Andrew McCallum
references.title: Energy and policy considerations for deep learning in nlp
references.venue: arXiv preprint arXiv:1906.02243,
references.authors: Han Cai Chuang Gan Tianzhe Wang Zhekai Zhang Song Han
references.title: Once-for-all: Train one network and specialize it for efficient deployment, 2019
references.authors: Neil C Thompson Kristjan Greenewald Keeheon Lee Gabriel F Manso
references.title: The computational limits of deep learning
references.venue: arXiv preprint arXiv:2007.05558,
references.authors: Alex Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.authors: Timothy Hospedales Antreas Antoniou Paul Micaelli Amos Storkey
references.title: Meta-learning in neural networks: A survey
references.venue: arXiv preprint arXiv:2004.05439,
references.authors: Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W Hoffman David Pfau Tom Schaul Brendan Shillingford Nando De Freitas
references.title: Learning to learn by gradient descent by gradient descent
references.venue: In Advances in neural information processing systems,
references.authors: Sachin Ravi Hugo Larochelle
references.title: Optimization as a model for few-shot learning
references.authors: Hanxiao Liu Karen Simonyan Yiming Yang
references.title: Darts: Differentiable architecture search
references.venue: arXiv preprint arXiv:1806.09055,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: arXiv preprint arXiv:2010.11929,
references.authors: Thomas N. Kipf Max Welling
references.title: Semi-supervised classification with graph convolutional networks
references.venue: In 5th International Conference on Learning Representations,
references.authors: Petar Veličković Guillem Cucurull Arantxa Casanova Adriana Romero Pietro Liò Yoshua Bengio
references.title: Graph attention networks
references.venue: In International Conference on Learning Representations,
references.authors: Vijay Prakash Dwivedi Chaitanya K Joshi Thomas Laurent Yoshua Bengio Xavier Bresson
references.title: Benchmarking graph neural networks
references.venue: arXiv preprint arXiv:2003.00982,
references.authors: Chris Zhang Mengye Ren Raquel Urtasun
references.title: Graph hypernetworks for neural architecture search
references.venue: arXiv preprint arXiv:1810.05749,
references.authors: Karen Simonyan Andrew Zisserman
references.title: Very deep convolutional networks for large-scale image recognition
references.venue: arXiv preprint arXiv:1409.1556,
references.authors: Saining Xie Ross Girshick Piotr Dollár Zhuowen Tu Kaiming He
references.title: Aggregated residual transformations for deep neural networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Gao Huang Zhuang Liu Laurens Van Der Maaten Kilian Q Weinberger
references.title: Densely connected convolutional networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Barret Zoph Quoc V Le
references.title: Neural architecture search with reinforcement learning
references.venue: arXiv preprint arXiv:1611.01578,
references.authors: Barret Zoph Vijay Vasudevan Jonathon Shlens Quoc V Le
references.title: Learning transferable architectures for scalable image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Chenxi Liu Barret Zoph Maxim Neumann Jonathon Shlens Wei Hua Li-Jia Li Li Fei-Fei Alan Yuille Jonathan Huang Kevin Murphy
references.title: Progressive neural architecture search
references.venue: In Proceedings of the European conference on computer vision (ECCV),
references.authors: Esteban Real Alok Aggarwal Yanping Huang Quoc V Le
references.title: Regularized evolution for image classifier architecture search
references.venue: In Proceedings of the aaai conference on artificial intelligence,
references.authors: Xin Chen Lingxi Xie Jun Wu Qi Tian
references.title: Progressive darts: Bridging the optimization gap for nas in the wild
references.authors: Andrew Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen Mingxing Tan Weijun Wang Yukun Zhu Ruoming Pang Vijay Vasudevan
references.title: Searching for mobilenetv3
references.venue: In Proceedings of the IEEE/CVF International Conference on Computer Vision,
references.authors: Yujia Li Daniel Tarlow Marc Brockschmidt Richard Zemel
references.title: Gated graph sequence neural networks
references.venue: arXiv preprint arXiv:1511.05493,
references.authors: Kyunghyun Cho Bart Van Merriënboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning phrase representations using rnn encoder-decoder for statistical machine translation
references.venue: arXiv preprint arXiv:1406.1078,
references.authors: Andrew Brock Soham De Samuel L Smith
references.title: Characterizing signal propagation to close the performance gap in unnormalized resnets
references.venue: arXiv preprint arXiv:2101.08692,
references.authors: Andrew Brock Soham De Samuel L Smith Karen Simonyan
references.title: High-performance large-scale image recognition without normalization
references.venue: arXiv preprint arXiv:2102.06171,
references.authors: Laurent Sifre Stéphane Mallat
references.title: Rigid-motion scattering for texture classification
references.venue: arXiv preprint arXiv:1403.1687,
references.authors: Andrew G Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: Mobilenets: Efficient convolutional neural networks for mobile vision applications
references.venue: arXiv preprint arXiv:1704.04861,
references.authors: Jie Hu Li Shen Gang Sun
references.title: Squeeze-and-excitation networks
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: arXiv preprint arXiv:1706.03762,
references.authors: Harold W Kuhn
references.title: The hungarian method for the assignment problem
references.venue: Naval research logistics quarterly,
references.authors: Aric Hagberg Pieter Swart Daniel S Chult
references.title: Exploring network structure, dynamics, and function using networkx
references.venue: Technical report,
references.authors: Alain Barrat Marc Barthelemy Romualdo Pastor-Satorras Alessandro Vespignani
references.title: The architecture of complex weighted networks
references.venue: Proceedings of the national academy of sciences,
references.authors: Anna Golubeva Behnam Neyshabur Guy Gur-Ari
references.title: Are wider nets better given the same number of parameters
references.venue: arXiv preprint arXiv:2010.14495,
references.authors: Sergey Zagoruyko Nikos Komodakis
references.title: Wide residual networks
references.venue: arXiv preprint arXiv:1605.07146,
references.authors: Quynh Nguyen Matthias Hein
references.title: The loss surface of deep and wide neural networks
references.venue: In International conference on machine learning,
references.authors: Rupesh Kumar Srivastava Klaus Greff Jürgen Schmidhuber
references.title: Training very deep networks
references.venue: arXiv preprint arXiv:1507.06228,
references.authors: Sara Hooker
references.title: The hardware lottery, 2020
references.authors: Angus Galloway Anna Golubeva Thomas Tanay Medhat Moussa Graham W Taylor
references.title: Batch normalization is a cause of adversarial vulnerability
references.authors: Dan Hendrycks Thomas Dietterich
references.title: Benchmarking neural network robustness to common corruptions and perturbations
references.venue: arXiv preprint arXiv:1903.12261,
references.authors: Siyuan Qiao Huiyu Wang Chenxi Liu Wei Shen Alan Yuille
references.title: Micro-batch training with batchchannel normalization and weight standardization
references.authors: Hongyi Zhang Yann N Dauphin Tengyu Ma
references.title: Fixup initialization: Residual learning without normalization
references.venue: arXiv preprint arXiv:1901.09321,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
references.venue: In Proceedings of the IEEE international conference on computer vision,
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks
references.venue: In Proceedings of the thirteenth international conference on artificial intelligence and statistics,
references.authors: Oscar Chang Lampros Flokas Hod Lipson
references.title: Principled weight initialization for hypernetworks
references.venue: In International Conference on Learning Representations,
references.authors: Uri Alon Eran Yahav
references.title: On the bottleneck of graph neural networks and its practical implications
references.venue: arXiv preprint arXiv:2006.05205,
references.authors: Salah El Hihi Yoshua Bengio
references.title: Hierarchical recurrent neural networks for long-term dependencies
references.venue: In Advances in neural information processing systems,
references.authors: Meng Liu Zhengyang Wang Shuiwang Ji
references.title: Non-local graph neural networks
references.venue: arXiv preprint arXiv:2005.14612,
references.authors: Hongbin Pei Bingzhe Wei Kevin Chen-Chuan Chang Yu Lei Bo Yang
references.title: Geom-gcn: Geometric graph convolutional networks
references.venue: arXiv preprint arXiv:2002.05287,
references.authors: Jiaxuan You Rex Ying Jure Leskovec
references.title: Position-aware graph neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Yiding Yang Xinchao Wang Mingli Song Junsong Yuan Dacheng Tao
references.title: Spagan: Shortest path graph attention network
references.venue: arXiv preprint arXiv:2101.03464,
references.authors: Pavlo M Radiuk
references.title: Impact of training set batch size on the performance of convolutional neural networks for diverse datasets
references.venue: Information Technology and Management Science,
references.authors: Zhaohui Yang Yunhe Wang Xinghao Chen Boxin Shi Chao Xu Chunjing Xu Qi Tian Chang Xu
references.title: Cars: Continuous evolution for efficient neural architecture search
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Chaoyang He Haishan Ye Li Shen Tong Zhang
references.title: Milenas: Efficient neural architecture search via mixed-level reformulation
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Guohao Li Guocheng Qian Itzel C Delgadillo Matthias Muller Ali Thabet Bernard Ghanem
references.title: Sgas: Sequential greedy architecture search
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Olga Wichrowska Niru Maheswaranathan Matthew W Hoffman Sergio Gomez Colmenarejo Misha Denil Nando Freitas Jascha Sohl-Dickstein
references.title: Learned optimizers that scale and generalize
references.venue: In International Conference on Machine Learning,
references.authors: Luke Metz Niru Maheswaranathan C Daniel Freeman Ben Poole Jascha Sohl-Dickstein
references.title: Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
references.authors: Jingzhao Zhang Sai Praneeth Karimireddy Andreas Veit Seungyeon Kim Sashank J Reddi Sanjiv Kumar Suvrit Sra
references.title: Why adam beats sgd for attention models
references.authors: Priya Goyal Piotr Dollár Ross Girshick Pieter Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He
references.title: Accurate, large minibatch sgd: Training imagenet in 1 hour
references.venue: arXiv preprint arXiv:1706.02677,
references.authors: Alex Krizhevsky Ilya Sutskever Geoffrey E Hinton
references.title: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing
references.authors: Wei Li Shaogang Gong Xiatian Zhu
references.title: Neural graph embedding for neural architecture search
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Wei Wen Hanxiao Liu Hai Li Yiran Chen Gabriel Bender Pieter-Jan Kindermans
references.title: Neural predictor for neural architecture search
references.authors: Haifeng Jin Qingquan Song Xia Hu
references.title: Auto-keras: An efficient neural architecture search system
references.venue: In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
references.authors: Nils M Kriege Fredrik D Johansson Christopher Morris
references.title: A survey on graph kernels
references.venue: Applied Network Science,
references.authors: Ilya Makarov Dmitrii Kiselev Nikita Nikitinsky Lovro Subelj
references.title: Survey on graph embeddings and their applications to machine learning problems on graphs
references.venue: PeerJ Computer Science,
references.authors: Wei Wen Hanxiao Liu Yiran Chen Hai Li Gabriel Bender Pieter-Jan Kindermans
references.title: Neural predictor for neural architecture search
references.venue: In European Conference on Computer Vision,
references.authors: Jovita Lukasik David Friede Heiner Stuckenschmidt Margret Keuper
references.title: Neural architecture performance prediction using graph neural networks
references.venue: arXiv preprint arXiv:2010.10024,
references.authors: Bowen Baker Otkrist Gupta Ramesh Raskar Nikhil Naik
references.title: Accelerating neural architecture search using performance prediction
references.venue: arXiv preprint arXiv:1705.10823,
references.authors: Simon Kornblith Jonathon Shlens Quoc V Le
references.title: Do better imagenet models transfer better
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Minyoung Huh Pulkit Agrawal Alexei A Efros
references.title: What makes imagenet good for transfer learning
references.venue: arXiv preprint arXiv:1608.08614,
references.authors: Behnam Neyshabur Hanie Sedghi Chiyuan Zhang
references.title: What is being transferred in transfer learning
references.venue: arXiv preprint arXiv:2008.11687,
references.authors: Maithra Raghu Chiyuan Zhang Jon Kleinberg Samy Bengio
references.title: Transfusion: Understanding transfer learning for medical imaging
references.venue: arXiv preprint arXiv:1902.07208,
references.authors: Xiaohua Zhai Joan Puigcerver Alexander Kolesnikov Pierre Ruyssen Carlos Riquelme Mario Lucic Josip Djolonga Andre Susano Pinto Maxim Neumann Alexey Dosovitskiy
references.title: A large-scale study of representation learning with the visual task adaptation benchmark
references.venue: arXiv preprint arXiv:1910.04867,
references.authors: Liming Wang Jianbo Shi Gang Song I-fan Shen
references.title: Object detection combining recognition and segmentation
references.venue: In Asian conference on computer vision,
references.authors: Jürgen Schmidhuber AI Blog
references.title: Metalearning machines learn to learn (1987-)
references.authors: Louis Kirsch Jürgen Schmidhuber
references.title: Meta learning backpropagation and improving it
references.venue: arXiv preprint arXiv:2012.14905,
references.authors: Hugo Siqueira Gomes Benjamin Léger Christian Gagné
references.title: Meta learning black-box population-based optimizers
references.venue: arXiv preprint arXiv:2103.03526,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Jake Snell Kevin Swersky Richard S Zemel
references.title: Prototypical networks for few-shot learning
references.venue: arXiv preprint arXiv:1703.05175,
references.authors: Adriana Romero Pierre Luc Carrier Akram Erraqabi Tristan Sylvain Alex Auvolat Etienne Dejoie Marc-André Legault Marie-Pierre Dubé Julie G Hussin Yoshua Bengio
references.title: Diet networks: thin parameters for fat genomics
references.venue: arXiv preprint arXiv:1611.09340,
references.authors: James Requeima Jonathan Gordon John Bronskill Sebastian Nowozin Richard E Turner
references.title: Fast and flexible multi-task classification using conditional neural adaptive processes
references.authors: Huaiyu Li Weiming Dong Xing Mei Chongyang Ma Feiyue Huang Bao-Gang Hu
references.title: Lgm-net: Learning to generate matching networks for few-shot learning
references.venue: In International conference on machine learning,
references.authors: Luca Bertinetto João F Henriques Jack Valmadre Philip HS Torr Andrea Vedaldi
references.title: Learning feed-forward one-shot learners
references.venue: arXiv preprint arXiv:1606.05233,
references.authors: Dongze Lian Yin Zheng Yintao Xu Yanxiong Lu Leyu Lin Peilin Zhao Junzhou Huang Shenghua Gao
references.title: Towards fast adaptation of neural architectures with meta learning
references.venue: In ICLR. JMLR. org,
references.authors: Thomas Elsken Benedikt Staffler Jan Hendrik Metzen Frank Hutter
references.title: Meta-learning of neural architectures for few-shot learning
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Xin Chen Yawen Duan Zewei Chen Hang Xu Zihao Chen Xiaodan Liang Tong Zhang Zhenguo Li
references.title: Catch: Context-based meta reinforcement learning for transferrable architecture search
references.venue: In European Conference on Computer Vision,
references.authors: Jiahui Yu Pengchong Jin Hanxiao Liu Gabriel Bender Pieter-Jan Kindermans Mingxing Tan Thomas Huang Xiaodan Song Ruoming Pang Quoc Le
references.title: Bignas: Scaling up neural architecture search with big single-stage models
references.venue: arXiv preprint arXiv:2003.11142,
references.authors: Xin He Kaiyong Zhao Xiaowen Chu
references.title: Automl: A survey of the state-of-the-art
references.venue: Knowledge-Based Systems,
references.authors: Andrew Brock Theodore Lim James M Ritchie Nick Weston
references.title: Smash: one-shot model architecture search through hypernetworks
references.venue: arXiv preprint arXiv:1708.05344,
references.authors: Gabriel Bender Pieter-Jan Kindermans Barret Zoph Vijay Vasudevan Quoc Le
references.title: Understanding and simplifying one-shot architecture search
references.venue: In International Conference on Machine Learning,
references.authors: Guixiang Ma Nesreen K Ahmed Theodore L Willke S Yu Philip
references.title: Deep graph similarity learning: A survey
references.venue: Data Mining and Knowledge Discovery,
references.authors: Yunsheng Bai Hao Ding Song Bian Ting Chen Yizhou Sun Wei Wang
references.title: Simgnn: A neural network approach to fast graph similarity computation
references.venue: In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining,
references.authors: DC Dowson BV Landau
references.title: The fréchet distance between multivariate normal distributions
references.venue: Journal of multivariate analysis,
references.authors: Chia-Cheng Liu Harris Chan Kevin Luk AI Borealis
references.title: Auto-regressive graph generation modeling with improved evaluation methods
references.venue: In 33rd Conference on Neural Information Processing Systems. Vancouver,
references.authors: Julian Zilly Hannes Zilly Oliver Richter Roger Wattenhofer Andrea Censi Emilio Frazzoli
references.title: The frechet distance of training and test distribution predicts the generalization gap
references.authors: Rylee Thompson Elahe Ghalebi Terrance DeVries Graham W Taylor
references.title: Building lego using deep generative models of graphs
references.venue: arXiv preprint arXiv:2012.11543,
references.authors: Pinar Yanardag SVN Vishwanathan
references.title: Deep graph kernels
references.venue: In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining,
references.authors: Misha Denil Babak Shakibi Laurent Dinh Marc’Aurelio Ranzato Nando de Freitas
references.title: Predicting parameters in deep learning
references.venue: Advances in Neural Information Processing Systems
references.authors: Neale Ratzlaff Li Fuxin
references.title: Hypergan: A generative model for diverse, performant neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Iou-Jen Liu Jian Peng Alexander G Schwing
references.title: Knowledge flow: Improve upon your teachers
references.venue: arXiv preprint arXiv:1904.05878,
references.authors: Yu Cheng Duo Wang Pan Zhou Tao Zhang
references.title: A survey of model compression and acceleration for deep neural networks
references.venue: arXiv preprint arXiv:1710.09282,
references.authors: Yann Dauphin Samuel S
references.title: Schoenholz. Metainit: Initializing learning by learning to initialize
references.authors: Chen Zhu Renkun Ni Zheng Xu Kezhi Kong W Ronny Huang Tom Goldstein
references.title: Gradinit: Learning to initialize neural networks for stable and efficient training
references.venue: arXiv preprint arXiv:2102.08098,
references.authors: Debasmit Das Yash Bhalgat Fatih Porikli
references.title: Data-driven weight initialization with sylvester solvers
references.venue: arXiv preprint arXiv:2105.10335,
references.authors: Yue Yu Jie Chen Tian Gao Mo Yu
references.title: Dag-gnn: Dag structure learning with graph neural networks
references.venue: In International Conference on Machine Learning,
references.authors: Xiaojie Guo Liang Zhao
references.title: A systematic survey on deep generative models for graph generation
references.venue: arXiv preprint arXiv:2007.06686,
references.authors: Ilija Radosavovic Raj Prateek Kosaraju Ross Girshick Kaiming He Piotr Dollár
references.title: Designing network design spaces
references.venue: In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
references.authors: Jiaxuan You Zhitao Ying Jure Leskovec
references.title: Design space for graph neural networks
references.venue: Advances in Neural Information Processing Systems,
sections.heading: 1 Introduction
sections.text: Consider the problem of training deep neural networks on large annotated datasets, such as ImageNet [1]. This problem can be formalized as finding optimal parameters for a given neural network a, parameterized by w, w.r.t. a loss function L on the dataset D = {(xi, yi)}Ni=1 of inputs xi and targets yi:
  arg min w ∑N i=1 L(f(xi; a,w), yi), (1)
  where f(xi; a,w) represents a forward pass. Equation 1 is usually minimized by iterative optimization algorithms – e.g. SGD [2] and Adam [3] – that converge to performant parameters wp of the architecture a. Despite the progress in improving the training speed and convergence [4–7], obtaining wp remains a bottleneck in large-scale machine learning pipelines. For example, training a ResNet-50 [8] on ImageNet can take many GPU hours [9]. With the ever growing size of networks [10] and necessity of training the networks repeatedly (e.g. for hyperparameter or architecture search), the classical process of obtaining wp is becoming computationally unsustainable [11–13].
  A new parameter prediction task. When optimizing the parameters for a new architecture a, typical optimizers disregard past experience gained by optimizing other nets. However, leveraging this past experience can be the key to reduce the reliance on iterative optimization and, hence the high computational demands. To progress in that direction, we propose a new task where iterative optimization is replaced with a single forward pass of a hypernetwork [14] HD. To tackle the task,
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  ∗Part of the work was done while interning at Facebook AI Research.
  HD is expected to leverage the knowledge of how to optimize other networks F . Formally, the task is to predict the parameters of an unseen architecture a /∈ F using HD, parameterized by θp: ŵp = HD(a; θp). The task is constrained to a dataset D, so ŵp are the predicted parameters for which the test set performance of f(x; a, ŵp) is similar to the one of f(x; a,wp). For example, we consider CIFAR-10 [15] and ImageNet image classification datasets D, where the test set performance is classification accuracy on test images.
  Approaching our task. A straightforward approach to expose HD to the knowledge of how to optimize other networks is to train it on a large training set of {(ai,wp,i)} pairs, however, that is prohibitive2. Instead, we follow the bi-level optimization paradigm common in meta-learning [16–18], but rather than iterating over M tasks, we iterate over M training architectures F = {ai}Mi=1:
  arg min θ ∑N j=1 ∑M i=1 L ( f ( xj ; ai, HD(ai; θ) ) , yj ) . (2)
  By optimizing Equation 2, the hypernetwork HD gradually gains knowledge of how to predict performant parameters for training architectures. It can then leverage this knowledge at test time – when predicting parameters for unseen architectures. To approach the problem in Equation 2, we need to design the network space F and HD. For F , we rely on the previous design spaces for neural architectures [19] that we extend in two ways: the ability to sample distinct architectures and an expanded design space that includes diverse architectures, such as ResNets and Visual Transformers [20]. Such architectures can be fully described in the form of computational graphs (Fig. 1). So, to design the hypernetwork HD, we rely on recent advances in machine learning on graph-structured data [21–24]. In particular, we build on the Graph HyperNetworks method (GHNs) [24] that also optimizes Equation 2. However, GHNs do not aim to predict large-scale performant parameters as we do in this work, which motivates us to improve on their approach.
  By designing our diverse space F and improving on GHNs, we boost the accuracy achieved by the predicted parameters on unseen architectures to 77% (top-1) and 48% (top-5) on CIFAR-10 [15] and ImageNet [1], respectively. Surprisingly, our GHN shows good out-of-distribution generalization and predicts good parameters for architectures that are much larger and deeper compared to the ones seen in training. For example, we can predict all 24 million parameters of ResNet-50 in less than a second either on a GPU or CPU achieving ∼60% on CIFAR-10 without any gradient updates (Fig 1, (b)). Overall, our framework and results pave the road toward a new and significantly more efficient paradigm for training networks. Our contributions are as follows: (a) we introduce the novel task of predicting performant parameters for diverse feedforward neural networks with a single hypernetwork forward pass; (b) we introduce DEEPNETS-1M – a standardized benchmark with in-distribution and out-of-distribution architectures to track progress on the task (§ 3); (c) we define several baselines and propose a GHN model (§ 4) that performs surprisingly well on CIFAR-10 and ImageNet (§ 5.1); (d) we show that our model learns a strong representation of neural network architectures (§ 5.2), and our model is useful for initializing neural networks (§ 5.3). Our DEEPNETS-1M dataset, trained GHNs and code is available at https://github.com/facebookresearch/ppuda.
  D eco d er
  2Training a single network ai can take several GPU days and thousands of trained networks may be required.
  2
sections.heading: 2 Background
sections.text: We start by providing a brief background about the network design spaces leveraged in the creation of our DEEPNETS-1M dataset of neural architectures described in § 3. We then cover elements of graph hypernetworks that we leverage when designing our specific GHN HD in § 4.
sections.heading: 2.1 Network Design Space of DARTS
sections.text: DARTS [19] is a differentiable NAS framework. For image classification tasks such as those considered in this work, its networks are defined by four types of building blocks: stems, normal cells, reduction cells, and classification heads. Stems are fixed blocks of convolutional operations that process input images. The normal and reduction cells are the main blocks of architectures and are composed of: 3×3 and 5×5 separable convolutions, 3×3 and 5×5 dilated separable convolutions, 3×3 max pooling, 3×3 average pooling, identity and zero (to indicate the absence of connectivity between two operations). Finally, the classification head defines the network output and is built with a global pooling followed by a single fully connected layer.
  Typically, DARTS networks have one stem block, 14-20 cells, and one classification head, altogether forming a deep computational graph. The reduction cells, placed only at 1/3 and 2/3 of the total depth, decrease the spatial resolution and increase the channel dimensionality by a factor of 2. Summation and concatenation are used to aggregate outputs from multiple operations within each cell. To make the channel dimensionalities match, 1×1 convolutions are used as needed. All convolutional operations use the ReLU-Conv-Batch Norm (BN) [7] order. Overall, DARTS enables defining strong architectures that combine many principles of manual [25, 8, 26, 27] and automatic [24, 28–33] design of neural architectures. While DARTS learns the optimal task-specific cells, the framework can be modified to permit sampling randomly-structured cells. We leverage this possibility for the DEEPNETS-1M construction in § 3. Please see § A.1 for further details on DARTS.
sections.heading: 2.2 Graph HyperNetwork: GHN-1
sections.text: Representation of architectures. GHNs [24] directly operate on the computational graph of a neural architecture a. Specifically, a is a directed acyclic graph (DAG), where nodes V = {vi}|V |i=1 are operations (e.g. convolutions, fully-connected layers, summations, etc.) and their connectivity is described by a binary adjacency matrix A ∈ {0, 1}|V |×|V |. Nodes are further characterized by a matrix of initial node features H0 = [h01,h 0 2, ...,h 0 |V |], where each h 0 v is a one-hot vector representing the operation performed by the node. We also use such a one-hot representation for H0, but in addition encode the shape of parameters associated with nodes as described in detail in § B.1.
  Design of the graph hypernetwork. In [24], the graph hypernetwork HD consists of three key modules. The first module takes the input node features H0 and transforms them into d-dimensional node features H1 ∈ R|V |×d through an embedding layer. The second module takes H1 together with A and feeds them into a specific variant of the gated graph neural network (GatedGNN) [34]. In particular, their GatedGNN mimics the canonical order π of node execution in the forward (fw) and backward (bw) passes through a computational graph. To do so, it sequentially traverses the graph and performs iterative message passing operations and node feature updates as follows:
  ∀t ∈ [1, ..., T ] : [ ∀π ∈ [fw, bw] : ( ∀v ∈ π : mtv = ∑ u∈Nπv MLP(htu), h t v = GRU(h t v,m t v) )] , (3) where T denotes the total number of forward-backward passes; htv corresponds to the features of node v in the t-th graph traversal; MLP(·) is a multi-layer perceptron; and GRU(·) is the update function of the Gated Recurrent Unit [35]. In the forward propagation (π = fw), N πv corresponds to the incoming neighbors of the node defined by A, then in the backward propagation (π = bw) it similarly corresponds to the outgoing neighbors of the node. The last module uses the GatedGNN output hidden states hTv to condition a decoder that produces the parameters ŵ v p (e.g. convolutional weights) associated with each node. In practice, to handle different parameter dimensionalities per operation type, the output of the hypernetwork is reshaped and sliced according to the shape of parameters in each node. We refer to the model described above as GHN-1 (Fig. 1). Further subtleties of implementing this model in the context of our task are discussed in § B.1.
sections.heading: 3 DEEPNETS-1M
sections.text: The network design space of DARTS is limited by the number of unique operations that compose cells, and the low variety of stems and classification heads. Thus, many architectures are not realizable within this design space, including: VGG [25], ResNets [8], MobileNet [33] or more recent ones such as Visual Transformer (ViT) [20] and Normalization-free networks [36, 37]. Furthermore, DARTS does not define a procedure to sample random architectures. By addressing these two limitations we aim to expose our hypernetwork to diverse training architectures and permit its evaluation on common architectures, such as ResNet-50. We hypothesize that increased training diversity can improve hypernetworks’ generalization to unseen architectures making it more competitive to iterative optimizers.
  Extending the network design space. We extend the set of possible operations with non-separable 2D convolutions3, Squeeze&Excite4 (SE) [40] and Transformer-based operations [41, 20]: multihead self-attention (MSA), positional encoding and layer norm (LN) [42]. Each node (operation) in our graphs has two attributes: primitive type (e.g. convolution) and shape (e.g. 3×3×512×512). Overall, our extended set consists of 15 primitive types (Table 1). We also extend the diversity of the generated architectures by introducing VGG-style classification heads and ViT stems. Finally, to further increase architectural diversity, we allow the operations to not include batch norm (BN) [7] and permit networks without channel width expansion (e.g. as in [20]).
  Architecture generation process. We generate different subsets of architectures (see the description of each subset in the next two paragraphs and in Table 1). For each subset depending on its purpose, we predefine a range of possible model depths (number of cells), widths and number of nodes per cell. Then, we sample a stem, a normal and reduction cell and a classification head. The internal structure of the normal and reduction cells is defined by uniformly sampling from all available operations. Due to a diverse design space it is extremely unlikely to sample the same architecture multiple times, but we ran a sanity check using the Hungarian algorithm [43] to confirm that (see Figure 6 in § A.2 for details).
  In-distribution (ID) architectures. We generate a training set of |F| = 106 architectures and validation/test sets of 500/500 architectures that follow the same generation rules and are considered to be ID samples. However, training on large architectures can be prohibitive, e.g. in terms of GPU memory. Thus, in the training set we allow the number of channels and, hence the total number of parameters, to be stochastically defined given computational resources. For example, to train
  3Non-separable convolutions have weights of e.g. shape 3×3×512×512 as in ResNet-50. NAS works, such as DARTS and GHN, avoid such convolutions, since the separable ones [38] are more efficient. Non-separable convolutions are nevertheless common in practice and can often boost the downstream performance.
  4The Squeeze&Excite operation is common in many efficient networks [39, 12].
  our models we upper bound the number of parameters in the training architectures to around 3M by sampling fewer channels if necessary. In the evaluation sets, the number of channels is fixed. Therefore, this pre-processing step prior to training results in some distribution shift between the training and the validation/test sets. However, the shift is not imposed by our dataset.
  Out-of-distribution (OOD) architectures. We generate five OOD test sets that follow different generation rules. In particular, we define WIDE and DEEP sets that are of interest due the stronger downstream performance of such nets in large-scale tasks [47, 48, 10]. These nets are often more challenging to train for fundamental [49, 50] or computational [51] reasons, so predicting their parameters might ease their subsequent optimization. We also define the DENSE set, since networks with many operations per cell and complex connectivity are underexplored in the literature despite their potential [27]. Next, we define the BN-FREE set that is of interest due to BN’s potential negative side-effects [52, 53] and the difficulty or unnecessity of using it in some cases [54–56, 36, 37]. We finally add the RESNET/VIT set with two predefined image classification architectures: commonlyused ResNet-50 [8] and a smaller 12-layer version of the Visual Transformer (ViT) [20] that has recently received a lot of attention in the vision community. Please see § A.1 and § A.2 for further details and statistics of our DEEPNETS-1M dataset.
sections.heading: 4 Improved Graph HyperNetworks: GHN-2
sections.text: In this section, we introduce our three key improvements to the baseline GHN-1 described in § 2.2 (Fig. 1). These components are essential to predict stronger parameters on our task. For the empirical validation of the effectiveness of these components see ablation studies in § 5.1 and § C.2.1.
sections.heading: 4.1 Differentiable Normalization of Predicted Parameters
sections.text: When training the parameters of a given network from scratch using iterative optimization methods, the initialization of parameters is crucial. A common approach is to use He [57] or Glorot [58] initialization to stabilize the variance of activations across layers of the network. Chang et al. [59] showed that when the parameters of the network are instead predicted by a hypernetwork, the activations in the network tend to
  explode or vanish. To address the issue of unstable network activations especially for the case of predicting parameters of diverse architectures, we apply operation-dependent normalizations (Table 2). We normalize convolutional and fully-connected weights by following the fan-in scheme of [57] (see the comparison to fan-out in § C.2.1): ŵvp √ β/(CinHW), where Cin,H,W are the number of input channels and spatial dimensions of weights ŵvp , respectively; and β is a nonlinearity specific constant following the analysis in [57]. The parameters of normalization layers such as BN and LN, as well as biases typically initialized with constants, are normalized by applying a squashing function with temperature T to imitate the empirical distributions of models trained with SGD (see Table 2). These are differentiable normalizations, so that they are applied at training (and testing) time. Further analysis of our normalization and its stabilizing effect on activations is presented in § B.2.2.
  4.2 Enhancing Long-range Message Propagation
  Computational graphs often take the form of long chains (Table 1) with only a few incoming/outcoming edges per node. This structure might hinder long-range propagation of information between nodes [60]. Different approaches to alleviate the long-range propagation problem exist [61–63], including stacking GHNs in [24]. Instead we adopt simple graph-based heuristics in line with recent works [64, 65]. In particular, we add virtual edges between two nodes v and u and weight them based on the shortest
  path svu between them (Fig. 2). To avoid interference with the real edges in the computational graph, we introduce a separate MLPsp to transform the features of the nodes connected through these virtual edges, and redefine the message passing of Equation 3 as:
  mtv = ∑
  u∈Nπv MLP(htu) + ∑ u∈N (sp)v 1 svu MLPsp(htu), (4)
  whereN (sp)v are neighbors satisfying 1 < svu ≤ s(max), and s(max) is a hyperparameter. To maintain the same number of trainable parameters as in GHN-1, we decrease MLPs’ sizes appropriately. Despite its simplicity, this approach is effective (see the comparison to stacking GHNs in § C.2.1).
sections.heading: 4.3 Meta-batching Architectures During Training
sections.text: GHN-1 updates its parameters θ based on a single architecture sampled for each batch of images (Equation 2). In vanilla SGD training, larger batches of images often speed up convergence by reducing gradient noise and improve model’s performance [66]. Therefore, we define a meta-batch bm as the number of architectures sampled per batch of images. Both the parameter prediction and the forward/backward passes through the architectures in a meta-batch can be done in parallel. We then average the gradients across bm to update the parameters θ ofHD: ∇θL = 1/bm ∑bm i=1∇θLi. Further analysis of the meta-batching effect on the training loss and convergence speed is presented in § B.2.3.
sections.heading: 5 Experiments
sections.text: We focus the evaluation of GHN-2 on our parameter prediction task (§ 5.1). In addition, we show beneficial side-effects of i) learning a stronger neural architecture representation using GHN-2 in analyzing networks (§ 5.2) and ii) predicting parameters for fine-tuning (§ 5.3). We provide further experimental and implementation details, as well as more results supporting our arguments in § C.
  Datasets. We use the DEEPNETS-1M dataset of architectures (§ 3) as well as two image classification datasets D1 (CIFAR-10 [15]) and D2 (ImageNet [1]). CIFAR-10 consists of 50k training and 10k test images of size 32×32×3 and 10 object categories. ImageNet is a larger scale dataset with 1.28M training and 50k test images of variable size and 1000 fine-grained object categories. We resize ImageNet images to 224×224×3 following [19, 24]. We use 5k/50k training images as a validation set in CIFAR-10/ImageNet and 500 validation architectures of DEEPNETS-1M for hyperparameter tuning.
  Baselines. Our baselines include GHN-1 and a simple MLP that only has access to operations, but not to the connections between them. This MLP baseline is obtained by replacing the GatedGNN with an MLP in our GHN-2. Since GHNs were originally introduced for small architectures of∼ 50 nodes and only trained on CIFAR-10, we reimplement5 them and scale them up by introducing minor modifications to their decoder that enable their training on ImageNet and on larger architectures of up to 1000 nodes (see § B.1 for details). We use the same hyperparameters to train the baselines and GHN-2.
  Iterative optimizers. In the parameter prediction experiments, we also compare our model to standard optimization methods: SGD and Adam [3]. We use off-the-shelf hyperparameters common in the literature [24, 19, 32, 67–69]. On CIFAR-10, we train evaluation architectures with SGD/Adam, initial learning rate η = 0.025 / η = 0.001, batch size b = 96 and up to 50 epochs. With Adam, we train only 300 evaluation architectures as a rough estimation of an average performance. On ImageNet, we train them with SGD, η = 0.1 and b = 128, and, for computational reasons (given 1402 evaluation architectures in total), we limit training with SGD to 1 epoch. We have also considered meta-optimizers, such as [17, 18]. However, we were unable to scale them to diverse and large architectures of our DEEPNETS-1M, since their LSTM requires a separate hidden state for every trainable parameter in the architecture. The scalable variants exist [70, 71], but are hard to reproduce without open source code.
  Additional experimental details. We follow [24] and train GHNs with Adam, η = 0.001 and batch size of 64 images for CIFAR-10 and 256 for ImageNet. We train for up to 300 epochs, except for one experiment in the ablation studies, where we train one GHN with bm = 1 eight times longer, i.e. for 2400 epochs. All GHNs in our experiments use T = 1 propagation (Equation 3), as we found the original T = 5 of [24] to be inefficient and it did not improve the accuracies in our task. GHN-2 uses s(max) = 50 and bm = 8 and additionally uses LN that slightly further improves results (see these ablations in § C.2.1). Model selection is performed on the validation sets, but the results in our paper are reported on the test sets to enable their direct comparison.
sections.heading: 5.1 Parameter Prediction
sections.text: Experimental setup. We trained our GHN-2 and baselines on the training architectures and training images, i.e. a separate model is trained for CIFAR-10 and ImageNet. According to our DEEPNETS-1M benchmark, we assess whether these models can generalize to unseen in-distribution (ID) and out-of-distribution (OOD) test architectures from our DEEPNETS-1M. We measure this generalization by predicting parameters for the test architectures and computing their classification accuracies on the test images of CIFAR-10 (Table 3) and ImageNet (Table 4). The evaluation architectures with batch norm (BN) have running statistics, which are not learned by gradient descent [7], and
  5While source code for GHNs [24] is unavailable, we appreciate the authors’ help in implementing some steps.
  hence are not predicted by our GHNs. To alleviate that, we follow [24] and evaluate the networks with BN by computing per batch statistics with batch size of 64 images. This is further discussed in § C.1.
  Results. Despite GHN-2 never observed the test architectures, GHN-2 predicts good parameters for them making the test networks perform surprisingly well on both image datasets (Tables 3 and 4). Our results are especially strong on CIFAR-10, where some architectures with predicted parameters achieve up to 77.1%, while the best accuracy of training with SGD for 50 epochs is around 15% more. We even show good results on ImageNet, where for some architectures we achieve a top-5 accuracy of up to 48.3%. While these results are low for direct downstream applications, they are remarkable for three main reasons. First, to train GHNs by optimizing Equation 2, we do not rely on the prohibitively expensive procedure of training the architectures F by SGD. Second, GHNs rely on a single forward pass to predict all parameters. Third, these results are obtained for unseen architectures, including the OOD ones. Even in the case of severe distribution shifts (e.g. ResNet-506) and underrepresented networks (e.g. ViT7), our model still predicts parameters that perform better than random ones. On CIFAR-10, generalization of GHN-2 is particularly strong with a 58.6% accuracy on ResNet-50.
  On both image datasets, our GHN-2 significantly outperforms GHN-1 on all test subsets of DEEPNETS-1M with more than a 20% absolute gain in certain cases, e.g. 36.8% vs 13.7% on the BN-FREE networks (Table 3). Exploiting the structure of computational graphs is a critical property of GHNs with the accuracy dropping from 66.9% to 42.2% on ID (and even more on OOD) architectures when we replace the GatedGNN of GHN-2 with an MLP. Compared to iterative optimization methods, GHN-2 predicts parameters achieving an accuracy similar to ∼2500 and ∼5000 iterations of SGD on CIFAR-10 and ImageNet respectively. In contrast, GHN-1 performs similarly to only ∼500 and ∼2000 (not shown in Table 4) iterations respectively. Comparing SGD to Adam, the latter performs worse in general except for the ViT architectures similar to [72, 20].
  To report speeds on ImageNet in Table 4, we use a dedicated machine with a single NVIDIA V100-32GB and Intel Xeon CPU E5-1620 v4@ 3.50GHz. So for SGD these numbers can be reduced by using faster computing infrastructure and more optimal hyperparameters [73]. Using our setup,
  frequent operations in the training set.
  SGD requires on average 104× more time on a GPU (105× on a CPU) to obtain parameters that yield performance similar to GHN-2. As a concrete example, AlexNet [74] requires around 50 GPU hours (on our setup) to achieve a 81.8% top-5 accuracy, while on some architectures we achieve ≥48.0% in just 0.3 GPU seconds.
  Table 5: Ablating GHN-2 on CIFAR-10. An average rank of the model is computed across all ID and OOD test architectures.
  MODEL ID-TEST OOD-TEST AVG. RANK
  GHN-2 66.9±0.3 56.8±0.8 1.9 1000 training architectures 65.1±0.5 52.5±1.0 2.6 No normalization (§ 4.1) 62.6±0.6 47.1±1.2 3.9 No virtual edges (§ 4.2) 61.5±0.4 53.9±0.6 4.1 No meta-batch (bm = 1, § 4.3) 54.3±0.3 47.5±0.6 5.5 bm = 1, train 8× longer 62.4±0.5 51.9±1.0 3.7 No GatedGNN (MLP) 42.2±0.6 32.2±0.7 7.4 GHN-1 51.4±0.4 39.2±0.9 6.8
  0 10 20 30 40 50 60 Te st a cc ur ac y Test bm=1 (ID) bm=8 (ID) bm=1 (OOD) bm=8 (OOD)
  1 10 102 103 104 105 106 # Training architectures
  50
  60
  70
  80
  90
  Tr ai
  n ac
  cu ra
  cy
  Train bm=1 bm=8
  Figure 3: GHN-2 with meta batch bm = 8 versus bm = 1 for different numbers of training architectures on CIFAR-10.
  Ablations (Table 5) show that all three components proposed in § 4 are important. Normalization is particularly important for OOD generalization with the largest drops on the WIDE and BN-FREE networks (see § C.2.1). Using meta-batching (bm = 8) is also essential and helps stabilize training and accelerate convergence (see § B.2). We also confirm that the performance gap between bm = 1 and bm = 8 is not primarily due to the observation of more architectures, since the ablated GHN-2 with bm = 1 trained eight times longer is still inferior. The gap between bm = 8 and bm = 1 becomes pronounced with at least 1k training architectures (Fig. 3). When training with fewer architectures (e.g. 100), the GHN with meta-batching starts to overfit to the training architectures. Given our challenging setup with unseen evaluation architectures, it is surprising that using 1k training architectures already gives strong results. However, OOD generalization degrades in this case compared to using all 1M architectures, especially on the BN-FREE networks (see § B.2). When training GHNs on just a few architectures, the training accuracy soars to the level of training them with SGD. With more architectures, it generally decreases indicating classic overfitting and underfitting cases.
sections.heading: 5.2 Property Prediction
sections.text: Representing computational graphs of neural architectures is a challenging problem [75–79]. We verify if GHNs are capable of doing that out-of-the-box in the property prediction experiments. We also experiment with architecture comparison in § C.2.4. Our hypothesis is that by better solving our parameter prediction task, GHNs should also better solve graph representation tasks.
  Experimental setup. We predict the properties of architectures given their graph embeddings obtained by averaging node features8. We consider four such properties (see § C.2.3 for details): • Accuracy on the “clean” (original) validation set of images; • Accuracy on a corrupted set (obtained by adding the Gaussian noise to images following [53]); • Inference speed (latency or GPU seconds per a batch of images); • Convergence speed (the number of SGD iterations to achieve a certain training accuracy).
  Estimating these properties accurately can have direct practical benefits. Clean and corrupted accuracies can be used to search for the best performing architectures (e.g. for the NAS task); inference speed can be used to choose the fastest network, so by estimating these properties we can trade-off accurate, robust and fast networks [12]. Convergence speed can be used to find networks that are easier to optimize. These properties correlate poorly with each other and between CIFAR-10 and ImageNet (§ C.2.3), so they require the model to capture different regularities of graphs. While specialized methods to estimate some of these properties exist, often as a NAS task [80–82, 30, 75], our GHNs provide a generic representation that can be easily used for many such properties. For each property, we train a simple regression model using graph embeddings and ground truth property values. We use 500 validation architectures of DEEPNETS-1M for training the regression model and tuning its hyperparameters (see § C.2.3 for details). We then use 500 testing architectures of DEEPNETS-1M to measure Kendall’s Tau rank correlation between the predicted and ground truth property values similar to [80].
  8A fixed size graph embedding for the architecture a can be computed by averaging the output node features: ha =
  1 |V | ∑ v∈V h T v , where ha ∈ Rd and d is the dimensionality of node features.
  Additional baseline. We compare to the Neural Predictor (NeuPred) [80]. NeuPred is based on directed graph convolution and is developed for accuracy prediction achieving strong NAS results. We train a separate such NeuPred for each property from scratch following their hyperparameters.
  Results. GHN-2 consistently outperforms the GHN-1 and MLP baselines as well as NeuPred (Fig. 4). In § C.2.3, we also provide results verifying if higher correlations translate to downstream gains. For example, on CIFAR-10 by choosing the most accurate architecture according to the regression model and training it from scratch following [19, 24], we obtained a 97.26%(±0.09) accuracy, which is competitive with leading NAS approaches, e.g. [19, 24, 32, 67–69]. In contrast, the
  network chosen by the regression model trained on the GHN-1 embeddings achieves 95.90%(±0.08).
sections.heading: 5.3 Fine-tuning Predicted Parameters
sections.text: Neural networks trained on ImageNet and other large datasets have proven useful in diverse visual tasks in the transfer learning setup [83–87, 20]. Therefore, we explore how predicting parameters on ImageNet with GHNs compares to pretraining them on ImageNet with SGD in such a setup. We consider low-data tasks as they often benefit more from transfer learning [86, 87].
  Experimental setup. We perform two transfer-learning experiments. The first experiment is finetuning the predicted parameters on 1,000 training samples (100 labels per class) of CIFAR-10. We fine-tune ResNet-50, Visual Transformer (ViT) and a 14-cell architecture based on the DARTS best cell [19]. The hyperparameters of fine-tuning (initial learning rate and weight decay) are tuned on 200 validation samples held-out of the 1,000 training samples. The number of epochs is fixed to 50 as in § 5.1 for simplicity. In the second experiment, we fine-tune the predicted parameters on the object detection task. We closely follow the experimental protocol and hyperparameters from [88] and train the networks on the Penn-Fudan dataset [89]. The dataset contains only 170 images and the task is to detect pedestrians. Therefore this task is also well suited for transfer learning. Following [88], we replace the backbone of a Faster R-CNN with one of the three architectures. To perform transfer learning with GHNs, in both experiments we predict the parameters of a given architecture using GHNs trained on ImageNet. We then replace the ImageNet classification layer with the target task-specific layers and fine-tune the entire network on the target task. We compare the results of GHNs to He’s initialization [57] and the initialization based on pretraining the parameters on ImageNet with SGD.
  Results. The CIFAR-10 image classification results of fine-tuning the parameters predicted by our GHN-2 are≥10 percentage points better (in absolute terms) than fine-tuning the parameters predicted by GHN-1 or training the parameters initialized using He’s method (Table 6). Similarly, the object detection results of GHN-2-based initialization are consistently better than both GHN-1 and He’s initializations. The GHN-2 results are a factor of 1.5-3 improvement over He’s for all the three architectures. Overall, the two experiments clearly demonstrate the practical value of predicting parameters using our GHN-2. Using GHN-1 for initialization provides relatively small gains or hurts convergence (for ViT). Compared to pretraining on ImageNet with SGD, initialization using GHN-2 leads to performance sim-
  ilar to 1k-2.5k steps of pretraining on ImageNet depending on the architecture in the case of CIFAR-10. In the case of Penn-Fudan, GHN-2’s performance is similar to ≥1k steps of pretraining with SGD. In both experiments, pretraining on ImageNet for just 5 epochs provides strong transfer learning performance and the final ImageNet checkpoints are only slightly better, which aligns with previous works [85]. Therefore, further improvements in the parameter prediction models appear promising.
sections.heading: 6 Related Work
sections.text: Our proposed parameter prediction task, objective in Equation 2 and improved GHN are related to a wide range of machine learning frameworks, in particular meta-learning and neural architecture search (NAS). Meta-learning is a general framework [16, 90] that includes meta-optimizers and meta-models, among others. Related NAS works include differentiable [19] and one-shot methods [12]. See additional related work in § D.
  Meta-optimizers. Meta-optimizers [17, 18, 71, 91, 92] define a problem similar to our task, but where HD is an RNN-based model predicting the gradients ∇w, mimicking the behavior of iterative optimizers. Therefore, the objective of meta-optimizers may be phrased as learning to optimize as opposed to our learning to predict parameters. Such meta-optimizers can have their own hyperparameters that need to be tuned for a given architecture a and need to be run expensively (on the GPU) for many iterations following Equation 1.
  Meta-models. Meta-models include methods based on MAML [93], ProtoNets [94] and auxiliary nets predicting task-specific parameters [95–98]. These methods are tied to a particular architecture and need to be trained from scratch if it is changed. Several recent methods attempt to relax the choice of architecture in meta-learning. T-NAS [99] combines MAML with DARTS [19] to learn both the optimal architecture and its parameters for a given task. However, the best network, a, needs to be trained using MAML from scratch. Meta-NAS [100] takes a step further and only requires fine-tuning of a on a given task. However, the a is obtained from a single meta-architecture and so its choice is limited, preventing parameter prediction for arbitrary a. CATCH [101] follows a similar idea, but uses reinforcement learning to quickly search for the best a on the specific task. Overall meta-learning mainly aims at generalization across tasks, often motivated by the few-shot learning problem. In contrast, our parameter prediction problem assumes a single task (here an image dataset), but aims at generalization across architectures a with the ability to predict parameters in a single forward pass.
  One-shot NAS. One-shot NAS aims to learn a single “supernet” [102, 12, 103] that can be used to estimate the performance of smaller nets (subnets) obtained by some kind of pruning the supernet, followed by training the best chosen a from scratch with SGD. Recent models, in particular BigNAS [12] and OnceForAll (OFA) [102], eliminate the need to train subnets. However, the fundamental limitation of one-shot NAS is poor scaling with the number of possible computational operations [24]. This limits the diversity of architectures for which parameters can be obtained. For example, all subnets in OFA are based on MobileNet-v3 [33], which does not allow to solve our more general parameter prediction task. To mitigate this, SMASH [104] proposed to predict some of the parameters using hypernetworks [14] by encoding architectures as a 3D tensor. Graph HyperNetworks (GHNs) [24] further generalized this approach to “arbitrary” computational graphs (DAGs), which allowed them to improve NAS results. GHNs focused on obtaining reliable subnetwork rankings for NAS and did not aim to predict large-scale performant parameters. We show that the vanilla GHNs perform poorly on our parameter prediction task mainly due to the inappropriate scale of predicted parameters, lack of long-range interactions in the graphs, gradient noise and slow convergence when optimizing Equation 2. Conventionally to NAS, GHNs were also trained in a quite constrained architecture space [105]. We expand the architecture space adopting GHNs for a more general problem.
sections.heading: 7 Conclusion
sections.text: We propose a novel framework and benchmark to learn and evaluate neural parameter prediction models. Our model (GHN-2) is able to predict parameters for very diverse and large-scale architectures in a single forward pass in a fraction of a second. The networks with predicted parameters yield surprisingly high image classification accuracy given the extremely challenging nature of our parameter prediction task. However, the accuracy is still far from networks trained with handcrafted optimization methods. Bridging the gap is a promising future direction. As a beneficial side-effect, GHN-2 learns a strong representation of neural architectures as evidenced by our property prediction evaluation. Finally, parameters predicted using GHN-2 trained on ImageNet benefit transfer learning in the low-data regime. This motivates further research towards solving our task.
sections.heading: Acknowledgments
sections.text: BK is thankful to Facebook AI Research for funding the initial phase of this research during his internship and to NSERC and the Ontario Graduate Scholarship used to fund the other phases of this research. GWT and BK also acknowledge support from CIFAR and the Canada Foundation for Innovation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute: http://www.vectorinstitute.ai/#partners. We are thankful to Magdalena Sobol for editorial help. We are thankful to the Vector AI Engineering team (Gerald Shen, Maria Koshkina and Deval Pandya) for code review. We are also thankful to the reviewers for their constructive feedback.
title: Parameter Prediction for Unseen Deep Architectures