Breaking Neural Network Scaling Laws with Modularity
Akhilan Boopathy Sunshine Jiang William Yue Jaedong Hwang Abhiram Iyer Ila R Fiete
Modular neural networks outperform nonmodular neural networks on tasks ranging from visual question answering to robotics. These performance improvements are thought to be due to modular networks' superior ability to model the compositional and combinatorial structure of real-world problems. However, a theoretical explanation of how modularity improves generalizability, and how to leverage task modularity while training networks remains elusive. Using recent theoretical progress in explaining neural network generalization, we investigate how the amount of training data required to generalize on a task varies with the intrinsic dimensionality of a task's input. We show theoretically that when applied to modularly structured tasks, while nonmodular networks require an exponential number of samples with task dimensionality, modular networks' sample complexity is independent of task dimensionality: modular networks can generalize in high dimensions. We then develop a novel learning rule for modular networks to exploit this advantage and empirically show the improved generalization of the rule, both in- and out-of-distribution, on high-dimensional, modular tasks.
scaling laws, modularity, neural network, generalization, compositionality, combinatorial generalization
We show theoretically that modular neural networks trained on modular tasks can generalize to high-dimensional tasks with a fixed number of training points; we propose a learning rule to exploit this advantage empirically.