title: Modelling the influence of data structure on learning in neural networks
abstract: The lack of crisp mathematical models that capture the structure of real-world
  data sets is a major obstacle to the detailed theoretical understanding of deep
  neural networks. Here, we first demonstrate the effect of structured data sets
  by experimentally comparing the dynamics and the performance of two-layer
  networks trained on two different data sets: (i) an unstructured synthetic data
  set containing random i.i.d. inputs, and (ii) a simple canonical data set such
  as MNIST images. Our analysis reveals two phenomena related to the dynamics of
  the networks and their ability to generalise that only appear when training on
  structured data sets. Second, we introduce a generative model for data sets,
  where high-dimensional inputs lie on a lower-dimensional manifold and have
  labels that depend only on their position within this manifold. We call it the
  *hidden manifold model* and we experimentally demonstrate that training
  networks on data sets drawn from this model reproduces both the phenomena seen
  during training on MNIST.
authors: S. Goldt M. Mézard F. Krzakala L. Zdeborová
hIndex: M. Mézard 64