  Over-parametrized deep neural networks trained by stochastic gradient descent
are successful in performing many tasks of practical relevance. One aspect of
over-parametrization is the possibility that the student network has a larger
expressivity than the data generating process. In the context of a
student-teacher scenario, this corresponds to the so-called over-realizable
case, where the student network has a larger number of hidden units than the
teacher. For on-line learning of a two-layer soft committee machine in the
over-realizable case, we find that the approach to perfect learning occurs in a
power-law fashion rather than exponentially as in the realizable case. All
student nodes learn and replicate one of the teacher nodes if teacher and
student outputs are suitably rescaled.

Disordered Systems and Neural Networks;Machine Learning