  Deep learning empirically achieves high performance in many applications, but
its training dynamics has not been fully understood theoretically. In this
paper, we explore theoretical analysis on training two-layer ReLU neural
networks in a teacher-student regression model, in which a student network
learns an unknown teacher network through its outputs. We show that with a
specific regularization and sufficient over-parameterization, the student
network can identify the parameters of the teacher network with high
probability via gradient descent with a norm dependent stepsize even though the
objective function is highly non-convex. The key theoretical tool is the
measure representation of the neural networks and a novel application of a dual
certificate argument for sparse estimation on a measure space. We analyze the
global minima and global convergence property in the measure space.

Machine Learning;Machine Learning