How does neural network learn feature functions? I was wondering if there is some intuitive way of thinking about the fact that neural networks learn the feature function $\phi(x_i)$ along with the feature weights $\theta$. Because this is really the point of using a net: they will find the relevant feature functions and tune their weights to obtain decent results. But when you're training a net you're only updating the weights matrices $W^{(l)}$ of the net.
I was thus wondering what exactly is referred (in terms of maths) when we say that the neural net is learning the feature function? The goal would be to somehow compare this intuition with the standard regression $ \hat{y}_i = \theta^T \phi(x_i)$. Where we have to define ourselves the feature function and regression techniques take care of finding the appropriate weights $\theta$.


what exactly is referred (in terms of maths) when we say that the neural net is learning the feature function?

This is about representation learning. Deeper layers act as learning  entities for new unknown representations:

A central idea, referred to as greedy layerwise unsupervised
pre-training, was to learn a hierarchy of features one level at a
time, using unsupervised feature learning to learn a new
transformation at each level to be composed with the previously
learned transformations; essentially, each iteration of unsupervised
feature learning adds one layer of weights to a deep neural network.[*]

[*] Representation Learning: A Review and New Perspectives, Bengio et. al 2014, pdf.
