Hi Silk!



 Silk:

It seems that the process behind the network are linear (at lest most of the time)


This really isn’t true.  And if it were true, networks would be much less
broadly useful.
I think it might look like networks are “mostly linear” because we
have these big, multidimensional, fully-connected layers with lots of
parameters, whereas the nonlinearities (usually) occur in small,
“simple,” element-wise nonlinear “activations.”  That is, we use
linear layers to mix elements together (by adding them), and only
introduce nonlinearities element-wise.
But the linear layers and the activations work together to produce
much more general nonlinearities.
You might ask why we don’t “mix” elements together by multiplying
them instead of adding them.  But in fact we do.  Consider the
nonlinearity f (x) = x**2.  If we mix a and b by adding them,
a + b, then passing the mixture through the nonlinearity gives
f (a + b) = a**2  +  2 * a * b  +  b**2.  Note the a * b term.
We have mixed a and b together by multiplying them – we just
used a linear layer together with an element-wise nonlinearity to
do so.
One could imagine a neural-network framework where the “mixing”
layers also included cross-products as well as sums.  I don’t really
know the history of neural networks, but I imagine that making such
a cross-product layer general would require many more parameters
and that it turns out to be more manageable and efficient to rely on
the linear-layer / element-wise-nonlinearity division of labor to
generate those cross-products (and higher-order terms).


so how can a network learn something that is not directly related to the weighted sum of the input (through one or more layer).


Because the “weighted sums” together with the nonlinear activations
are able to create products of elements (as well as higher-order terms).


Let say, is it possible to build a network that would do nothing more than find the maximum value of an input vector built from random Gaussian noise ? How would you do that ?


Build a network with at least one “hidden layer” (and probably more),
but, of course, with nonlinear activations in between the linear layers.
However, this problem is not a good example.  If the goal, to choose
something precise, is to find the index of the largest value in the
input vector (compare to predicting the class label in a multi-class
classification), then you just use an “empty” network that does nothing,
and calculate the argmax() of the “output” vector (which is equal to
the input vector).
We could set up a really artificial problem where we try to forbid the
network somehow from using this obvious information, but doing so
would just confuse the issue.
Best.
K. Frank