A visual proof that neural nets can compute any function
There are many caveats to this, esp. that this &quot;fact&quot; has nothing to do with whether training a neural network on a dataset will be useful.<p>There is often no function to find in solving a problem, ie., there is no mapping from ImageSpace -&gt; DogCatSpace. Ie., most things are genuine ambiguitites --- a stick in a water appears bent, indistinguishably from an actually bent stick in some other transparent fluid.<p>Animals solve the problem of the &quot;ambiguity of inference&quot; by being in the world and being able to experiment. Ie., taking the stick out of the water. A neural network, in this sense, cannot &quot;take the stick out the water&quot; -- it cannot resolve ambiguities. So that it can &quot;approximate functions&quot; is neither necessary nor sufficient for a useful learning system.<p>More significantly, a NN is a very very bad approximator of many functions. Consider approximating a trajectory <i>so that one can then find an acceleration</i>, ie., here we need f(x) to find d2f&#x2F;dx2 -- NN approximations are typically &quot;OK&quot; at the f(x) level and really really crappy at the df&#x2F;dx level, because the non-linear functions NNs just glue together are only trainable if theyre very rough.<p>For these, and lots of other reasons, this theoretical approach to learning is largely marketing none-sense. If you go out and actually study the only known systems which learn effectively (ie., animals), one does not find the need for universal approximator theorems in explaining their capacities.<p>These theorems <i>only</i> show that NNs are, like many computational statistical techniques, <i>sufficient</i> for a wide class of &quot;mere approximation&quot; problems that are only narrowly useful within the whole field of learning-as-such.