Why train when you can optimize?
<i>Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)!</i><p>This is possible! One of the cool things about neural networks is that you can try to encode prior understanding into either the structure of the network or choice of activation function. See the paper <i>Neural Networks Fail to Learn Periodic Functions and How to Fix It</i> by Ziwin, Hartweg, and Uweda. Where they propose the activation function f(x) = x + sin(x)^2 that can encode an understanding that the underlying function should be periodic.<p>[1] <a href="https://proceedings.neurips.cc/paper/2020/file/1160453108d3e537255e9f7b931f4e90-Paper.pdf" rel="nofollow">https://proceedings.neurips.cc/paper/2020/file/1160453108d3e...</a>