Solved by tom in post #2 
              
              
                Chapter 6 of our book that you can download here has some discussion on this. 
There are two main uses of the activations functions 

In the inner bits of the network, you need to have one to go beyond linear (i.e. to get the ability to approximate arbitrary functions),
at the final layer, the main …