MuGo: A minimalist Go engine modeled after AlphaGo
Something I&#x27;m finding interesting is how so many of these deep-learning neural nets use RELU for their activation function; RELU is known as a &quot;lazy engineer&#x27;s activation&quot; function - very simple to implement, and despite looking like a hack, seems to work very well for many tasks.<p>I tend to wonder - beyond &quot;ease of implementation and good &#x27;nuff&quot; reasons - if there are other reasons to use RELU, over other activation functions like TANH or Sigmoid?<p>I&#x27;m beginning to suspect that we may be seeing the &quot;engineering side&quot; of neural networks coming into play; that instead of using the more &quot;biologically accurate&quot; activation of the sigmoid function, we instead use RELU (and other ELU derivatives) because it works well, and is easier to understand?<p>Much like how things progressed better in heavier-than-air flight once engineers realized that flapping wings weren&#x27;t absolutely needed, and low-weight engines turning propellers, with fixed wings, worked better for flying than what nature uses...?