The Readme actually clarified a lot of things for me that many longer texts skip over - why ReLU&#x27;s are used (they prevent vanishing (or exploding) gradients), exactly how dropout works, why there is &quot;momentum&quot;, etc.<p>I&#x27;m still not sure why he uses softplus instead of ReLU though. The implication is that it is better to have a smooth function, but is it? And does the benefit outweight the extra computational burden?<p>Also, the code is fantastically short.