Solved by alepack in post #2 
              
              
                I think the problem is the last RELU activation. 
Placing that prevents the output of the net from being negative and this in turns prevents the softmax from working properly. 
With softmax I mean the one which is used in the loss (if you don’t consider the tricks added for numerical stability) 
Try…