Yuchen_Mu:

also in this case the model is obviously underfitted.


You should be able to overfit a single layer MLP, by increasing width and/or learning rate. Then revert towards your deep model and you’ll see how it stops working.