This would at least mean that the current training code seems to properly update the model.
Once you were able to overfit the data, you could scale up the problem slowly (i.e. add more data stepwise) and tune the hyperparameters of your model with it.