What might be happening is that you network’s capacity might be too much for a very simple dataset which is why it is overfitting. I would recommend having a validation and use early stopping based on that.