Try to overfit a small dataset, e.g. just 10 samples, and make sure your model is able to do so by playing around with some hyperparameters. Once your model is able to do so, scale up the use case again.