I would recommend to try to overfit a small data sample (e.g. just 10 samples) and make sure your current model is able to do so e.g. by playing around with some hyperparameters. Once this is done, you could try to scale up the use case again.