I would generally try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparameters and make sure your model is able to do so. If this wouldn’t be possible your training script might have some issues, which would need to be debugged (e.g. forgetting to call optimizer.zero_grad() etc.). Once this is working, you could try to scale up the use case again.