It’s hard to tell what might be failing during the training and I would recommend to scale down the problem and try to overfit a small dataset first (e.g. just 10 samples) by playing around with hyperparameters.