While I understand you might get frustrated seeing your use case fail, I would like to ask you to use the proper language in this board, please.
I would recommend to scale down the use case and try to overfit a small data samples (e.g. just 10 samples) by playing around with the hyper-parameters of your training (learning rate, weigth decay etc.).
Once your model can properly overfit the sample, you could try to carefully scale up the use case again by adding more data and observing the training.