If your training fails with your model and custom dataset, I would recommend to scale down the problem a bit and try to first overfit a small data sample.
E.g. just take 10 samples out of your dataset and try to overfit these samples using your current code.
If that works, you could try to scale up the use case again by using more data.