I cannot find any obvious errors in your code, so you might want to scale down the use case a bit and try to overfit a small dataset, e.g. just 10 samples, by playing around with the hyperparameters, such as the learning rate.
Once this is done, you could try to scale up the use case again.