The code looks generally alright, so try to play around with some hyperparameters such as the learning rate.
If you get stuck, scale down the problem a bit and try to overfit a small dataset (e.g. just 10 samples) first to make sure there are no hidden bugs in the rest of the code.