Sorry for the delayed response. I followed your suggestion(ran on a small subset of 100 data, increased and decreased learning rate) and ran the script several times. My model does not even overfit on train data. Seems like backwards() not working. Although when I switch to the actual implementation, it works again.