Thank you Olivier for looking into it. Your hunch on the learning rate was in right direction. However, the problem was rather simple. I am not sure anyone can run into this. It may be very basic about pytorch. That being said, at the risk of sounding stupid, here’s the problem.
overall_loss += loss.tolist()

before
 loss.backward()

was the issue. It wasn’t optimizing at all. loss.tolist() is a method that shouldn’t be called I suppose. The correct way to access loss is loss.item().
Now the network does what it should. Thanks everyone for looking into this and it’s less likely but hope this saves someone else’s time if somebody runs into this.