Solved by karanjeswani in post #2 
              
              
                The torch clamp function has a different gradient than what you are expecting here. Check this post : Pytorch Autograd gives different gradients when using .clamp instead of torch.relu. Maybe try using relu instead of clamp.