I also see that, after applying torch.no_grad() time for backward pass reduced too much. So I think adding torch.no_grad() we may miss some computation and might not be better approach. @JuanFMontesinos Thank you for your time and suggestions. At last my quick ques is that do we really need to keep track of this kind of local variables in computation graph??