need to read:

  
    
    
    How does one obtain gradients as data efficiently? 
  
  
    the whole code just for reference: 
'''
This script is written to test:
 1)what happens if we differentiate through pytorch's backward pass and to see
    if its consistent with the doing a second derivative
'''
import torch
import torch.nn as nn

from grad_test_sympy import symbolic_test

import sys

from pdb import set_trace as st

def test_does_backward_backward_match_second_derivative(experiment_type,grad_computation_type='dl_dw_zero_grad'):
    '''
     1)what happens if we differentiate th…
  


and

  
    
    
    Clone and detach in v0.4.0 
  
  
    We’ll provide a migration guide when 0.4.0 is officially released. Here are the answers to your questions: 


tensor.detach() creates a tensor that shares storage with tensor that does not require grad. tensor.clone()creates a copy of tensor that imitates the original tensor's requires_grad field. 
You should use detach() when attempting to remove a tensor from a computation graph, and clone as a way to copy the tensor while still keeping the copy as a part of the computation graph it came from. …