I am trying to extract gradients and later use them as data (so have them non-trainable). For that I am having to call .backward() twice. Doing that seems to mess up with my code as noted by mother question ( What does doing .backward twice do? 2). However, for the code to work how I intend it (without the issue of calling backward twice) I am having to do clone the model parameters:
            ww = torch.randn(w.size(), requires_grad=True)
            ww.data = w.clone()
            loss = (ww*x-y)**2
            loss.backward()
            grad = ww.grad
            assert( (2*(w*x-y)*x).item() == grad.item() )

in this silly toy example its not an issue but I am worried that this might become an issue later. Anyone know how to get the gradients without having to clone the data and create extra vectors?

all the code:
    if experiment_type == 'grad_constant':
        ## computes backard pass on J (i.e. dJ_dw) assuming g is constant
        # compute g
        if g_hard_coded:
            grad = 2*(w*x-y)*x # assumption, loss = (w*x - y)**2
        else:
            ww = torch.randn(w.size(), requires_grad=True)
            ww.data = w.clone()
            loss = (ww*x-y)**2
            loss.backward()
            grad = ww.grad
            assert( (2*(w*x-y)*x).item() == grad.item() )
        g = grad.item()
        # compute w_new
        w_new = w - (g+w**2) * g
        # compute final loss J
        J = (w_new + x2 + y2)**2
        # computes derivative of J
        J.backward()
        #dw_new_dw = w_new.grad.item()
        dJ_dw = w.grad.item()


Cross posted:

https://stackoverflow.com/questions/58359410/how-does-one-obtain-gradients-as-data-efficiently-in-pytorch 1
How does one obtain gradients as data efficiently?
https://www.quora.com/unanswered/How-does-one-obtain-gradients-as-data-efficiently-in-PyTorch 1