Solved by ptrblck in post #2 
              
              
                You can deepcopy a model: 
model = nn.Linear(1, 1)
model_copy = copy.deepcopy(model)

with torch.no_grad():
    model.weight.fill_(1.)

print(model.weight)
> Parameter containing:
tensor([[10.]], requires_grad=True)

print(model_copy.weight)
> Parameter containing:
tensor([[-0.5596]], requires_grad=…