Thank you so much for such detailed explanation.
In my code, I use clone to do two things:

save model parameter whenever validation set performance reaches a new peak and restore them after several training rounds whose validation set performances stop improving(implement early stop). here’s my code:

#because my parameter size is small compared with GPU memory I don’t save it to disk
best_params = dict()
if map_score > best_map:
    for name, tensor in model.state_dict(keep_vars=True).items():
        #a large part of my parameter is embedding which is freeze during training so I don’t save it
        if tensor.requires_grad:
            #use tensor.data to empty its grad_fn then clone a new one to avoid being changed during training
            best_params[name] = tensor.data.clone()
            #if i understand correctly tensor.clone() will be ok too 
            #because no computation is performed with best parameters and no backward() is called. 
            #In such conditions having an none-empty grad_fn won’t cause any problem
model.load_state_dict(best_params, strict=False)


save model output when test model on the validation set. I divide validation task into several batches, save each batch’s ouput and concatenate them on GPU before transferring back to CPU here’s my code:

save_res = list()
for test_data in task:
    score_res = model(‘test’, test_data)
    #use tensor.data to empty its grad_fn then clone a new one to avoid being free
    #use tensor.data is essential because requires_grad need to be False when calling .numpy()
    save_res.append(score_res.data.clone())
    #i didn't use score_res.detach() because it will be deleted later
    #i think just score_res.detach() and no del score_res result in keeping graph in GPU memory
    del score_res
save_res = torch.cat(save_res).cpu().numpy()
do_some_eval(save_res)