Thanks a lot @albanD . That is what I ended up doing indeed, avoid the copy_ for tensors in the gradient computation graph, and using it only in an additional buffer to save the tensor I need, where we use copy_(whatever_tensor.detach()) (because of DataParallel requirements).