@J_Johnson  Thanks. Let me explain my process more specifically. I want to achieve this process:
For a given large dataset, I slice the dataset into minibatches, and run them batch by batch in the forward process. (Yes, no batch of data will run backward before all batches finish, which result in a lot of computation graphs reside in GPUs, and that’s why I use torch.autograd.graph.save_on_cpu to put the generated tensors back to CPU). After all batches finish FP, I will place the output tensors from CPU back to GPU and run BP.
My current question is that: In the backward pass, I don’t know how to put the output tensor from CPU back to the corresponding computation graph place in GPU. I think PyTorch might preserve certain hook APIs for me to achieve this, and I wonder if you could give me any suggestion.