Hi, @albanD
Thanks for the explanation! I’m curious that is there any simple way that I can offload the weight / intermediate output to CPU memory while getting back to GPU memory when it needs to (e.g., during backward, or at other place some op needs it)? Here I mean the true offloading that can save GPU memory.
I’m trying to make it by autograd function, like in-placement update on tensor data, but still not work (some backward error on gradient format)