I eventually used autograph.graph to implement custom save_tensor_hook with some sort of grad-checkpoint based on the application specifics. It worked very well now, and thanks for the tip on the autograd.graph!