This use case sounds like it would require a lot of synchronization, which would most likely make the whole training procedure slow. Sorry, but I’m not sure if this is possible at all.
However, you could have a look at torch.utils.checkpoint 2 and see, if this might help you save some memory.