Thank you all for the answers. I guess it is just something I cannot avoid.
@ptrblck Can I use torch.utils.checkpoint on the whole model? Is there an example to see if I understood the docs correctly?