You may also be interested in this: https://github.com/IBM/pytorch-large-model-support 21
Large Model Support allows you to overcommit GPU memory by using host memory as a swap space for inactive tensors.