I’ve just tested the code on a V100 + CUDA10.2 and after the forward pass with a batch size of 100 approx. 23GB GPU memory will be used, which would explain why your swap is being filled.
You could reduce the batch size to save memory or use e.g. torch.utils.checkpoint to trade compute for memory.