My machine has several hundred GB of free RAM during training so I do not think that is a factor…
Additional tests I have tried: switching to PyTorch 1.6 and using DistributedDataParallel() did not help significantly