any updates on this?
i am hitting the same issue.
gpu of 32gb is CUDA error: out of memory while i am still parsing the args… it is so weird.
the code runs fine on a gpu with 16gb and uses about 11gb on a local machine.
it happened at barrier. i am using ddp but using only one gpu.
dist.barrier()
import torch.distributed as dist
 call dist.barrier()
  File "python3.7/site-packages/torch/distributed/distributed_c10d.py", line 2524, in barrier
    work = default_pg.barrier(opts=opts)
RuntimeError: CUDA error: out of memory

any idea? this happened on a cluster with multi-nodes. i am using only one node with single gpu.
pytorch 1.9.0
cuda 11.1.1
gpu type: nvidia v100 volta
thanks