I got this error randomly on the cluster. I am clueless and don’t know what the problem could be. One possibility is because multiple jobs may share the same machine, and it may cause some problem. But it’s unlikely that the gpus are occupied because I printed the nvidia-smi in the log and they are fine before running the code.
 File "/home/code/utils/distributed.py", line 128, in synchronize
   dist.barrier()
 File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1710, in barrier
   work = _default_pg.barrier()
RuntimeError: CUDA error: out of memory

If someone has seen this error and found the solution, please let me know.
If someone has a hint why this happens let me know.
One more thing, this is running others’ code, I don’t know why they put this barrier before making the model distributeddataparallel and after dataloader initialization. Is barrier suggested to be used?