While running training on BERT model with cluster of AWS with 16 GPUs (p2.16xlarge) I get run time error:
cuda runtime error (60) : peer mapping resources exhausted at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/THC/THCGeneral.cpp:141

If I’m using cluster with 8 GPUs (p2.8xlarge) it works fine.