I believe it is Heng’s implementation of mask-rcnn for Kaggle DS bowl 2018. I met exactly the same error but couldn’t find the root cause. My situation is more weird as the loss.backward() fails occasionally. My assumption is that the gather operation in data_parallel was not performed correctly and some data are left on other GPUs that leads to such error. Any suggestions on how to debug a complicated network and what might be the cause are appreciated.