The problem crashed with the initialization error. From the call stack, it seems like it crashed when it tries to release some resource. The whole program is kinds of large. The basic logic is that, it first calls train() which creates the model, and run the training. Then, it calls the predict(), which also creates the model and run the prediction. The program is running distributed in 8 GPUs. Each GPU only predicts part of the test files and then rank=0 will concatenate those files.
The crash is at the prediction phase. Only one GPU crashes with this error and all other GPUs can finish the predictions well.
Any idea on the potential issues?
pytorch = 1.7.1
cuda = 10.2
image3568×768 337 KB