I’m getting this same error using different models in Detectron2 (DeepLab and now MaskFormer). Does anyone have a solution to this other than checkpointing and re-trying? Maybe a different communication backend or something?
terminate called after throwing an instance of 'std::system_error'
what():  Transport endpoint is not connected
Traceback (most recent call last):                                                                 
File "/train_net.py", line 264, in <module>                          
    launch(                                                                                        
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/detectron2/engine/launch.py", line 67, in launch                                                                           mp.spawn(                                                                                      
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn                                                                    
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')                   
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes                                                              while not context.join():                                                                      
File "/opt/conda/envs/detectron2/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 130, in join                                                                     
    raise ProcessExitedException(                                                                
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGABRT