Solved by hanbit in post #2 
              
              
                I’ve got an idea from the repository of official Mask RCNN. They cast any picklable object into byteTensor and gather it from each rank.