Hi, I’m trying to implement object detection code with apex distributed data parallel. Actually, many of my codes are following the apex official example 9.
However, the problem is that when evaluating ROC. To do that I have to set a threshold which makes the number of False Positives (FPs) to fixed-size. Therefore, I need to gather (or concatenate) all the objectness score of predicted bounding boxes from each GPU and have to sort them to decide the threshold.
Here, the problem is

Since the length of a tensor for objectness score is all different from GPUs I think I cannot use the function ‘distributed.all_gather’. Here’s my snippet.

scores = torch.from_numpy(scores).cuda()
tensor_list = [torch.zeros_like(scores) for _ in range(dist.get_world_size())
rt = scores.clone()
dist.all_gather(tensor_list, rt)



Then, how can I concatenate these tensors? I have only found ‘dist.reduce’ which enables summation, multiplication … but no concatenation. That’s why I use all_gather.


Is there any sample code or tutorial for object detection that I could refer to?