Ok. A side question.  Do you know if all_gather gathers tensors in the same order? I need to all gather 2 tensors and the they have to be all gathered in the same order so that item i in the first list can be matched to item i in the second list.