If you let me know, what’s the point of gathering back everything?
I mean,
you have batches x segments x others in a per-gpu level
if you gather everything back you will get
N *batches x segments x others in a global level
when you reshape you will get
N * batches * segments x others
But you are gonna get same ordering once you re-send everything to gpus.
Another problem I see is that if you have not enough memory to allocate everything in the same device, how do you expect to perform reshaping?