Hi @ptrblck, I have a question along similar lines.
I don’t think nn.DistrubtedDataParallel  behaves differently (printing shape before and after forward pass gives the same value i.e. per gpu batch size). What should I do, if I want to concatenate the outputs of the forward passes across all gpus while using nn.DistributedDataParallel ?
Note: I want this as I want to use the model as a feature extractor, and I want to cluster the embeddings (so I need all the data concatenated).