The collective communications are designed for tightly coupled processes, to give you a better idea of how they work, I borrowed some images from mpi:



Each collective communication primitive is a blocking process for all processes in your created processgroup (whether it is the global WORLD group or a sub group of the global group).
I would suggest you do it in the following way:
import torch.distributed as dist

# make sure all processes in your_group will run this step
model = (model, ..., output_device=your_device, process_group=your_group)
your input per process= ...

# output is only visible to that process
# parameters are synchronized behind the stage
output = model(your_input) 

# make sure all processes in your_group will run this step
dist.gather(..., dst=0)
# perform ranking on process 0
dist.scatter(..., src=0)

# all workers are now synchronized

One crutial thing to notice: the receive buffer (tensor) must be equal or larger than the tensor you have sent, otherwise nasty errors will be thrown.