I want to collect tensors in all GPUs for each minibatch and save them.

Do you want all tensors to be on a single process before saving?
You can save a tensor using torch.save — PyTorch 1.9.0 documentation 1.