Thank you so much for your reply!
There is any way to make these processes computed parallel?
My network is ensemble model, each branch network of my model quite small, I test model with ONE branch and 32 branch, the GPU utilization is the same and quite low (~10%). I think torch.cat all last layers of branch networks then feed to ONE optimizer will make computation graph consider them as parallel process but it seem to be not as you said (CatBackward just be returning a few views of the  grad_out)
def forward(self, x):
        return torch.cat([self.model[i](x[i]) for i in range(number_branch_net)])

Model is a moduleList, model[i]s are similar.
Thanks.