why to gather all outputs into GPU0 in step5 , it merely calculate distance of output_i with target_i ? why not calculate it within each GPU separately ?

It’s more like an implementation side-effect. In fact, you can do what you proposed, but you would have to rewrite your code then to compute the loss inside the model – usually, the loss is computed in the training loop based on the outputs of the model.
I mean, it’s not a problem at all to do what you propose, but it’s a bit less convenient, because when you decide to use DataParallel (occasionally), you would have to modify your model code as well.
Also, computing the loss is super cheap, so there is not much speed gain in distributing it across the devices, I suppose.