ranger:

How to parallelise a pytorch model on GPU?


why to gather all outputs into GPU0  in step5 ,  it merely calculate distance of output_i with target_i ?  why not calculate it within each GPU separately   ?
one possible reason  :  all target (ground truth) are just stored in GPU0 , however ,  target can also be scattered as a addition of minibatch, like scatter data chunk  , in step1