For Question 2, I tracked the scatter of inputs.
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids) 8
=> scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim) 2
=> Scatter.apply(target_gpus, None, dim, obj) 3
=> comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams) 1
=> tuple(torch. C. scatter(tensor, devices, chunk_sizes, dim, streams)) 2
=> scatter(tensor, devices, chunk_sizes, dim, streams); 3
=> tensor.chunk(/ chunks= /devices.size(), / dim= /dim); 1
=> self.split(split_size, dim); 1
=> self.narrow(dim, i * split_size, length); 1
It seems inputs are scattered to GPU in its original order, so aggregating the outputs of each GPU in GPU # order would be matched with original inputs. Indeed, I can find it at  gather of nn.DataParallel.