Solved by wwiiiii in post #3 
              
              
                For Question 2, I tracked the scatter of inputs. 
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids) 
=> scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim) 
=> Scatter.apply(target_gpus, None, dim, obj) 
=> comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams) 
=> tuple(t…