Thanks for the quick reply. My further question is that, what is the number of self.device_ids, when I use 2 gpus?
if len(self.device_ids) == 1:
    inputs, kwargs = self.to_kwargs(inputs, kwargs, self.device_ids[0])
    output = self.module(*inputs[0], **kwargs[0])
else:
    inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
    outputs = self.parallel_apply(self._module_copies[:len(inputs)], inputs, kwargs)
    output = self.gather(outputs, self.output_device)