DataParallel multiplexes the whole model across different GPUs. Saying that, the code would still work, but the .to operations would became a no-op, since the devices of self.linear1 and self.linear2 would be the same.