Actually the print out is a little messy, and it seems that there is some conflicts among the four GPUs to print out values:

a:  a: a:   tensor([140339592887176], device=‘cuda:7’)
tensor([140339592887176], device=‘cuda:6’)tensor([140339592887176], device=‘cuda:5’)a:
b:  b: tensor([140339592887176], device=‘cuda:4’)b:  tensor([2018], device=‘cuda:7’)
tensor([2018], device=‘cuda:5’)tensor([2018], device=‘cuda:6’)
b:  tensor([2018], device=‘cuda:4’)
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535493744281/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535493744281/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535493744281/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
a:  tensor([140339592887176], device=‘cuda:4’)a:
a: a:   b:  tensor([140339592887176], device=‘cuda:6’)
tensor([2018], device=‘cuda:4’)
b:  tensor([2018], device=‘cuda:6’)
tensor([140339592887176], device=‘cuda:5’)tensor([140339592887176], device=‘cuda:7’)
b: b:   tensor([2018], device=‘cuda:5’)tensor([2018], device=‘cuda:7’)
a:  tensor([140339592887176], device=‘cuda:4’)
a:  b:  tensor([140339592887176], device=‘cuda:6’)

I can only understand this by that different layers are assigned to different GPUs, or maybe the model is assigned multiple times to different GPUs. I do not quite understand parallel computing.