jungminc88:

Each GPU returns a 4 by 4 tensor as a result.
However, the final output of the model
sims = model(b_que_iis, b_que_ams, b_art_iis, b_art_ams)
return a tensor of shape (16,4).


This is expected, since the result will be concatenated in dim0 again.
Each device returns 4x4=16 values and thus the result of 4 devices is 4*4*4=16*4=64.



 jungminc88:

How should I solve this problem and get the desired output of (16, 16) tensor?


Assuming the additional values are created as function between the samples in a batch, you could try to calculate them after nn.DataParallel returned the output.