Solved by ptrblck in post #2 
              
              
                This is expected, since the result will be concatenated in dim0 again. 
Each device returns 4x4=16 values and thus the result of 4 devices is 4*4*4=16*4=64. 

Assuming the additional values are created as function between the samples in a batch, you could try to calculate them after nn.DataParallel …