Hi ptrblck, I still do not think using buffer is a solution, unless I have the batch size in advance, and use it to declare the size of this buffer. After checking the code for the DataParallel module, and the associated methods of scatter, gather, replicate and parallel_apply, it seems to me that DataParallel will reference to the same module, so the attribute will be the same for different devices, and there can not be the same number of such attribute as devices. So the only reasonable solution is using buffer, but will cause the problem I mentioned above. Another problem is I can calculate gradient of this buffer with other parameters. I think another solution is to make the attribute I desired as output, but although simple this is not a very good solution. I wonder if there is other points that I missed and better solution to this. Thanks.