dahyun:

Here, the easiest way is to code (somehow magically) to replicate the network weight and bias of nn.Parameters instead torch.Tensor during nn.DataParallel.forward .
Is it possible somehow? I sincerely appreciate your advices.


Could you explain a bit, what you mean by this approach?
How are you setting the parameters to the initially empty attributes?
Note that you cannot set these parameters in the forward method using nn.DataParallel, as these parameters will only be registered in the replica models, not the root model.