Hi!
What I wondering is: is it possible to make self.weight and self.bias to be passed as the type of nn.Parameter in multi-GPU setting?
At the beginning of forward, self.weight.__class__ and self.bias.__class__ are torch.Tensor type at the multi-GPU setting, which looses access their member variable "fast" . In the single-GPU settings, self.weight.__class__ and self.bias.__class__ keep their type nn.Parameter at forward as they were made, thus accessing fast is not a problem. In other words, setting those to nn.Parameter inside of forward is not desired in this case.
Thank you.