Hi ptrblck,
Thanks for your quick answer and it works well now. But I found that nn.DataParallel doesn’t replicate module’s member variables. Specifically,  nn.Dataparallel does not work on self._coeffs since I checked that all the forward and backward operations with respect to self._coeffs are on cuda:0 .