It seems like your input is on GPU 1, but your network is on GPU 0. From the error trace, it seems like the issue stems from how the only_forward function is defined and passed as a reference to self.forward.  This looks similar to the issue described in https://github.com/pytorch/pytorch/issues/8637 5 - check out the suggestion by Ssnl about why duplicating non-tensor objects in DataParallel could lead to these device mismatch errors.