Hey @irleader, I think the problem is in lines (there are multiple lines) that try to use self.device in the following way. DataParallel would replicate your model to all provides/visible devices. So, in the above case, there will be four thread, with which thread having one replica of self.PE_net on a different device. However, DataParallel is not smart enough to modify self.device for you. So self.device all all threads will point to the same device, which is the one you passed to ContactAttention_simple_fix_PE() ctor. Hence, there will be a device mis-match.
self.PE_net(pe.view(-1, 111).to(self.device))

If pe is a tensor, DataParallel should have already scattered it to the correct device. Is there any reason for calling .to(self.device) again?
cc @VitalyFedyunin