The parallel methods are used in e.g. nn.DataParallel to scatter and gather the tensors and parameters to and from multiple GPUs.
Generally speaking, the data and model have to be on the same device, if you want to execute an operation on both of them.
I’m not sure to understand your use case completely, but you could have a look at nn.DistributedDataParallel and see, if this implementation would work for you.