DataParallel won’t help here if there is only one GPU, but you could use multiple CUDA streams. For example,
s0 = torch.cuda.Stream()
s1 = torch.cuda.Stream()
with torch.cuda.stream(s0):
    enc1 = self.encoder1(x1)

with torch.cuda.stream(s1):
    enc2 = self.encoder2(x2)