Regarding the first part, calling torch.cuda.synchronize() did not help. It still seems that using 4 GPUs instead of 1 makes the code 15 times slower.
After fixing the second part, my output for
print(model.module.first_term(input_)[0])

is always on the first worker:
tensor([0.0020], device='cuda:0', grad_fn=<SelectBackward>)

So it is not even sharing the work between the different worker.