Thanks @ptrblck . I was about to do this after reading this suggestion of yours somewhere else, when miraculously it worked now (touch wood).
The only thing I changed was the way I save and load the model like this (I’m using DataParallel):
model = MyModel(config, vocab)

model.to(config.device)
print(config.device)
if torch.cuda.device_count() > 1:
    model = nn.DataParallel(model)
model.module.load_state_dict(torch.load(load_path))
model.eval()

And save:
torch.save(model.module.state_dict(), best_path)

In other words, I’m saving the state dict directly rather than it wrapped in the module.
Does it make any sense that this could be causing the issue?