Not sure about the cause of the problem, but I would change the order to:

construct a single-GPU model
load the weights to the single-GPU model
parallelize the model to multi-GPU format (DP or DDP)

In your case, it could be:
model.to(device, ...)
model.load_state_dict(...)
model = nn.DataParallel(model, ...)