You are right, the checkpoint file is from a DataParallel object. Does that mean I have to make one again and then load the weights?
I save like this:
checkpoint = {
                'run': self.run_count,
                'epoch': self.epoch_count,
                'model_state': self.model.state_dict(),
                'optimizer_state': self.optimizer.state_dict(),
                'accuracy_val': self.accuracy_val
            }
torch.save(checkpoint, 'filename.pth')

The strict=False is needed since I load weights from a ResNet with one fully-connected layer into a ResNet with a different fully-connected layer in the end.