What do you mean by “significantly”?
Maybe not populating LitModel has effect on initializing model parameters after it.
Try setting random seed to fix number right before populating your actual model (after part that you comment ), and see if there is different still.
random.seed(args.seed)
torch.manual_seed(args.seed)
np.random.seed(args.seed)
cudnn.deterministic = True