I was able to resolve my issue.
I noticed that the files for the larger models were smaller than to be expected. Then I checked the logs of the training and the run crashed when saving the model due to OOM.