I tried #1 however, I run into this issue while loading the state_dict back:
KeyError: 'unexpected key "module.cnn.0.weight" in state_dict'
Essentially, the model is nested in the module of DataParallel.
I guess it might be better to use #2 for saving?