Following up on this, I’ve updated my CPU PyTorch version to 1.10.2
>>> import torch
>>> print(torch.__version__)
1.10.2

As I’ve mentioned, the version I have access to on my GPU computing cluster is locked.
After adding the _use_new_zipfile_serialization=False argument and re-loading the state_dict to the model, the CPU output is better, but still off.

image1179×864 265 KB

Meanwhile, instantiating a new model and re-loading the state_dict on the GPU gets back what I expect:

net_iter_105000_warmup_5000_t_5000_dt_1.0_batch_31_samples_31_layers_5_lr_0.01_decay_step_5000_warmup_lr_1e-06_sd_scale_0.25_SCON-C_CO2_logit_short_gpu_save_2022_01_28_11_03_13_GPU_test1920×1440 157 KB

I’ve made sure that the model module codes on CPU and GPU are the exact same! It’s even more odd to me now that things weirdly improved after two things changed (CPU PyTorch version and saving argument addition), but aren’t correct yet.
Does this point to my needing to find some way of getting PyTorch updated past 1.10.0 on the GPU cluster I’m using?
Thanks again, @ptrblck.
EDIT: I was able to get the PyTorch install updated to 1.10.2, but still using cudatoolkit = 10.2. Not able to use cudatoolkit = 10.3 on my institution’s GPU cluster. Will re-run my trainings and revert back.