Hi All,
I’m at a loss for the time being on how to proceed with debugging problematic loading of a GPU-trained model on CPU. The model is a generative flow that I sample time series information from. I can provide more detail as necessary, but starting more simply for now, here’s the steps of what I’m doing as I go by the documentation for loading a GPU net on CPU.
device = torch.device('cpu')
net = SDEFlow(device, obs_model, state_dim_SCON, t, dt_flow, n, num_layers = 5) #instantiates model, leaving arguments for future explanation if necessary
net.load_state_dict(torch.load(net_state_dict_save_string), map_location = 'cpu')

net.eval()
x, _ = net(batch_size) #Forward method of net outputs two things, and x is the matrix time series.

#various plotting code from here

and then I end up with a problematic figure that doesn’t align with my GPU results:

image900×864 241 KB

Re-loading the model on the device on which the training occurred
device = torch.device('cuda')
net = SDEFlow(device, obs_model, state_dim_SCON, t, dt_flow, n, num_layers = 5)
net.load_state_dict(torch.load(net_state_dict_save_string), map_location = 'cuda')

net.eval()
x, _ = net(batch_size)

#various plotting code from here.

I’m able to get back the results that I expect:

net_iter_105000_warmup_1000_t_5000_dt_1.0_batch_31_samples_31_layers_5_lr_0.01_warmup_lr_1e-06_sd_scale_0.25_SCON-C_CO2_logit_short_2022_01_21_05_31_02_test21920×1920 197 KB

I don’t think the bug is in my plotting code, since I can already tell things are wrong via print output of x on the CPU, so I didn’t include my plotting code for now either to avoid extraneous information.
Am I missing something obviously wrong with respect to the order that I’m doing things? I made sure that the flow model being instantiated was the exact same, so it appears that something is not being converted correctly in the saving of the state dict? In the training, I’m saving the net state dict along the lines of torch.save(net.state_dict(), net_state_dict_file_string). Evaluation batch_size is equivalent on both GPU and CPU.
Happy to provide more info. I appreciate your time and patience in reading through this question.