Sorry for the very late reply, I couldn’t use the GPU to test the code till today.
I used the following code to compare the two loaded models on the cpu and the gpu
import argparse
import torch
from torch.autograd import Variable
import scipy.io as sio
import os
import utils

parser = argparse.ArgumentParser(description="Pytorch DRRN Eval")
parser.add_argument("--model", type=str, default="dense_A_t_J_w_regW", help="model path")
parser.add_argument("--ep", type=int, default=24, help="epoch path")

opt = parser.parse_args()

save_path = os.path.join('/home/krm/ext/Dehazing/results/', opt.model)
utils.checkdirctexist(save_path)

model_path = os.path.join('model', opt.model, "model_epoch_{}.pth".format(opt.ep))

model, epoch, dict = utils.load_checkpoint(model_path, iscuda=False)
model_cuda, epoch, dict = utils.load_checkpoint(model_path, iscuda=True)

not_equal = []
for (name, p), (name_cuda, p_cuda) in zip(model.named_parameters(), model_cuda.named_parameters()):
    if not (p==p_cuda).any():
        not_equal.append(name)

print("not_equal = ")
print(not_equal)

Apparently they are identical. I have the problem with some test images as well, where I’m directly reading the images using opencv. So, there is no problem with NaNs or Infs in the input.
The problem is not consistent as well, sometimes it happens and sometimes it doesn’t, even for the same exact input in testing! the output is not the same when I run the code several times on the cpu. It is always consistent on the gpu, however.
I don’t know but could there be somehow a problem with the OS?