when I loaded a pretrained model and try to continue the training.I found when model executes optimizer.step() it cause error as following:
File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/optim/adam.py", line 110, in step
p.addcdiv_(exp_avg, denom, value=-step_size)
RuntimeError: output with shape [1, 256, 1, 1] doesn't match the broadcast shape [2, 256, 1, 1]

So I check the p.addcdiv by using try-except
However when breakpoint appears in the except case, I output the exp_avg and denom. I find they getting same shape.
denom.shape
Out[2]: torch.Size([2, 256, 1, 1])
exp_avg.shape
Out[3]: torch.Size([2, 256, 1, 1])

However the p.addcdiv_  still get same error information.

Dose output is different from exp_avg or denom?
I use two gpu to train it. And I have already used one gpu and two gpu to reloaded it and both failed.
So what should I do ?