My apologies. The error is actually coming from a 3rd party transformer library. I saw the error was coming from optimizer.step() so I didn’t look in more detail. The error stack trace is pasted below, but I will ask in their repo. Thanks!
File "file.py", line 71, in optimizer_step
optimizer.step()
File ".../lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
return wrapped(*args, **kwargs)
File ".../lib/python3.7/site-packages/transformers/optimization.py", line 155, in step
exp_avg.mul_(beta1).add_(grad, alpha=1.0 - beta1)
RuntimeError: expected device cuda:0 but got device cpu