Sure, let us know, if you figured out the issue.
Based on the error message my best guess is that some internal buffers using in the optimizer are not pushed to the right device, so maybe an optimizer.cuda() method was implemented in this repository?
If not, try to push the model to the device before passing it to the optimizer.