Fix for case of multi-gpu
When loading the optimizer and the scheduler in a multi gpu, the loading will put the optimizer and scheduler in cuda:0, it might not have enough mem to temporary store them (until the to.device below).
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
