Performing optimization on CPU
In README.md, it is explained that the optimization step was performed on CPU to train a SQuAD model. 

> perform the optimization step on CPU to store Adam's averages in RAM.

https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-bert-large-on-gpus

Is it still supported in `run_squad.py`?
If I understand correctly, the current implementation always places the Adam's averages on GPUs if we train the model using GPUs.
Oh yes you are right, we removed that, I'll update the readme<|||||>Thank you for your reply!
Because Adam's averages (i.e., `next_m` and `next_v`) are large, it is very helpful to support performing optimization step on CPU to reduce GPU memory.
Therefore, I would like to know how you implemented this.
Did you move the parameters to CPU (e.g., `grad.to(torch.device('cpu'))`) in `BertAdam.step` function?<|||||>@ikuyamada In the old repository, the model parameters on gpu obtained by forward and backward are copied to the optimizer that stores the model parameters on cpu, and the updated model parameters are returned to the model on gpu at each training step.
Check https://github.com/huggingface/pytorch-pretrained-BERT/blob/v0.3.0/examples/run_squad.py#L682

@thomwolf I have questions. Why did you decide to delete `optimize_on_cpu`? Does cpu and gpu parameter transfer affect training speed?
<|||||>@ryonakamura thank you for the pointer!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
