"Only evaluate when single GPU otherwise metrics may not average well"
Hi,

The script examples/run_lm_finetuning.py skips evaluation on the validation dataset when run in distributed mode on multiple GPUs. The code includes this comment regarding this:
"Only evaluate when single GPU otherwise metrics may not average well"

I'd appreciate it if someone could explain this issue in a few words and a maybe suggest a way around it? Can I simply run the evaluation on a single GPU (e.g. only for local_rank==0)?

Thanks!


I have wondered about this comment as well. I have implemented multi-GPU evaluation and it works perfectly fine. By evaluation I mean that the the work of evaluating is distributed and all results are then gathered to the main GPU (e.g. 0) or CPU which then calculates loss and secondary metrics (f1/pearson). I haven't experienced any issues with it but perhaps there is a reason that I don't know about. <|||||>Thanks, @BramVanroy. BTW, since you mentioned CPU, did you succeed in distributing LM fine-tuning on multiple CPUs? I tried that using torch.distributed and the 'gloo' backend and it seemed to be working fine, except that the total speed hardly improved.<|||||>Oh no, what I meant was doing some calculations such as correlations in the CPU. I've never done fine-tuning on CPU. I can imagine that it takes a long time. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@BramVanroy I was thinking of doing the same. Can you point me to your training script which uses multi-gpu eval the way you have described? I am hoping that if I see that first, I can avoid common mistakes. 
Thanks!<|||||>@dhruvdcoder Unfortunately, that code is in no state to be made public, in part because it is too complex and not incredibly well written. If I find the time, I plan to improve it and to add a PR here to update the example scripts to make use of multi-GPU evaluation.