Is there a way to evaluate models during training in Multi-gpu setting
## ❓ Questions & Help

Hi all,
I always see comments in examples saying that "when single GPU otherwise metrics may not average well". So is this really something that shouldn't be done? I mean, is there a way to evaluate the model safely after each epoch in the multi-gpu training setting? Thanks.
<!-- A clear and concise description of the question. -->

Yes. After each batch has been completed by a GPU, you can store its results and corresponding labels in a shared space (e.g. CPU/memory). Then, when all batches are done, you can evaluate the epoch by calculating your metric/avg loss over all gathered results. It has been suggested to only keep track of the batch averages, but I feel like that is just an approximation of an approximation.

My code is a bit too bombastic to share, but I use something like this:

- custom Trainer class with a train/evaluate/test loop. It also has a 'performer' property
- the performer is an instance of a custom Performer singleton class that keeps track of losses, labels, and/or predictions at the end of each processed batch. Note that this means that each separate process keeps track of its own progress. Results aren't shared between processes until the end of the epoch
- at the end of each epoch, all results are `gather`ed from the different processes to a single one, which then calculates the average over all collected batches, and broadcasts that information (e.g. avg_loss, avg_secondary_metric) back to the other processes

It is rather complex and perhaps too much code and hassle for what it is, but for me it was more the learning experience of how to work with multi-GPU and `gather` with, as a bonus, fast evaluation and testing.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hey @BramVanroy  thank you for your reply and sorry for late reply. Unfortunately I am not sure if I understood what you suggested. Do you suggest to transfer predictions and gold labels of each bacth to the CPU  and then calculate metrics by using them if I want to evaluate my model during training ? 
As far as I see in most of the examples in the repo, it is okay to evaluate the model by using multi gpus once the training is over. They only do not suggest to eval it during training. There must be some reason for that ?  
 <|||||>I did not even know how to "evaluate models during training in single-gpu setting".