Is loss.mean() needed?
In `run_classifier.py`, there is a:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_classifier.py#L841-L842

However, a couple of lines higher, the logits are already flattened
https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_classifier.py#L834-L839

So I assume the loss returned will only be a single-element tensor, correct? I also tested this code on 2 gpus and there indeed is only one single-element tensor.
This line is used when people use multi-gpu in a single python process (parallel instead of distributed). This is not the recommended setting (distributed is usually faster).

I wrote a blog post on this (parallel/distributed and the like) a few months ago: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255<|||||>So on a multi-GPU machine, if we run `run_classifier.py`, by default it uses the distributed setting, correct? I'm assuming the answer is yes because when I ran this code on a 2 gpu machine, the `loss` was already a single-element tensor before the `.mean()` call.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
