Bert Loses Patience - Batch Inference Doubt
In PABEE, with evaluate every layer output for a given patience threshold.

This can be done with if else when batch size is 1. 
How does this code work when batch_size > 1:
https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/examples/research_projects/bert-loses-patience/pabee/modeling_pabee_bert.py#L224

When batch size is greater than 1, I believe every input in the batch will have its own patience outcome. But torch.all would wait for the longest patience outcome.
@JetRunner 
Won't the patience counter dimension be equal to batch size ?
Thanks for your interest in our work Tanmay!

The code only considers the scenario when `batch_size=1`. Theoretically, we can do batch inference but as you noted we have to wait for the longest patience outcome anyway so it's not very useful. <|||||>@JetRunner Thanks for the prompt response! 
Is there any better way to handle batch size greater than 1? <|||||>I don't think so. If you really want to fully use the GPU, you can try multi-threads but I don't really recommend that. `bs=1` is fast enough to me.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.