Optimized squad.py multi-threading
If there are few examples, don't bother multi-threading . The base cost of multi-threading in Python is expensive. I can't upload a picture of the call-stack visual profile because of my stupid company firewall, but multi-threading in python depends on serializing each object before sending it off. That process has a 5 second overhead coming from <method 'dump' of '_pickle.Pickler' objects>. Don't take my word for it - test it yourself. This small optimization reduces invocation run-time from 6s down to 1.1s for a single example inference where the len(examples) == 1. This optimization singular but it should be echoed across all pipelines in the Transformers repo.

ps: I also see a lot of lists with appends across the Transformers repository as a whole. This is a big speed suck. Look into using collections.deques more - deques are like lists but are highly optimized for appends.

Good Luck HuggingFace - you guys rock!
This was a big performance hiccup for us. We managed to take a `QuestionAnswerAnsweringPipeline` down from 12 seconds to 6 seconds (with CUDA) and then to ~1 second (with the serialization removal optimization).<|||||>The code quality check is literally just that one line is too long.<|||||>> The code quality check is literally just that one line is too long.

Have a look at the contributing guidelines, in particular step 5. https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests

Doing style and quality checks locally makes sure that your pull request doesn't get that annoying '1 failing' note.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Thank you, this is huge. On a g4dn.xlarge, with distilbert-base-cased-distilled-squad, I get about 60% speedup.