Wrong answers from Longformer model even on simple questions
I am using the pretrained model `allenai/longformer-large-4096-finetuned-triviaqa`, however upon inspecting it on my system and the demo on the huggingface website, the outputs seem off even for very simple examples and samples from the dataset.

1. [Example 1](https://www.dropbox.com/s/9h3dcqpwq0n1b05/download%20%283%29.png?dl=0)
2. [Example 2](https://www.dropbox.com/s/40e93m2odix8x1p/download%20%284%29.png?dl=0)
3. [Example 3](https://www.dropbox.com/s/s5t1k6jluyzfs33/download%20%286%29.png?dl=0)
4. [Example 4](https://www.dropbox.com/s/oyps5a5gr2e4c25/download%20%287%29.png?dl=0)

Other models for QA (like `bert-large-uncased-whole-word-masking`) get such simple examples right.

Yeah this is related to a bug, see: https://github.com/huggingface/transformers/pull/4615

cc @mfuntowicz @julien-c - we should refactor the squad preprocessing in pipelines to make longformer work.<|||||>Hi @patrickvonplaten: are there are any updates with respect to this?<|||||>We will probably start working on a fix in ~2 weeks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
