test_rag_sequence_generate_batch failing on CUDA
https://github.com/huggingface/transformers/runs/1157849932?check_suite_focus=true
```
>       self.assertEqual(output_text_1, EXPECTED_OUTPUT_TEXT_1)
E       AssertionError: 'The song peaked at number 17 in the' != '"I Know Him So Well"'
```
Yeah, If you run on CPU the test passes - I added a comment that the test fails on GPU: https://github.com/huggingface/transformers/blob/9e68d075a4100906509170498480823e7e61874a/tests/test_modeling_rag.py#L659

Beam search seems very sensible to small changes.<|||||>you mean sensitive, but OK. Maybe we should skip the test on CUDA so that slow CI isn't broken?<|||||>Actually, I fixed all `RagSequence` related bugs today and added better integration tests that should all pass on both CPU and GPU => so I think it's fine now.

See https://github.com/huggingface/transformers/blob/cf1c88e0921243e760d306e63a5938e1bac880f3/tests/test_modeling_rag.py#L664