Speedup tf tests
Fyi @sgugger @patrickvonplaten @stas00, I'm temporarily marking these tests as slow as they take more than 1h30 minutes and prevent the CI from completing, therefore preventing any relevant information from the TF tests.

I'm working on improving the CI times so this is temporary (will revert by Friday). 
This single change made the tests on the TF CI pass from 6+ hours (non slow) to 29/31 minutes: https://github.com/huggingface/transformers/actions/runs/635711849

Will look for a way to reduce their time.