2 slow TF T5 common tests failing on master
These should probably be run on CI somewhere.
Didn't know whether to assign @patrickvonplaten or @jplu.

These fail in both tf 2.2 and tf 2.3.


#### Command
```bash
RUN_SLOW=1 pytest tests/test_modeling_tf_t5.py -k saved
```
#### Traceback
```
================================================================================= FAILURES =================================================================================
__________________________________________________________ TFT5ModelTest.test_saved_model_with_attentions_output ___________________________________________________________
tests/test_modeling_tf_common.py:223: in test_saved_model_with_attentions_output
    self.assertEqual(len(outputs), num_out)
E   AssertionError: 5 != 4
_________________________________________________________ TFT5ModelTest.test_saved_model_with_hidden_states_output _________________________________________________________
tests/test_modeling_tf_common.py:185: in test_saved_model_with_hidden_states_output
    self.assertEqual(len(outputs), num_out)
E   AssertionError: 5 != 4
---------------------------------
```
Yeah, that's a know failure and I didn't manage to make it work yet with the `cast_bool_to_primite(...)` function<|||||>This should be fixed in the next big TF rework.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
