Fix TFEncDecModelTest - Pytorch device
# What does this PR do?

Fix pytorch device issue for `test_pt_tf_equivalence` in `TFEncoderDecoderMixin`.

@patrickvonplaten @sgugger 

## Remark:

There are other issues to fix for TFEncoderDecoder test pass:
  1. use small models for `test_real_model_save_load_from_pretrained`
  2. `test_bert2bert_summarization` test fails after #15562
 
Think these should be addressed in separate PRs (especially for 2.)
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15979). All of your documentation changes will be reflected on that endpoint.