[T5] Fix speed degradation bug t5
# What does this PR do?


Checking every value of a tensor for `inf` is expensive. This was added to T5 to allow for fp16 training, but should then also be used when the model is in fp16 to not slow down normal fp32 mode.

Using @dsgissin  script:

```python
device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
print(f"Using device: {device}")

t5_tokenizer = T5TokenizerFast.from_pretrained('t5-base')
t5_model = T5ForConditionalGeneration.from_pretrained('t5-base')
t5_model = t5_model.to(device)

t5_input_ids = t5_tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids  # Batch size 1
t5_input_ids = t5_input_ids.to(device)

import time
import numpy as np
N = 100
times = []
for _ in range(N):
  start = time.time()
  t5_outputs = t5_model.generate(t5_input_ids)
  end = time.time()
  times.append(end-start)
print(f"transformers version: {transformers_version}")
print(f"torch version: {torch_version}")
print(f"{1000*np.mean(times):.0f} ms \u00B1 {1000*np.std(times):.2f} ms per loop (mean \u00B1 std of {N} runs)")
```

with:

- Python 3.8.5
- PyTorch 1.7.1
- CUDA 11.1 on a NVIDIA V100 GPU

The time was improved from:
```441 ms ± 41.67 ms per loop (mean ± std of 100 runs)```
to 
```388 ms ± 44.75 ms per loop (mean ± std of 100 runs)```
> Looks good to me!
> 
> Some of the other library models also use this trick (BART-like models), we should also investigate those.

Good point - yeah, let me fix this in this PR actually