TF Beam Search generation seems to be flaky sometimes
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ALL TF generate models 

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
* [ ] all generate beam search tests in TF

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Some commits are failing due to `Beam size should alway be full` in circle ci - this should actually never happen. See a failed circle ci here: https://circleci.com/gh/huggingface/transformers/39780?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link .

## Expected behavior

Circle ci should not fail with this message.