RunTimeError when using prefix_allowed_tokens_fn and top-k/top-p sampling in model.generate
Hi, I'm using T5-large, torch 1.9 and transformers 4.8.2 to generate sentences in the predefined corpus. I tried to use both `prefix_allowed_tokens_fn` and top-k sampling, `do_sample=True; top_k=50`. When I use the two separately, I don't get any problem, but when I try to use the two together, I get a runtimeerror, `RunTimeError: probability tensor contains either 'inf', 'nan' or element<0`. Do you have a guess on why this happens?

The simple code I used for generation is 
```
with open("corpus.pkl", "rb") as f:
   trie = dict(pickle.load(f))

def get(input_ids: List[int]):
   return _get_from_trie(input_ids, trie)

def _get_from_trie(input_ids: List[int], trie_dict: Dict):
   if len(input_ids) == 0:
      output = list(trie_dict.keys())
      return output
   elif input_ids[0] in trie_dict:
      return _get_from_trie(input_ids[1:], trie_dict[input_ids[0]])
   else:
      return []

model = T5ForConditionalGeneration.from_pretrained('t5-large').eval()

outputs = model.generate(
   **input_args.
   num_beams=5,
   prefix_allowed_tokens_fn=lambda batch_id, sent: get(sent.tolist()),
   do_sample=True,
   top_k=50
)
```


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.