Unstable generation results when using Top-p decoding
### System Info

I trained a T5-base model for a translation task. I use Top-p decoding strategy to generate text. 

Something is weird in the model; every time I ask the model to generate text for the given same input, it generates a different text. When I fix the random seed, the model starts to generate the exact text every time.

 My question is, why the model generates different texts for the same input if the random seed is not fixed? Top-P decoding strategy has no randomness.

```
y = data['target_ids'].to(device, dtype=torch.long)
ids = data['source_ids'].to(device, dtype=torch.long)
mask = data['source_mask'].to(device, dtype=torch.long)

generated_ids = model.generate(input_ids=ids, attention_mask=mask, max_length=512, do_sample=True, top_p=0.9, top_k=0, num_return_sequences=1) 
```
@patrickvonplaten 

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I just trained the T5 model on a translation task and then generate text using the above code.

### Expected behavior

The model generates the same text even if I don't fix the random seed.
Gently pinging @gante here<|||||>Hi @bilalghanem 👋 

Top-p has randomness -- when using `top_p=0.9`, `generate()` picks a token among the top candidate tokens, where the sum of the probability of those top candidates is >= 0.9. In other words, unless you model predicts a token with probability > 0.9 at each generation step, it will not be deterministic.

I'd recommend reading this blog post: https://github.com/huggingface/blog/blob/main/how-to-generate.md<|||||>> 

Thanks @gante.
Can you clarify, how it won't be deterministic if we don't find a token that satisfies the P condition? <|||||>@bilalghanem It considers all tokens, from most to least likely, such that its summed probability is `top_p`, and samples (proportionally) from those tokens.

Consider the following logits array: `[0.5, 0.4, 0.1]`. If you run sampling with `top_p=0.9`, it will pick the first token `(0.5/0.9)*100 = 55.6%` of the times, the second token `(0.4/0.9)*100 = 44.4%` of the times, and the last token `0%` of the times.

If you want a deterministic behavior, use `do_sample=False`. Again, I'd recommend reading the following blog post, which explains all of this: https://github.com/huggingface/blog/blob/main/how-to-generate.md

<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.