Diverse number of return sequences for greedy search and sampling generation
# What does this PR do?

A new option proposed, `diverse_sequences`, for cases, when one wants really different sequences to be generated (conversational bot, for example). For greedy search, it starts generating new sequences from top `num_return_sequences` tokens (as first tokens in sequences). For sample generation mode, `num_return_sequences` first tokens are taken from a multinomial distribution.
Default `diverse_sequences=False` leaves generation in a way it was before this PR.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

 GPT2: @patrickvonplaten
 Text Generation: @TevenLeScao
 T5: @patrickvonplaten


Hey @LSinev, 

thanks for your PR! Since the `generate()` refactor we are really trying to not add any use-case specific code anymore to the individual `generate()` methods. This quickly led to an unmaintainable code previously (especially if start adding lots of those `if` statements) again, so we can't merge the PR as it is in this state. It would be ideal if we could just add a new `LogitsPreprocessor` class or a `LogitsWarperClass`. 

If this is not sufficient, we have to think a bit more about how to add this PR. 
One thing, I don't really understand is how greedy search with `diverse_sequences=True` is different from Beam Search with `num_return_sequences > 1` -> it seems to be the same thing for me...Also could you add some links/pointers (paper, blog, other codebase) that makes use of this method? <|||||>> It would be ideal if we could just add a new LogitsPreprocessor class or a LogitsWarperClass.

Ok. I will check if it is possible (but this can move `if` statements inside, as I have to check processing of first token somehow).

>  how greedy search with `diverse_sequences=True` is different from Beam Search with `num_return_sequences > 1` -> it seems to be the same thing for me...

Never thought about this. Will check.

> Also could you add some links/pointers (paper, blog, other codebase) that makes use of this method?

Nothing openly available as far as i know. Because of `transformers` popularity, if such possibility not implemented, few developers will try these ideas. Main usecase is additional ranking of generated sequences. As for now, nothing stops to have exactly same sequences as output. It can also be used with probabilities of final sequences from second head of GPT2DoubleHeadsModel, for example (https://github.com/huggingface/transformers/issues/5164). <|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.