Getting only <|endoftext|> token  in GPT-NEOX-20B model
### System Info

Transformer Version: 4.20.1
Python : 3.8
ubuntu : 18.04

### Who can help?

@patil-suraj  



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

 - We are creating a streaming app using HF transformer generate() but during the token decoding we are only getting " <|endoftext|> " . when an input prompt is passed, No tokens are generated from the model. The model we are using is GPT-NEOX-20B.

- We have tried 3 Tokenizers, GPTNeoXTokenizerFast, AutoTokenizer, and GPT2TokenizerFast but all of them returned the same output.
``
- Below is the generate parameters.
```
output_sequences = model.generate(
        input_ids=input_ids,
        max_length=200,
        temperature=temperature,
        top_k=k,
        top_p=p,
        repetition_penalty=repetition_penalty,
        do_sample=False,
        num_return_sequences=num_return_sequences,
        filename=filename,
        tokenizer=tokenizer,
        num_beams=1,
        use_cache=False
        
    )
```

-  where we are receiving Empty tokens, is https://github.com/huggingface/transformers/blob/main/src/transformers/generation_utils.py line 1740, 

We are trying to decode the next tokens.

# finished sentences should have their next token be a padding token
          ```
  if eos_token_id is not None:
                if pad_token_id is None:
                    raise ValueError("If `eos_token_id` is defined, make sure that `pad_token_id` is defined.")
                next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
                
            tensor_text = tokenizer.decode(next_tokens, clean_up_tokenization_spaces=True)

```
- A confirmation for the code in file https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py 
and line 146. what logic should be there. if use or if not? 

 `present = None if  use_cache else (key, value)`


### Expected behavior

The above code works well with all the models of the GPT-NEO  but with GPT-NEOX we are facing the <|endoftext|> tokens issues. Instead of <|endoftext|> tokens.  we need to generate the correct tokens.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.