Llama-2-hf non stopping token generation.
### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-1035-azure-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

model_id = "/home/modelweights/llama2-hf-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
  model_id,
  device_map='auto',
  load_in_8bit=True,
  )

pipe = pipeline(
    "text-generation", 
    model=model, 
    tokenizer=tokenizer,
    do_sample=True,
    top_k=50,
    top_p=0.95, 
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=256,
    temperature=0.1,
)

sequences = pipe(
    'Hello there! How are you doing?',
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")
```

### Expected behavior

Hi, 
My Llama 2 model is not generating the stopping tokens.
For example the reply of the question Hello there! How are you doing? is:
Result: Hello there! How are you doing? I hope you are doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing well. I am doing w
The reply only stops when the max token criteria is met
What am I doing wrong?
cc @gante and @ArthurZucker <|||||>Hey! This is expected, the llama model kind of rarely generates the `eos_token`. It was the same with Llama 1, and if you run your script with the original llama, you will get the same output:
```python
# Copyright (c) Meta Platforms, Inc. and affiliates.
# This software may be used and distributed according to the terms of the GNU General Public License version 3.

import fire

from llama import Llama


def main(
    ckpt_dir: str,
    tokenizer_path: str,
    temperature: float = 0.6,
    top_p: float = 0.9,
    max_seq_len: int = 512,
    max_gen_len: int = 256,
    max_batch_size: int = 8,
):
    generator = Llama.build(
        ckpt_dir=ckpt_dir,
        tokenizer_path=tokenizer_path,
        max_seq_len=max_seq_len,
        max_batch_size=max_batch_size,
    )


    results = generator.text_completion(
        ['Hello there! How are you doing?'],
        max_gen_len=max_gen_len,
        temperature=temperature,
        top_p=top_p,
    )

    print('Hello there! How are you doing?')
    print(f"> {results['generation']}")
    print("\n==================================\n")


if __name__ == "__main__":
    fire.Fire(main)
```
and 
```bash
torchrun --nproc_per_node=1  llama/example_text_completion.py  --ckpt_dir Llama-2-7b  --tokenizer_path Llama-2-7b/tokenizer.model --temperature 0.1 --top_p 0.95
```
will produce:
```txt
> I hope you are doing well. I am doing well. I am happy to be back here. I have been away for a while. I have been busy with my studies. I have been busy with my work. I have been busy with my life. I have been busy with my family. I have been busy with my friends. I have been busy with my hobbies. I have been busy with my interests. I have been busy with my passions. I have been busy with my dreams. I have been busy with my goals. I have been busy with my ambitions. I have been busy with my aspirations. I have been busy with my plans. I have been busy with my projects. I have been busy with my ideas. I have been busy with my thoughts. I have been busy with my feelings. I have been busy with my emotions. I have been busy with my mind. I have been busy with my heart. I have been busy with my soul. I have been busy with my spirit. I have been busy with my body. I have been busy with my mind. I have been busy with my soul. I have been busy with my spirit. I have been
```

Note that the setting you are providing (`temperature = 0.1`  etc)  can influence on the generation.<|||||>So what are the best practices currently known to reduce this random ramble?<|||||>You should probably ask this question on [the forum](https://discuss.huggingface.co/), using the default parameters should already help (`temperature=0.9`), you can try to use `LogitsProcessor` for length penalty to reduce potential hallucination. You can also change the sampling stategies, use `top_k`, `contrastive search` etc etc. @gante will have better solution!<|||||>@AnishAlapattu-GEP @ArthurZucker 

This is actually a hard problem to solve, and I have yet to see a solution that generalizes well! A few things that can be tried, ranked by implementation complexity:
1. In the prompt, mention that you want a short output (be specific if you can, like "Reply in 3 sentences or less");
2. Add a custom logits processor that increases the score of the eos token according to some rule (e.g. scaling with the generated length)
3. Generate text in excess and have a post-processing step to crop unwanted text (e.g. based on the conditional probability of each sequence -- when the model starts rambling, I suspect there is a significant drop in the probability of the sentence given the past sentences)
4. Fine-tune the model 🙃 <|||||>Thanks @ArthurZucker and @gante!