LlamaForCausalLM returning prompt without answer
### System Info

transformers: 4.30.2
Python: 3.9.17
OS: MacOS 13.3.1 (a)

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I took the below code from the official documentation on the HuggingFace website: https://huggingface.co/openlm-research/open_llama_13b_easylm, and slightly adapted it to match my use case that is information extraction from unstructured text (named entity recognition) using LLMs.


```
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM

model_path = 'openlm-research/open_llama_13b'

tokenizer = LlamaTokenizer.from_pretrained(model_path)

model = LlamaForCausalLM.from_pretrained(
    model_path, 
    torch_dtype=torch.float16, 
    device_map='auto',
)

prompt = "What are the named entities in the following text: 'The Moon revolves around the Earth for over 4 billion years.'"

input_ids = tokenizer(prompt, return_tensors="pt").input_ids

generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=32
)

answer = tokenizer.decode(generation_output[0])
```


This code returns the following output:

`"<s>What are the named entities in the following text: 'The Moon revolves around the Earth for over 4 billion years.'?\nThe named entities in the following text are:\nThe Moon revolves around the Earth over 4 billion years.\nThe Moon revolves around the"`

which is not what I was hoping for. Interestingly, when I use the default example shown in the documentation, i.e.

`prompt = 'Q: What is the largest animal?\nA:'`

I get the following output:

`'<s>Q: What is the largest animal?\nA: A whale.\nQ: What is the largest animal?\nA: A whale.\nQ: What is the largest animal?\nA: A whale'`

which is slightly better, although I don't quite understand how to limit the engine not to keep repeating itself. 


### Expected behavior

The expected output would be something like: 

`{"ASTRONOMICAL_NAME": "Moon", "ASTRONOMICAL_NAME": "Earth", "PERIOD": "4 billion years"}`


Hi @leweex95, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.