LLaMA `generate` output changes depending on batch size
### System Info

```
- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.8 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False
```

### Who can help?

@younesbelkada @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

These [cells](https://colab.research.google.com/drive/1nAz2MUphzg5ifWW3CfRCJdYUgSYN5ynH?usp=sharing) should reproduce the error.

### Expected behavior

**Concise version:** I was expecting the results to not change whether or not the inference was batched or not.

Long version: Basically when I run `generate` with just one tokenized sequence, I get a certain result, and when I process the same sequence but inside a batch instead, the result changes. To make sure it wasn't any tokenization shenanigans, I tokenized the whole batch and took just the parts that related to the weird sequence (basically just `{k: v[1:] for k, v in tokenized_inputs.items()}` where the batch is just two items and the weird one is the second item).

For clarity, when I pass just

```python3
"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
Combine the question and answer into an image caption as succinctly as possible. Be sure to include the phrase "a photo of". Do not draw false conclusions.

### Input:
Is this a baseball game? no

### Response:
"""
```

to the generate function of a `LlamaForCausalLM` wrapped in a `PeftModel`, I get

```
This is not a baseball game.
```

However, if I put it in a two-item batch, the output for some reason is instead

```
A photo of people playing a game.
```

For more details, the base model was 8-bit quantized and the LoRA weights should be at half-precision. The weights were taken from `decapoda-research/llama-7b-hf` and `tloen/alpaca-lora-7b` respectively.

Edit: forgot to make the notebook public, should be fixed now.
Hey @ryan-caesar-ramos 👋 

The particular base checkpoint you're using (`decapoda-research/llama-7b-hf`) is not compatible with transformers, so we do not provide support for related problems :) 

If you have access to the original Meta weights, you can use other checkpoints as a starting point (e.g. [these](https://huggingface.co/huggyllama)). If the issue you're seeing still persists after updating the checkpoint, I'd be happy to take a look!<|||||>Hi @gante ! Thanks, I'll try to look into this, but I'm unable to use repos like `huggyllama/llama-7b` that have shard sizes around 10GB since my CPU ram can't handle it. Any way I can tell if a checkpoint is supported or not? Maybe there's a sharded one out there that is compatible<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.