transformers.set_seed seems to do nothing
### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante, @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. install transformes with support for alpaca model
2. run this code with one seed
3. run this code with any other seed
4. see that the results are the same
```
from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM, set_seed
from torch import float16, compile, no_grad


set_seed(621)

# Enhances prompt
def enhance_prompt(prompt, input=None):
    if input:
        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Input:
{input}

### Response:"""
    else:
        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:"""

# Gets response from Alpaca
def get_response(prompt):
    with no_grad():
        outputs = alpaca.generate(input_ids=tokenizer(prompt, return_tensors="pt").input_ids.to("cuda"), generation_config=generation_config, return_dict_in_generate=True, output_scores=True)
    outputs = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
    return outputs.split("### Response:")[1]


# Sets up Alpaca
tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")
alpaca = LlamaForCausalLM.from_pretrained("chainyo/alpaca-lora-7b", load_in_8bit=True, torch_dtype=float16, device_map="auto")
generation_config = GenerationConfig(temperature=0.2, top_p=0.75, top_k=40, num_beams=4, max_new_tokens=64)
alpaca.eval()
compile(alpaca)

# Gets output from Alpaca
prompt = enhance_prompt("Write a simple poem about flowers.")
out = get_response(prompt)

# Prints Alpaca's output
print(out)
```

### Expected behavior

Model will output two different answers but now it gives the same every seed I try.
@mojejmenojehonza 👋

Two notes:
1. You should pass `do_sample=True` in your generation config or in your `.generate()` call. Most models have it off by default, causing the generation to be deterministic (and ignoring parameters like `temperature`, `top_k`, etc).
2. With `temperature=0.2`, the relative weight of the most likely logits is massively increased, making generation almost deterministic. Even if there are no bugs in your script, it's far from guaranteed that two different seeds produce different outputs with such low temperature :)<|||||>@gante
Thanks worked like a charm :)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.