Errror even if offload_dir is provided
### System Info

Windows 10
16 GB RAM
4GB NVIDIA

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

You need CUDA enabled system.
You also need a version of transofrmers that has no other bugs and one that has the LlamaTokenizer available.
I am using transformers 4.28.0-dev, but not the latest as it has bugs.
Executing the following code: 
```
import torch
from peft import PeftModel
import transformers
import os, time
import tempfile

assert ("LlamaTokenizer" in transformers._import_structure["models.llama"]), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")

BASE_MODEL = "decapoda-research/llama-7b-hf"
LORA_WEIGHTS = "tloen/alpaca-lora-7b"

force_cpu = False

if torch.cuda.is_available() and not force_cpu:
    device = "cuda"
    print("Video memory available:", torch.cuda.get_device_properties(0).total_memory / 1024 / 1024, "MBs")
else:
    device = "cpu"
print("Compute device is:", device)

try:
    if torch.backends.mps.is_available():
        device = "mps"
except:
    pass

print("Loading model with selected weights ...")

if device == "cuda":
    
    print("model on cuda")
    
    model = LlamaForCausalLM.from_pretrained(
        BASE_MODEL,
        load_in_8bit=False,
        torch_dtype=torch.float16,
        device_map="auto",
        offload_folder="offload", #required on GPU with not enough memory
    )
    
    model = PeftModel.from_pretrained(
        model, 
        LORA_WEIGHTS, 
        torch_dtype=torch.float16,
        device_map="auto",
        offload_folder="offload", #required on GPU with not enough memory
        
    )
```

Gives the error:

`ValueError: We need an `offload_dir` to dispatch this model according to this `device_map`, the following `

The offload folder is specified in my code and it works the first time, but not the second. I am forced to hard code the folder like this: `offload_dir = "offload"` to big_modeling.py at line 341 in order to avoid the error.

### Expected behavior

No error.
Actually this is in accelerate. I need to move it there.