Pipeline cannot be initialized with the "state_dict" parameter
### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.10 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante, @Narsil, @sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
state_dict = torch.load(path_to_bin, map_location=torch.device('cpu'))
base_model = LlamaForCausalLM.from_pretrained(
    None, config=config, state_dict=state_dict,
    torch_dtype=torch.float16,
    device_map="auto")
```

### Expected behavior

When I try to initialize the pipelining model with `state_dict`, I get an error:
`AttributeError: 'NoneType' object has no attribute 'endswith'`
from `transformers/modeling_utils.py:448` in `load_state_dict`
I thought that if you specified `pretrained_model_name_or_path=None`, the path to the model would be ignored, and all necessary parameters and weights themselves would be taken from `config` and `state_dict`. Isn't that how it's done? How do I do initialization using `state_dict`?
You cannot use the `state_dict` argument with `device_map="auto"`, we only support the base case
```py
base_model = LlamaForCausalLM.from_pretrained(None, config=config, state_dict=state_dict)
```<|||||>Thanks @sgugger. Your advice really helped. I don't have enough memory to load the model even on fp16, so I try to initialize the model with `load_in_4bit=True` and get the same error:

`AttributeError: 'NoneType' object has no attribute 'endswith'`
from `transformers/modeling_utils.py:448` in `load_state_dict`

There is a way to fix this, isn't there?

P.S. And another question: I have not found an option to load tokenizer weights from RAM. In, for example, LlamaTokenizer can I feed the config and state_dict?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.