Configuration initialized from checkpoint does not keep the checkpoint identifier in its attributes
Since version v3.4.0, initializing a model using the `from_pretrained` method adds a `name_or_path` attribute to the configuration, referencing the checkpoint used for initialization:

```py
from transformers import BertModel

model = BertModel.from_pretrained(model_name)
print(model.config.name_or_path)
# model_name
```

However, initializing the configuration on its own with the `from_pretrained` method does not yield the same attribute:

```py
from transformers import BertConfig

config = BertConfig.from_pretrained(model_name)
# config has no `name_or_path` attribute
```

This means that the configuration object initialized is not the same in both cases, whereas it probably should be.
Good point.

Maybe what we could do is:
- initializing the configuration with `from_pretrained` initializes the `name_or_path` attribute of the config as you mention, and
- using the configuration in a *model* `from_pretrained` method override the `name_or_path` attribute with the one of the model so that it's in priority linked to the weights path.

Another option would be to have two attributes in the configuration:
- `configuration_name_or_path`
- `weights_name_or_path`

respectively populated by the config `from_pretrained` and the model `from_pretrained`. Maybe with a property linking to one in priority.

but I'm wondering if it's worth so many attributes... 🤔<|||||>I would go for the first version you proposed: having `name_or_path` for the configuration initialized if used alongside `from_pretrained`, which gets overridden by the model `from_pretrained`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
