Custom kwargs not passed when extending PretrainedConfig class
### System Info

- `transformers` version: 4.26.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.9.12
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.0.dev20220925 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger @Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I want to load existing model's configs with `.from_pretrained` but also want to pass my own kwargs `moo` & `boo`.
I'm extending `PretrainedConfig` like below:

```python
from transformers import PretrainedConfig

class MyClassificationConfig(PretrainedConfig):
    def __init__(self, 
                moo='poo',
                boo=5,
                **kwargs):
        print(boo) # prints 5 because boo doesn't get passed
        print(kwargs)
        # do custom calculations and set some custom config values here
        super().__init__(**kwargs)

MyClassificationConfig.from_pretrained('google/canine-s',# example model, any other is same
id2label={1:"g",2:"b"}, 
moo="poo", 
boo="hoo")

```
only predefined `id2label,label2id,num_classes` values get updated in the config. Happens [here](https://github.com/huggingface/transformers/blob/78a93d17c0e0bca0bc4477e0ee362a95d79f9b22/src/transformers/configuration_utils.py#L702).

the custom `moo` and `poo` param doesn't get passed to `MyClassificationConfig`. Because kwargs don't get passed [here](https://github.com/huggingface/transformers/blob/78a93d17c0e0bca0bc4477e0ee362a95d79f9b22/src/transformers/configuration_utils.py#L696)
Results in `moo` & `boo` argument values not changing from the default. 
Since kwargs are optional can result in silent errors where you are actually using default values while thinking you are passing values!

I think this is a bug. But if it is intentional maybe nice to warn the user so there are no silent errors

### Expected behavior

Should be able to extend PretrainedConfig class allowing custom kwargs.
This is intended. Hadding the kwargs is done [here](https://github.com/huggingface/transformers/blob/78a93d17c0e0bca0bc4477e0ee362a95d79f9b22/src/transformers/configuration_utils.py#L712) but to filter out the value that have nothing to do in the config, we detect whether they are attribute of the config or not. So you should change your code like this:

```py
class MyClassificationConfig(PretrainedConfig):
    def __init__(self, 
                moo='poo',
                boo=5,
                **kwargs):
        print(boo) # prints 5 because boo doesn't get passed
        print(kwargs)
        # do custom calculations and set some custom config values here
        self.moo = moo
        self.poo = poo
        super().__init__(**kwargs)<|||||>this is saving the custom `moo`, `boo` to the returned config, 

`config = MyClassificationConfig.from_pretrained("hf-internal-testing/config-no-model",boo="hoo")`

but I still can't access the kwarg value I set inside the `__init__` to do some calculations. 
~~Is there a post config init function I can override?~~ I guess I can do it at call.<|||||>Also I'm a bit confused about the following behaviour (example):

Even though the DinatConfig class doesn't take as a param `image_size`
and thus doesn't do `self.image_size=image_size` anywhere if I pass 
`config=DinatConfig(...,image_size=128,...)` this (imagined) image_size param becomes part of the config(because super() assigns them)

`DinatConfig(image_size=5).image_size`

which can be good if you want to save some stuff along in your config 
but isn't it also prone to confusion say that I confuse config key names?
Also if I can just pass what ever kwarg and it becomes part of config then do I really need to extend the PretrainedConfig class if I'm just assigning passed kwargs to self 🤔 I guess just to assign default values.<|||||>Sorry you actually don't need to set them as attributes, you just need to pass them along to the super method:
```
class MyClassificationConfig(PretrainedConfig):
    def __init__(self, 
                moo='poo',
                boo=5,
                **kwargs):
        print(boo) # prints 5 because boo doesn't get passed
        print(kwargs)
        # do custom calculations and set some custom config values here
        super().__init__(moo=moo, boo=boo, **kwargs)
```
will have the right value set in the config.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.