Initialized DETR backbone weights do not match with actual pretrained weights 
Hello everybody! 

My task is to initialize DETR Object Detection model with my own pretrained backbone (for example, ResNet-50). So, in Detr class (I took the code from [this Hugging Face tutorial](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) as a basis), I create model from DetrConfig:
```
class Detr(pl.LightningModule):

    def __init__(self, lr, lr_backbone, weight_decay, backbone_name):
        super().__init__()

        self.model = DetrForObjectDetection(DetrConfig(num_labels=len(id2label)
                                                      ,ignore_mismatched_sizes=True
                                                      ,backbone=backbone_name
                                                       ))
    ...
```
Then I create model:
```
model = Detr(lr=1e-4, lr_backbone=1e-5, weight_decay=1e-3, backbone_name='resnet50')
```
And I want to see, for example, first layer weights in backbone:
```
model.model.model.backbone.conv_encoder.model.conv1.weight
```
Output:
```
Parameter containing:
tensor([[[[-1.0862e-02,  5.6558e-02, -2.5807e-02,  ..., -1.8801e-03,
            2.1483e-02, -1.1144e-02],
...
```
And I notice that this weights are different from timm backbone model, which is [fed to the input](https://github.com/huggingface/transformers/blob/68dec6bffd68e62d151758d8ac7378bf9b40ac20/src/transformers/models/detr/modeling_detr.py#:~:text=backbone%20%3D%20create_model(name%2C%20pretrained%3DTrue%2C%20features_only%3DTrue%2C%20out_indices%3D(1%2C%202%2C%203%2C%204)%2C%20**kwargs)) of the Detr model:
```
backbone_model = timm.create_model('resnet50', pretrained=True, features_only=True, out_indices=(1, 2, 3, 4))
backbone_model.conv1.weight
```
Output:
```
Parameter containing:
tensor([[[[-2.6017e-02, -1.7312e-02, -1.0648e-02,  ...,  1.1380e-02,
           -1.0471e-02, -1.1242e-02],
...
```

So... My question is, how can this happen? Why are the weights of the model that is fed into the Detr input differ from the weights of the same model in the already initialized Detr? Is this a bug or am I missing something?

One more detail: If I create model again, weights will also change! But when I restart Kernel and create Detr model, weights will be the same as in my first model creation at previous Kernel! So, the weights are initialized differently, but with a certain cycle? Why is this happening? In my understanding, backbone weights should always be the same and equal to the original ResNet weights.

Maybe, I'm setting the configuration wrong, but it could be a bug. I would really appreciate an answer and possible clarification!
Hello again! Tagged @NielsRogge, maybe you can help? I tried by hand to track the moment when the weights of the backbone change and until [that moment](https://github.com/huggingface/transformers/blob/main/src/transformers/models/detr/modeling_detr.py#:~:text=apply%20final%20processing-,self.post_init(),-%23%20taken%20from%20https) the weights remain the same, then I could not track the changes. So, I suppose model is initializing correctly, but why (and where) weights are changing?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Thanks for spotting that, @univanxx. I am not sure myself why that is the case. Marking this issue as a "good second issue" for now. <|||||>Hello! I reviewed the issue. It seems that the classes `DetrModel` and `DetrForObjectDetection` apply weight initialization using method `self.post_init()` which calls `DetrPreTrainedModel._init_weights(...)` and changes backbone model weights.

`DetrForObjectDetection` and `timm` models have different backbone weights:
```python
from transformers import DetrConfig, DetrForObjectDetection
import timm

config = DetrConfig(ignore_mismatched_sizes=True, backbone='resnet50')
model = DetrForObjectDetection(config)

timm_backbone = timm.create_model('resnet50', pretrained=True, features_only=True, out_indices=(1, 2, 3, 4))

print((model.model.backbone.conv_encoder.model.conv1.weight == timm_backbone.conv1.weight).all())
```
Output: `tensor(False)`

But the weights are the same as in `timm` model when using only `DetrTimmConvEncoder`:
```python
from transformers.models.detr.modeling_detr import DetrTimmConvEncoder

backbone = DetrTimmConvEncoder(config.backbone, config.dilation)

print((backbone.model.conv1.weight == timm_backbone.conv1.weight).all())
```
Output: `tensor(True)`

This seems like an unwanted behaviour, but I am not sure what would be the best way to fix this.<|||||>Hello everyone! Unfortunately, I still could not find out the cause of the problem, but I developed a temporary solution (however, I'm not sure if it's correct to do this) using @chamidullinr code: 

1. First, we initialize the model as usual, for example:
```python
from transformers import DetrConfig, DetrForObjectDetection

config = DetrConfig(ignore_mismatched_sizes=True, backbone='resnet50')
model = DetrForObjectDetection(config)
```
2. After that, we create timm-model for the future backbone:
```python
import timm

timm_backbone = timm.create_model('resnet50', pretrained=True, features_only=True, out_indices=(1, 2, 3, 4))
```
3. And finally, we manually copy the weights from backbone model to DETR using in-place copying operator:
```python
# weighs
for name_backbone, parameter_backbone in timm_backbone.named_parameters():
    for name, parameter in model.model.backbone.conv_encoder.model.named_parameters(): 
        if name_backbone == name:
            parameter.data.copy_(parameter_backbone.data)
# buffers
for name_backbone, parameter_backbone in timm_backbone.named_buffers():
    for name, parameter in model.model.backbone.conv_encoder.model.named_buffers(): 
        if name_backbone == name:
            parameter.data.copy_(parameter_backbone.data)
```
Now, if we try:
```python
print((model.model.backbone.conv_encoder.model.conv1.weight == timm_backbone.conv1.weight).all())
```
Output: ```tensor(True)```<|||||>Hi @univanxx I was looking into this issue, as @chamidullinr mentioned, this is due to the `post_init()` functions being called [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/detr/modeling_detr.py#L1350), and reinitializing the weights, which is actually expected since we are not calling `.from_pretrained()` and are basically asking the model to just create a model our of the given skeleton, without any guarantees on the weight being pretrained. 

Looking deeper I also found that the `post_init()` function is disabled when we call `.from_pretrained()`, which causes the weights to not be re-initialized when we are loading pretrained weights from a checkpoint, but it works on the entire model level, and not at the individual level it seems.

I might be wrong, but if you want a part of the model to be initialized according to your given weights, your way of manually copying them does seem correct. Although there could be some way to add functionality to make for more fine-grained tuning of what part remains pretrained and what doesn't<|||||>Adding an informative comment here: https://github.com/huggingface/transformers/pull/17572