Urgent: RoBERTa-Large-MNLI does not work for 2-way classification anymore
## 🐛 Bug

<!-- Important information -->

Model I am using (RoBERTa):

Language I am using the model on (English):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (RTE)

## To Reproduce

Steps to reproduce the behavior:

    pretrain_model_dir = 'roberta-large-mnli' #'roberta-large' , 'roberta-large-mnli'
    model = RobertaForSequenceClassification.from_pretrained(pretrain_model_dir, num_labels=2)

It will have error message as follows:

>     model = RobertaForSequenceClassification.from_pretrained(pretrain_model_dir, num_labels=num_labels)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_utils.py", line 411, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).

This only happened yesterday when I used the pretrained 3-way roberta-large-mnli model for a 2-way classification task; seems like the a bug in initializating or neglecting the classifier's parameters


<!-- Add any other context about the problem here. -->
Please choose a better title for your post and specify (or remove) the first part of your post. 

As far as I can tell this is an issue specific to the mnli model. As you say it's pre-trained with three final out features. When loading the state dict into the model, all weights from the pretrained model are "transferred" to the initialized model, this is a one-to-one mapping. Since RobertaForSequenceClassification has a classification head, which you can configure a.o. with `num_labels` it _can_ clash with the classification head of the pretrained model.

The intuitive solution would be to just load all weight excluding the classifier - so that torch doesn't try to load those mis-matching states, if and only if the num_labels specified in `from_pretrained` are not the same as the ones inside the models `self.config`. **However**, I'm not sure if that is the right approach, since one method (from_pretrained) then does different things with the same given pretrained model. In one case you use it completely, in the other you only use part.<|||||>Thanks for the hint. But I do not think this is the mnli model problem, because it worked before even the label size is not 3. This is my old log:

09/11/2019 22:37:46 - INFO - pytorch_transformers.modeling_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-config.json from cache at /root/.cache/torch/pytorch_transformers/54eef9bf74f919edd81b765fee413c8229620f3e271a51bdcdc67797422ef3f3.233bd69ec613d2ebcb1d55823dfc5b1e109157918e13bdbde6db7f694e1a0039
09/11/2019 22:37:46 - INFO - pytorch_transformers.modeling_utils -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "num_attention_heads": 16,
  "num_hidden_layers": 24,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 1,
  "vocab_size": 50265
}

09/11/2019 22:37:46 - INFO - pytorch_transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-mnli-pytorch_model.bin from cache at /root/.cache/torch/pytorch_transformers/1c2e185bc053ae7261ce2289653438a4c05b871ff7f30eaee1cdb787154410e0.c1823b934e18e923174ff260ba955eef25b2205f48fe2655c432a5fb805f8c8a
09/11/2019 22:38:02 - INFO - pytorch_transformers.modeling_utils -   Weights of RobertaForSequenceClassification not initialized from pretrained model: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
09/11/2019 22:38:02 - INFO - pytorch_transformers.modeling_utils -   Weights from pretrained model not used in RobertaForSequenceClassification: ['lm_head.weight', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias']

You can see that the system can automatically detect which part of the mnli parameters not used to initialize, then it will neglect it; but now the transformers only output error message, I found it from yesterday.

My code is the same, but behave differently now<|||||>It's odd since `modeling_utils` hasn't seen any updates apart from the naming update. I'm not sure where else to look for this issue.<|||||>My "pytorch_transformers" was installed maybe 3 weeks ago, but the latest "transformers" was installed yesterday. But both will make the same error, in different lines:

"transformers" in line 411:
File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_utils.py", line 411, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).

"pytorch_transformers" in line 594:
File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 594, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).


But both come from the same reason.  Since the "pytorch_transformers" worked for me a couple days ago (even though i found "transformer" did not work for me yesterday, but very likely something changed before yesterday, since I haven't run the piece of code for some days).

Now, I kind of agree with you that the "roberta-large-mnli" model itself had something changed recently, which makes it unable to neglect the mismatch of hyperparameters.
<|||||>Also having this issue using the `roberta-large-mnli` model on a single-document (not paired) multiclass classification task.<|||||>I guess the simplest solution would be to load the model with previous num_labels and than directly change its num_labels and initialize a new classifier layer in the `run_glue.py` script. This way you won't need to modify any of the `transformers` code.

This is what I do:
```
    # for num_labels(mnli)
    num_labels_old = config_class.from_pretrained(args.model_name_or_path).num_labels
    config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
                                          num_labels=num_labels_old,
                                          finetuning_task=args.task_name,
                                          cache_dir=args.cache_dir if args.cache_dir else None)
    tokenizer = tokenizer_class.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
                                                do_lower_case=args.do_lower_case,
                                                cache_dir=args.cache_dir if args.cache_dir else None)
    if num_labels != num_labels_old:
        config.num_labels = num_labels_old
        model = model_class.from_pretrained(args.model_name_or_path,
                                            from_tf=bool('.ckpt' in args.model_name_or_path),
                                            config=config,
                                            cache_dir=args.cache_dir if args.cache_dir else None)
        config.num_labels = num_labels
        logger.info('Reintializing model classifier layer...')
        model.num_labels = num_labels
        model.classifier = RobertaClassificationHead(config)

    else:
        model = model_class.from_pretrained(args.model_name_or_path,
                                            from_tf=bool('.ckpt' in args.model_name_or_path),
                                            config=config,
                                            cache_dir=args.cache_dir if args.cache_dir else None)

```

Of course, it would be better to modify the `transformers` code directly. 
<|||||>Hi felicity,

Sorry for the late reply. I actually have forgotten how i solve that, or
gave up. I will check my code when I finish a deadline in the next  couple
of days. Thanks for sharing your experience.

Best.

On Thu, Nov 28, 2019 at 8:01 AM felicitywang <notifications@github.com>
wrote:

> I'm getting the same error. Did you solve this problem? @wyin-Salesforce
> <https://github.com/wyin-Salesforce> @pmbaumgartner
> <https://github.com/pmbaumgartner> Would really appreciate it if you
> could share your solutions. Thank you.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/1348?email_source=notifications&email_token=AM2XN4JEMO7KYWJ4OFXFSYDQV7TPFA5CNFSM4I3D6BP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM7XIA#issuecomment-559545248>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM2XN4IUZKSK4D22E2WI3ZDQV7TPFANCNFSM4I3D6BPQ>
> .
>


-- 


Wenpeng Yin
Research Scientist @ Salesforce Research, Palo Alto
https://sites.google.com/site/yinwenpeng1987/
<|||||>Thanks @wyin-Salesforce  . If people are still having trouble with this, the solution above worked. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I want to evaluate the pre-trained **roberta-large-mnli** model on a **2-way classification task**. I tried to imitate what @felicitywang posted by adding these four lines after calling config/tokenizer/model in run_glue.py (after line 134):
```
    num_labels = 2 # ADDED
    config.num_labels = num_labels # ADDED
    model.num_labels = num_labels # ADDED
    model.classifier = RobertaClassificationHead(config) # ADDED
```

However, I'm still getting the following error (from line 131) when I run my modified run_glue.py:
```
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
```

where the line 131 is the last line ('cache_dir=...') of this code block:
```
    model = AutoModelForSequenceClassification.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config,
        cache_dir=model_args.cache_dir,
    )
```

Does anyone know how to make it work?<|||||>> I want to evaluate the pre-trained **roberta-large-mnli** model on a **2-way classification task**. I tried to imitate what @felicitywang posted by adding these four lines after calling config/tokenizer/model in run_glue.py (after line 134):
> 
> ```
>     num_labels = 2 # ADDED
>     config.num_labels = num_labels # ADDED
>     model.num_labels = num_labels # ADDED
>     model.classifier = RobertaClassificationHead(config) # ADDED
> ```
> 
> However, I'm still getting the following error (from line 131) when I run my modified run_glue.py:
> 
> ```
> RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
> 	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
> 	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
> ```
> 
> where the line 131 is the last line ('cache_dir=...') of this code block:
> 
> ```
>     model = AutoModelForSequenceClassification.from_pretrained(
>         model_args.model_name_or_path,
>         from_tf=bool(".ckpt" in model_args.model_name_or_path),
>         config=config,
>         cache_dir=model_args.cache_dir,
>     )
> ```
> 
> Does anyone know how to make it work?

@scarletcho 
If you just want to evaluate the pretrained roberta-large-mnli on a new dataset without any fine-tuning; let's say your new dataset has two classes "entail" and "non_entail", then you just manually combine the outputs "neutral" and "contradict" as a single output "non_entail".

If you want to load this pretrained model and fine-tune on your 2-way dataset, today I just found the following approach works for using N-way fine-tuning:

`    model_config = BartConfig.from_pretrained(pretrain_model_dir)
    model_config.num_labels=new_num_labels
    model = BartForSequenceClassification.from_pretrained(pretrain_model_dir, config=model_config)`
    
I tried Bart, but it should work for roberta too (here "pretrain_model_dir" is string "facebook/bart-large", you can use "roberta-large-mnli" instead)<|||||>> I guess the simplest solution would be to load the model with previous num_labels and than directly change its num_labels and initialize a new classifier layer in the `run_glue.py` script. This way you won't need to modify any of the `transformers` code.
> 
> This is what I do:
> 
> ```
>     # for num_labels(mnli)
>     num_labels_old = config_class.from_pretrained(args.model_name_or_path).num_labels
>     config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
>                                           num_labels=num_labels_old,
>                                           finetuning_task=args.task_name,
>                                           cache_dir=args.cache_dir if args.cache_dir else None)
>     tokenizer = tokenizer_class.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
>                                                 do_lower_case=args.do_lower_case,
>                                                 cache_dir=args.cache_dir if args.cache_dir else None)
>     if num_labels != num_labels_old:
>         config.num_labels = num_labels_old
>         model = model_class.from_pretrained(args.model_name_or_path,
>                                             from_tf=bool('.ckpt' in args.model_name_or_path),
>                                             config=config,
>                                             cache_dir=args.cache_dir if args.cache_dir else None)
>         config.num_labels = num_labels
>         logger.info('Reintializing model classifier layer...')
>         model.num_labels = num_labels
>         model.classifier = RobertaClassificationHead(config)
> 
>     else:
>         model = model_class.from_pretrained(args.model_name_or_path,
>                                             from_tf=bool('.ckpt' in args.model_name_or_path),
>                                             config=config,
>                                             cache_dir=args.cache_dir if args.cache_dir else None)
> ```
> 
> Of course, it would be better to modify the `transformers` code directly.

Hi,

I am using this code to solve this issue.  What is `RobertaClassificationHead(config)` ? I cannot find this from huggingface.<|||||>> > I want to evaluate the pre-trained **roberta-large-mnli** model on a **2-way classification task**. I tried to imitate what @felicitywang posted by adding these four lines after calling config/tokenizer/model in run_glue.py (after line 134):
> > ```
> >     num_labels = 2 # ADDED
> >     config.num_labels = num_labels # ADDED
> >     model.num_labels = num_labels # ADDED
> >     model.classifier = RobertaClassificationHead(config) # ADDED
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > However, I'm still getting the following error (from line 131) when I run my modified run_glue.py:
> > ```
> > RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
> > 	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([3, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).
> > 	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([2]).
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > where the line 131 is the last line ('cache_dir=...') of this code block:
> > ```
> >     model = AutoModelForSequenceClassification.from_pretrained(
> >         model_args.model_name_or_path,
> >         from_tf=bool(".ckpt" in model_args.model_name_or_path),
> >         config=config,
> >         cache_dir=model_args.cache_dir,
> >     )
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Does anyone know how to make it work?
> 
> @scarletcho
> If you just want to evaluate the pretrained roberta-large-mnli on a new dataset without any fine-tuning; let's say your new dataset has two classes "entail" and "non_entail", then you just manually combine the outputs "neutral" and "contradict" as a single output "non_entail".
> 
> If you want to load this pretrained model and fine-tune on your 2-way dataset, today I just found the following approach works for using N-way fine-tuning:
> 
> ` model_config = BartConfig.from_pretrained(pretrain_model_dir) model_config.num_labels=new_num_labels model = BartForSequenceClassification.from_pretrained(pretrain_model_dir, config=model_config)`
> 
> I tried Bart, but it should work for roberta too (here "pretrain_model_dir" is string "facebook/bart-large", you can use "roberta-large-mnli" instead)

It is ok to use roberta-large, but it stills has the error in roberta-large-mnli.<|||||>I fix the issue when I use `transformers=2.3.0` and I put num_labels in the config and then put the config into the model.<|||||>I'm using `transformers=4.20.1` and [this example code](https://github.com/huggingface/transformers/blob/24a85cca61fda92b9376fe45da1dcb10c8853066/examples/pytorch/text-classification/run_glue.py) (the most recent commit that passed all the automated testing) and I'm still running into this error.

In the code, it looks like they do [add the num_labels](https://github.com/huggingface/transformers/issues/1348#issuecomment-888779209) to the config, and then put the config into the model, but I'm still getting the error.

The exact command I'm running is `python run_glue.py --train_file sg_train_dataset.csv --validation_file sg_test_dataset.csv --do_train --do_eval --model_name roberta-large-mnli --output_dir output --overwrite_output_dir` where `sg_test_dataset.csv` is a CSV file with three columns, "sentence1", "sentence2" and "label" and "label" is either 0 or 1.

Any suggestions on how to fix it?