EncoderDecoderModel loss decreased unbelievably and the generated text were repetitive 4.12.0
Hi,

I am training an EncoderDecoderModel with BertGenerationEncoder and BertGenerationDecoder from scratch (no pretrain) on a custom dataset with about 10k sequences. I found that the loss decreased unbelievably fast to a level which I thought was impossible for seq2seq task. And the generated texts were just repetition of the same token. When I downgrade from v4.12.0 to v4.11.3, this problem disappear. 

I understand that in v4.12.0 I don't need to pass decoder_input_ids to EncoderDecoderModel as it will be generated automatically, but I wonder it introduced other changes/bugs.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.0
- Platform: Linux-4.18.0-305.12.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.5
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten
## Information

Model I am using (Bert, XLNet ...):
EncoderDecoder, BertGeneration
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
My model was written like this:

        import transformers as T
        class Translation(nn.Module):
            def __init__(self):
                super().__init__()
                self.encoder_config = T.BertConfig(vocab_size=30,
                                                   hidden_size=128,
                                                   num_hidden_layers=4,
                                                   num_attention_heads=4,
                                                   intermediate_size=512,
                                                   max_position_embeddings=512,
                                                   output_hidden_states=True)
                self.decoder_config = T.BertConfig(vocab_size=30,
                                                   hidden_size=128,
                                                   num_hidden_layers=4,
                                                   num_attention_heads=4,
                                                   intermediate_size=512,
                                                   max_position_embeddings=512,
                                                   output_hidden_states=True,
                                                   is_decoder=True,
                                                   add_cross_attention=True)
                self.encoder = T.models.bert_generation.BertGenerationEncoder(self.encoder_config)
                self.decoder = T.models.bert_generation.BertGenerationDecoder(self.decoder_config)
                self.transformer = T.models.encoder_decoder.EncoderDecoderModel(encoder=self.encoder, decoder=self.decoder)
        
            def forward(self, input_ids, encoder_attention_mask=None, labels=None):
                output = self.transformer(input_ids=input_ids,
                                          attention_mask=encoder_attention_mask,
                                          # decoder_input_ids=decoder_input_ids, # add this line in v4.11.3
                                          labels=labels)
                return output.loss


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
I am working on a seq2seq task which "summarize" long input sequence to short one
## To reproduce

Steps to reproduce the behavior:
I trained the model in v4.11.3, v4.12.0 and v 4.15.0, the problem seemed to only appear since v4.12.0
I used the following tokenization:

       {'CLS': 1, 'SEP': 2, 'PAD': 0, 'A': 3, 'B': 4, 'C': 5...}

1. in v4.11.3, I set `labels` to [1,3,4,5,...,2], `decoder_input_ids` to [1,3,4,5,...,2], when calculating loss in BertGenerationDecoder, the model finds the crossEntropy between `labels[:, 1:]` and  `prediction_scores[:, :-1, :]` as stated in the source code of BertGenerationDecoder
2. since v4.12.0, I changed `labels` to [3,4,5,...,2] and the automatically generated `decoder_input_ids` are [1,3,4,5,...], when calculating loss in EncoderDecoderModel, the model finds the crossEntropy between `labels` and  `decoder_outputs.logits` as stated in the source code of EncoderDecoderModel. So I think 1 and 2 are calculating the same thing.
3. I trained the model using pytorch-lightning trainer, but I don't think it caused the problem
4. I've also tried to provide `decoder_input_ids` to the model in v4.12.0, which should disable automatic generation of `decoder_input_ids` from `labels`, but the problem still exist.
5. I've also tried to calculate the loss by myself in both v4.11.3 and v4.12.0, and there was no problem in v4.11.3, but problem in v4.12.0

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
When I trained the model in v4.11.3, everything was fine, but in v4.12.0, the loss curve became strange:
![image](https://user-images.githubusercontent.com/68891831/147748264-707ad31e-fdb6-4648-8958-c44a4b11f879.png)
![image](https://user-images.githubusercontent.com/68891831/147748521-587a20ba-3447-487c-9c92-16f89aa51eff.png)

blue curve is v4.11.3, red curve is v4.12.0. Both curves become flat after several epochs, so I don't think the blue curve means the model is underfitting.

Generated text also changed from ['CLS', 'B', 'C', 'A', ..., 'Sep'] to ['CLS', 'A', 'A', 'A', ..., 'A'] in v4.12.0

I can use v4.11.3 because it works well but I am just wondering why the update of EncoderDecoderModel in v4.12.0 changed the behavior.

Thanks a lot!
<!-- A clear and concise description of what you would expect to happen. -->

Hey @zzhongzz,

Thanks for the issue! Sorry this will be quite difficult to debug from my side as I would have to guess what the difference might be. Could you try to find an **easy to reproduce** difference between 4.11.3 and 4.12.0? E.g. something like

```python
dummy_inputs = # some input_ids
lables = # some labels

encoder_decoder_model = EncoderDecoderModel.from_pretrained("path/to/your/model")
loss = encoder_decoder(input_ids=dummy_input_ids, labels=labels).loss
```

where the loss is different between 4.11.3 and 4.12.0? 

Thank you!<|||||>I tried to train a model in v4.11.3, then load it and calculate the loss:

    encoder = transformers.models.bert_generation.BertGenerationEncoder.from_pretrained('encoderModel.pt')
    decoder = transformers.models.bert_generation.BertGenerationDecoder.from_pretrained('decoderModel.pt')
    encoder_decoder_model = transformers.models.encoder_decoder.EncoderDecoderModel(encoder=encoder, decoder=decoder)
    
    dummy_input_ids = torch.tensor([[2, 21, 8, 11, 8, 10, 10, 17, 8, 10, 10, 17, 7, 13, 16, 3]])
    decoder_input_ids = torch.tensor([[2, 21, 12, 13, 10, 5, 11, 6, 19, 18, 6, 5, 3]])
    labels = torch.tensor([[2, 21, 12, 13, 10, 5, 11, 6, 19, 18, 6, 5, 3]])
    
    loss = encoder_decoder_model(input_ids=dummy_input_ids, decoder_input_ids=decoder_input_ids, labels=labels).loss
    print(loss)

The output in v4.11.3 is like this:

    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    tensor(3.0904, grad_fn=<NllLossBackward0>)

The output in v4.15.0 is like this:

    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    /home/zzhong/miniconda3/lib/python3.9/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py:524: FutureWarning: Version v4.12.0 introduces a better way to train encoder-decoder models by computing the loss inside the encoder-decoder framework rather than in the decoder itself. You may observe training discrepancies if fine-tuning a model trained with versions anterior to 4.12.0. The decoder_input_ids are now created based on the labels, no need to pass them yourself anymore.
      warnings.warn(DEPRECATION_WARNING, FutureWarning)
    tensor(3.2755, grad_fn=<NllLossBackward0>)

The loss is different, then I remove `decoder_input_ids`, and remove `CLS` from `label`:

    dummy_input_ids = torch.tensor([[2, 21, 8, 11, 8, 10, 10, 17, 8, 10, 10, 17, 7, 13, 16, 3]])
    labels = torch.tensor([[21, 12, 13, 10, 5, 11, 6, 19, 18, 6, 5, 3]])
    encoder_decoder_model.config.decoder_start_token_id = 2
    encoder_decoder_model.config.pad_token_id = 0
    loss = encoder_decoder_model(input_ids=dummy_input_ids, labels=labels).loss
    print(loss)


    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    /home/zzhong/miniconda3/lib/python3.9/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py:524: FutureWarning: Version v4.12.0 introduces a better way to train encoder-decoder models by computing the loss inside the encoder-decoder framework rather than in the decoder itself. You may observe training discrepancies if fine-tuning a model trained with versions anterior to 4.12.0. The decoder_input_ids are now created based on the labels, no need to pass them yourself anymore.
      warnings.warn(DEPRECATION_WARNING, FutureWarning)
    tensor(2.9198, grad_fn=<NllLossBackward0>)

The loss is still different, then I tried to calculate the loss as it in BertGenerationDecoder:

    dummy_input_ids = torch.tensor([[2, 21, 8, 11, 8, 10, 10, 17, 8, 10, 10, 17, 7, 13, 16, 3]])
    decoder_input_ids = torch.tensor([[2, 21, 12, 13, 10, 5, 11, 6, 19, 18, 6, 5, 3]])
    labels = torch.tensor([[2, 21, 12, 13, 10, 5, 11, 6, 19, 18, 6, 5, 3]])
    
    logits = encoder_decoder_model(input_ids=dummy_input_ids, decoder_input_ids=decoder_input_ids, labels=labels).logits
    # we are doing next-token prediction; shift prediction scores and input ids by one
    shifted_prediction_scores = logits[:, :-1, :].contiguous()
    labels = labels[:, 1:].contiguous()
    loss_fct = nn.CrossEntropyLoss()
    loss = loss_fct(shifted_prediction_scores.view(-1, decoder.config.vocab_size), labels.view(-1))
    print(loss)

    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    /home/zzhong/miniconda3/lib/python3.9/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py:524: FutureWarning: Version v4.12.0 introduces a better way to train encoder-decoder models by computing the loss inside the encoder-decoder framework rather than in the decoder itself. You may observe training discrepancies if fine-tuning a model trained with versions anterior to 4.12.0. The decoder_input_ids are now created based on the labels, no need to pass them yourself anymore.
      warnings.warn(DEPRECATION_WARNING, FutureWarning)
    tensor(2.8944, grad_fn=<NllLossBackward0>)

Still different, then I switch back to v 4.11.3:

    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    You are using a model of type bert to instantiate a model of type bert-generation. This is not supported for all configurations of models and can yield errors.
    tensor(3.0904, grad_fn=<NllLossBackward0>)

It's now the same as my first try. So I think maybe there are something different in the model itself between 4.11.3 and 4.15.0 (or 4.12.0, they behave the same).<|||||>Hi,

Thanks for reporting. We indeed changed the behaviour of `EncoderDecoderModel` in v4.12. I've investigated this by training an `EncoderDecoderModel` in native PyTorch with the weights of 'bert-base-uncased' used to initialize the weights of the encoder as well as those of the decoder.

As always, I pick a tiny dataset and see if the model is able to properly overfit it (as this is a great way to [debug neural networks](http://karpathy.github.io/2019/04/25/recipe/)). For me, it seems to work fine, loss is going down nicely and the ROUGE metric is going up, after 30 epochs I get the following:

```
Train ROUGE precision: 0.4337499999999999
Train ROUGE recall: 0.9916624999999999
Train ROUGE F1: 0.5708249999999999
```
Here's the notebook: https://colab.research.google.com/drive/1YSfpbZbrjxpF11PU-b_dtO_OceRRlVBb?usp=sharing

However, I'm not able to reproduce this with `BertGeneration`, cc @patrickvonplaten. I guess that if you're going for 'bert-base-uncased', it makes more sense to use `BertModel` rather than `BertGeneration`. <|||||>Hi @NielsRogge,

Inspired by your last sentence (though I didn't use any pretrained model in my case), 
> However, I'm not able to reproduce this with `BertGeneration`, cc @patrickvonplaten. I guess that if you're going for 'bert-base-uncased', it makes more sense to use `BertModel` rather than `BertGeneration`.

I did an experiment using `BertModel` as the encoder and `BertLMHeadModel` as the decoder in v4.15.0, and left all other settings unchanged, the problem seemed to disappear. So would it be a bug in `BertGeneration`?

![image](https://user-images.githubusercontent.com/68891831/147828942-c2a6a892-72d3-4d86-982d-0844ee62d070.png)

The blue curve is `BertModel`, the red curve is `BertGeneration`.


<|||||>Hey @zzhongzz,

Could you provide a training command I could run to see what could be the problem with `BertGeneration` ? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.