BartForConditionalGeneration output is not dependent on input when trained from scratch
### System Info

- `transformers` version: 4.21.0
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.9.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@patil-suraj
@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I've been trying to pretrain Bart to use as a baseline for comparison with other models I'd like to evaluate. However, I've found that when trained from scratch, its outputs have nothing to do with the inputs. You can set the input as anything you want, and the output will always be the same. It essentially acts like a causal language model. The funny thing is, if you start with the pretrained Bart instead of the randomly initialised Bart, it works fine. Could there be some problem with the way cross attention parameters are initialised? Or maybe there's an issue with Seq2SeqTrainer. Equally likely is that I've made a mistake somewhere. If anyone can help I'd greatly appreciate it. Thanks in advance.

The following code reproduces the issue. This attempts to train the model on the simplest conceivable seq2seq task: output the input, exactly as it is. If Bart can't even learn that, there must surely be something wrong.

```python
from datasets import load_dataset
from transformers import BartForConditionalGeneration, BartTokenizer, Seq2SeqTrainer, Seq2SeqTrainingArguments

dataset = load_dataset("c4", "en", streaming=True)
seed, buffer_size = 42, 10_000
train_set = dataset['train'].shuffle(seed, buffer_size=buffer_size).with_format('torch')
val_set = dataset['validation'].shuffle(seed, buffer_size=buffer_size).take(5000).with_format('torch')

tokeniser = BartTokenizer.from_pretrained("facebook/bart-base")

def transform(data_array):
    texts = []
    for data in data_array:
        texts.append(data['text'])

    batch = tokeniser(texts, padding=True, truncation=True, max_length=max_length)
    
    with tokeniser.as_target_tokenizer():
        labels = tokeniser(texts, padding=True, truncation=True, max_length=max_length)
        
    batch['labels'] = labels['input_ids']
    
    for k in batch:
        batch[k] = torch.tensor(batch[k])
    
    return dict(batch)

config = BartConfig.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration(config)

batch_size = 2

args = Seq2SeqTrainingArguments(
    output_dir="checkpoints-bart-baseline-2",
    do_train=True,
    do_eval=True,
    evaluation_strategy="steps",
    eval_steps=5000,
    save_strategy="steps",
    save_steps=5000,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    learning_rate=1e-4,
    max_steps=50_000,
    fp16=True,
    remove_unused_columns=False,
)

trainer = Seq2SeqTrainer(
    model=model,
    args=args,
    data_collator=transform,
    train_dataset=train_set,
    eval_dataset=val_set,
)
trainer.train()

model.eval()
input_texts = [
    "Please provide a code sample that reproduces the problem you ran into.",
    "It can be a Colab link or just a code snippet.",
    "If you have code snippets, error messages, stack traces please provide them here as well.",
]

inputs = tokeniser(input_texts, padding=True)

input_ids = torch.tensor(inputs['input_ids']).cuda()
model.cuda()
output_ids = model.generate(input_ids)
print(tokeniser.batch_decode(output_ids, skip_special_tokens=False, clean_up_tokenization_spaces=False))
```

Output:

```
['</s><s>This entry was posted in Uncategorized. Bookmark the permalink.</s>',
 '</s><s>This entry was posted in Uncategorized. Bookmark the permalink.</s>', 
'</s><s>This entry was posted in Uncategorized. Bookmark the permalink.</s>']
```

### Expected behavior

I would expect the Bart model to learn an approximation of the function represented by the training data. Specifically, I would expect even the poorest approximation to produce different outputs depending on what input is given.
This has been resolved by using a smaller learning rate.<|||||>hello, 

I meet exactly the same problem. My bart model always generate the same content no matter what the input is. How you solve your problem?

Thanks 

<|||||>@xienian87 I retrained the model with a smaller learning rate and the problem went away. <|||||>I have the same problem. Bart generated the same output, no matter what the model inputs.<|||||>@enze5088 have you tried a smaller learning rate?<|||||>> @enze5088 have you tried a smaller learning rate?

The problem seems to disappear, when using smaller learning rates.