Training not converging with `transformers==4.26.1`
### System Info

- `transformers` version: 4.26.1
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@sanchit-gandhi 

### Information

- [X] The official example scripts (hardly 3-4 line of code is changed)
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [] My own task or dataset (give details below)

### Reproduction

```
CUDA_VISIBLE_DEVICES=1 python run_speech_recognition_ctc.py \
	--train_dataset_name="./data_sub/rs_en/train" \
	--model_name_or_path="facebook/hubert-base-ls960" \
	--train_dataset_name ="./data_sub/rs_en/test" \
	--output_dir="./ft-full-run-test-hubert" \
	--overwrite_output_dir \
	--num_train_epochs="5" \
	--per_device_train_batch_size="4" \
	--gradient_accumulation_steps="2" \
	--learning_rate="3e-4" \
	--warmup_steps="300" \
	--evaluation_strategy="steps" \
	--text_column_name="transcription" \
	--length_column_name="input_length" \
	--save_steps="400" \
	--eval_steps="25" \
	--layerdrop="0.0" \
	--save_total_limit="3" \
	--freeze_feature_encoder \
	--chars_to_ignore , ? . ! \
	--group_by_length \
	--do_train --do_eval
```

In `run_speech_recognition_ctc.py` I made these minor changes in `DataCollatorCTCWithPadding` as it won't run otherwise. With `transformers==4.30.1` it runs perfectly but I currently have 4.26.1 and the eval loss is not going down even after 3 epochs. My dataset has just 1500 samples of librispeech.
```
   def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # split inputs and labels since they have to be of different lenghts and need
        # different padding methods
        input_features = [{"input_ids": np.array(feature["input_values"])} for feature in features]
        label_features = [{"input_ids": feature["labels"]} for feature in features]

        output = {}

        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt",
        )

        labels_batch = self.processor.pad(
            label_features,
            padding=self.padding,
            pad_to_multiple_of=self.pad_to_multiple_of_labels,
            return_tensors="pt",
        )
        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

        output["labels"] = labels
        output["input_values"] = batch["input_ids"]
        # output["attention_mask"] = batch["attention_mask"]

        if "attention_mask" in batch:
            output["attention_mask"] = batch["attention_mask"].to(torch.long)

        return output
```


### Expected behavior

Eval loss should not plateau at 2.91. It doesn't reduce any further. Also, what's ideal number of `warmup_steps` that you'd recommend?
Hi, @bhavitvyamalik thanks for raising an issue! 

Questions about customising scripts for your own requirements e.g. optimal warmup steps, are best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

If the training works for the most recent version of transformers, then there's really nothing for us to do. <|||||>I understand and appreciate your response. I apologize for any confusion caused by raising the issue here. Since I am utilizing `adapter-transformers` for my project, which is built upon `transformers v4.26.1`, I am primarily relying on the functionality of `transformers` itself, specifically for fine-tuning Hubert. My intention was to inquire whether there might be a bug within the Trainer that could potentially explain why my training is not converging as expected or the changes I made to the code (described above) as it was giving problems with v4.26.1, hence why I brought up the issue in this context. Thank you!
<|||||>@bhavitvyamalik It's possible there was a bug. If there was, it's now been resolved and so it's not something we would spend time digging into. If you wanted to dig into this yourself, you could always use `git bisect` to find which commit introduced a change of behaviour.

We're not responsible for maintenance of third-party libraries built upon this one. I would suggest opening an issue in the `adapter-transformers` library, possibly asking for the pinned transformers version to be increased. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.