Wav2Vec2 adapter layer being ignored at random
### System Info

Hi there!

During training, the hidden states of the base Wav2Vec2 model seem to be randomly skipping the adapter layer [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1322).

Just repeatedly running a forward pass with the same inputs will, on occasion, produce different output sequence lengths. When this happens I've logged the shapes before the adapter layer is applied as well as after, and they are the same, indicating that the layer is being skipped completely.

### Who can help?

Pinging @patrickvonplaten and @anton-l, please tell me I'm going crazy.

### Information

- [X] The official example scripts
- [ ] My own modified scripts

I'm using the current Colab environment.

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("anton-l/wav2vec2-base-lang-id", 
                                      add_adapter=True, 
                                      adapter_stride=2,
                                      adapter_kernel_size=3, 
                                      num_adapter_layers=2)

model.train() # NB

dummy_input = torch.randn((1, 16000))

expected_output_sequence_length = 13

for _ in range(200):
    output_shape = model(input_values=dummy_input)[0].shape[1]
    if output_shape != expected_output_sequence_length:
        print(output_shape)

### Expected behavior

The above loop shouldn't print anything out.
Hi, I think it's because `Wav2Vec2AdapterLayer` layers got dropped out here.

https://github.com/huggingface/transformers/blob/61d3928bfb3029bceb5be3e68ca3d4bf8456758f/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1006-L1009

You can pass `layerdrop=0` to `from_pretrained()` to deactivate it.

cc @patrickvonplaten, I understand the use of layerdrop in transformers structure, but why should we also have it in CNNs (Wav2Vec2AdapterLayer) ?<|||||>> Hi, I think it's because `Wav2Vec2AdapterLayer` layers got dropped out here.
> 
> https://github.com/huggingface/transformers/blob/61d3928bfb3029bceb5be3e68ca3d4bf8456758f/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1006-L1009
> 
> You can pass `layerdrop=0` to `from_pretrained()` to deactivate it.
> 
> cc @patrickvonplaten, I understand the use of layerdrop in transformers structure, but why should we also have it in CNNs (Wav2Vec2AdapterLayer) ?

Wow, completely missed that. Thank you!<|||||>Hey @OllieBroadhurst! Are you using the adapter layer to fine-tune the Wav2Vec2 model standalone? The adapter layer works best when combining the Wav2Vec2 model in a sequence-to-sequence combination (see https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#warm-started-speech-encoder-decoder-model)<|||||>> Hey @OllieBroadhurst! Are you using the adapter layer to fine-tune the Wav2Vec2 model standalone? The adapter layer works best when combining the Wav2Vec2 model in a sequence-to-sequence combination (see https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#warm-started-speech-encoder-decoder-model)

Hi @sanchit-gandhi!

It actually is for an encoder-decoder model. The reason it caused an issue is that I'm passing the encoder outputs to `inputs_embeds` instead of `encoder_hidden_states` and the positional embedding dim was smaller than the encoder output dim whenever the adapter layer was skipped. So definitely an edge case :)