layerdrop in Wav2Vec2Adapter
### System Info

Hi!

As mentioned in https://github.com/huggingface/transformers/issues/20451, the layer dropout in `Wav2Vec2Adapter` may produce outputs with different lengths.

I understand the use of layerdrop in transformer structure, but do we need it in CNNs (`Wav2Vec2Adapter`)?

https://github.com/huggingface/transformers/blob/61d3928bfb3029bceb5be3e68ca3d4bf8456758f/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L1006-L1009

### Who can help?

cc @anton-l @patrickvonplaten 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I copied the reproduction code of @OllieBroadhurst  in https://github.com/huggingface/transformers/issues/20451

```python
from transformers import Wav2Vec2Model

model = Wav2Vec2Model.from_pretrained("anton-l/wav2vec2-base-lang-id", 
                                      add_adapter=True, 
                                      adapter_stride=2,
                                      adapter_kernel_size=3, 
                                      num_adapter_layers=2)

model.train() # NB

dummy_input = torch.randn((1, 16000))

expected_output_sequence_length = 13

for _ in range(200):
    output_shape = model(input_values=dummy_input)[0].shape[1]
    if output_shape != expected_output_sequence_length:
        print(output_shape)
```

### Expected behavior

The above loop shouldn't print anything out.
@sanchit-gandhi could you take a look here? <|||||>Hey @bofenghuang! Thanks for opening this issue. 

For context, we use the adapter layer when combining the Wav2Vec2 model in a sequence-to-sequence combination (see https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#warm-started-speech-encoder-decoder-model). Here, the purpose of the adapter layer is to better match the time scale of the encoder with that of the decoder (see aforementioned doc).

In this respect, it's fine if the CNN downsamples the Wav2Vec2 output sequence at a stochastic rate for all training samples. This should add some 'robustness' to our text decoder which has to infer the correct target transcription from Wav2Vec2 output sequences of slightly varying length.

You can also disable layer drop by setting `layerdrop=0.0` in the config: https://huggingface.co/facebook/wav2vec2-base-960h/blob/main/config.json#L59<|||||>Thanks @sanchit-gandhi !