Wav2vec model further training [RuntimeError: you can only change requires_grad flags of leaf variables]
### System Info

```shell
Latest version
```


### Who can help?

@patrickvonplaten, @anton-l
I use wav2vec model as part of my own pytorch model. 
`self.configuration = Wav2Vec2Config()
 self.wav2vec_feature = Wav2Vec2Model(self.configuration)
self.wav2vec_feature = self.wav2vec_feature.train()`
However, it raises error when I call wav2vec model in my own forward function.
The error is:
`File "/home/tiger/.local/lib/python3.7/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 440, in forward
    hidden_states.requires_grad = True`



### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

`configuration = Wav2Vec2Config()
 model = Wav2Vec2Model(configuration)
model = model.train()`

put this 'model' into a forward function in pytorch.

### Expected behavior

```shell
The wav2vec model inserted in my own model is expected to be trainable.
```

Hey @xinghua-qu,

Could you please provide a fully reproducible code snippet here? E.g. something along the lines:

```python
from transformers import Wav2Vec2Model
....
```

More than happy to look into solving it then - thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>@patrickvonplaten I am facing the same issue! Here is the snippet:
`from transformers import Wav2Vec2ForSequenceClassification
model = Wav2Vec2ForSequenceClassification.from_pretrained("facebook/wav2vec2-base-960h")`
when I freeze the FE `self.model.freeze_feature_extractor()` training is fine otherwise I get:
`RuntimeError: you can only change requires_grad flags of leaf variables.`