Why don't we set use_cache=False in default when training?
### Feature request

Let's take GPT-2 as an example, in the current implementation (modeling_gpt2.py: Line 856~861):
```
if self.gradient_checkpointing and self.training:
    if use_cache:
        logger.warning_once(
            "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
        )
        use_cache = False
```
Why don't we set like this:
```
if self.training:    # As long as it is being trained, we set use_cache=True. 
    if use_cache:
        logger.warning_once(
            "`use_cache=True` makes no sense when training. Setting `use_cache=False`..."
        )
        use_cache = False
```
Because when training, `use_cache=True` makes no sense (at least for decoder-only auto-regressive model) and if you use gradient_checkpointing, it should be under training instead of inference.

### Motivation

Hello contributors, I realize that we set `use_cache=False` in default for almost all the transformer-based models. I understand that it can speed up the generation when using the cache from the previous step. However, when training (pertaining or fine-tuning), we don't need it and it consumes many memories when processing a very long sequence, especially when the model is very large. But it cannot provide us any advantage.

### Your contribution

If needed, I can help to correct this for all models in the transformers library.
We can't change the models as it would be a breaking change, sadly.