RwkvForCausalLM does not support gradient checkpointing.
### System Info

Is there some reason for RwkvForCausalLM does not support gradient checkpointing, since  RWKV-LM supports it?

@ArthurZucker and @younesbelkada

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

`model.gradient_checkpointing_enable()`

```
ValueError(f"{self.__class__.__name__} does not support gradient checkpointing.")
ValueError: RwkvForCausalLM does not support gradient checkpointing.
```

### Expected behavior

No errors, as long as RWKV-LM supports it.
Thanks for reporting @jonataslaw 
https://github.com/huggingface/transformers/pull/24955 has introduced the GC support for RWKV models, can you try that out by installing `transformers` from source and let us know how it goes?

```bash
pip uninstall transformers
pip install git+https://github.com/huggingface/transformers.git
```<|||||>Thanks for the quick update!
I tested your PR, and it works like a charm.
However it stops with an error about 10% of training (during eval):

  File "/home/jonataslaw/miniconda3/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/jonataslaw/miniconda3/lib/python3.9/site-packages/transformers/models/rwkv/modeling_rwkv.py", line 645, in forward
    self._rescale_layers()
  File "/home/jonataslaw/miniconda3/lib/python3.9/site-packages/transformers/models/rwkv/modeling_rwkv.py", line 738, in _rescale_layers
    block.attention.output.weight.quant_state[0].div_(
RuntimeError: result type Float can't be cast to the desired output type Byte


Edit: NVM, it is a unrelated problem about inference.<|||||>Ohh, I get it, it is the nested quantization problem related in https://github.com/huggingface/transformers/issues/23848<|||||>Yes, sadly nested quantization is not supported for RWKV, please use the un-nested one ! <|||||>Thanks for the update.
I will change my code and stay alert in case it changes in the future.

Thanks again for fixing the GC, it helps a lot.<|||||>Thanks very much @jonataslaw  !