Activating gradient checkpointing
Working on a script which uses `GPTNeoForCausalLM` as the model object. As I understand it, gradient checkpointing requires checkpointing every layer. How do I change `GPTNeoForCausalLM` to incorporate gradient checkpointing, given that it doesn't show the layers explicitly but rather uses
```
self.transformer = GPTNeoModel(config)
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
```
PS. working on [visua-grounding](https://github.com/EleutherAI/visual-grounding/tree/main) with which @stas00 has already been a huge help, and knows the issues.
In some places I think "activation checkpointing" is referred to as "gradient checkpointing". The former sounds more logical as it's activations that aren't being saved.  So it's https://pytorch.org/docs/stable/checkpoint.html.

It should be already there, as you can see the `GPTNeoModel` has it setup:

https://github.com/huggingface/transformers/blob/bc2571e61c985ec82819cf01ad038342771c94d0/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L834<|||||>Ah. Thanks for the clarification.