ValueError: DistilBertModel does not support gradient checkpointing.
How to enable the "gradient_checkpointing" for DistilBert model ? However, it's working fine for the Bert model, I've followed the steps given on this page to enable it
https://huggingface.co/docs/transformers/v4.18.0/en/performance

I've gone through the huggingface code of respective classes and found that the feature is present only for the Bert model and not the DistilBert. 
https://github.com/huggingface/transformers/blob/188a8bfcccc6b862fe7ccc2859d977c01dd98136/src/transformers/models/bert/modeling_bert.py#L593

https://github.com/huggingface/transformers/blob/188a8bfcccc6b862fe7ccc2859d977c01dd98136/src/transformers/models/distilbert/modeling_distilbert.py#L470


Yes DistilBert does not support gradient checkpointing. DistilBERT is a small model, so that feature is not needed for it.<|||||>I want to run this model across large batch sizes to see how much I can benefit from this. Is there any way I can enable for this model as well using torch.utils.checkpoint.checkpoint, but not sure where to apply checkpointing for this.  <|||||>I want to try out DeepSpeed’s activation checkpointing but can't use this on above model as it requires to enable the "gradient_checkpointing" flag in the HF trainer.
I was going through the DeepSpeed details on the below page and it mentions that we've to enable the "gradient_checkpointing" flag in HF trainer to use this 

"**HF Transformers models don’t know anything about DeepSpeed’s activation checkpointing,** so if you try to enable that feature in the DeepSpeed config file, nothing will happen." 
(https://huggingface.co/docs/transformers/main_classes/deepspeed)

What code changes required I need to do to replace with the Deepspeed API or enable the  model.gradient_checkpointing_enable() for distilbert
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>> Yes DistilBert does not support gradient checkpointing. DistilBERT is a small model, so that feature is not needed for it.

I don't think this feature is redundant if we want to train it with extremely large batch size. To my knowledge, minilm, which has fewer parameters than distilbert, supports gradient checkpointing.<|||||>Yeah, I agree jordan. When I tried to compare some transformer models, I could not train DistilBert because of the large batch size while i could train bert/roberta.