Thanks for the response! I’ve run additional test, in which I replaced Transformer model with single linear layer encoder, and everything ran without any problems, so it must be related only to the Transformer. I’m not sure about the weight sharing - I’ve looked at the Roberta model implementation in huggingface (I’m training distilroberta as an encoder):  transformers/modeling_roberta.py at master · huggingface/transformers · GitHub , and I don’t see there any weight sharing. Additionally, with gradient_checkpointing enabled, the backtrace points to the LayerNorm layer of RobertaOutput class (5th layer of the model). I’ll try to provide a reproducible example