Hi, I think the causal masking you are referring to happens here. https://github.com/huggingface/transformers/blob/ce2298fb5f84a8d0d8860c15fb677b7ada07a8ad/src/transformers/modeling_gpt2.py#L149 155 The bias parameter is a lower triangular matrix with (max_ctx_len, max_ctx_len) dimensions. The linked line performs slicing of this matrix with the appropriate sequence length for the current input.