Where is the casual mask when using BertLMHeadModel and set config.is_decoder = True?
I hope to use BERT for the task of causal language modeling.  

`BertLMHeadModel ` seems to meet my needs, but I did not find any code snippets about the causal mask, even if I set the `config.is_decoder=True`.

I only find the following related code in https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L968.

however, I do not have any values to pass into the argument `encoder_hidden_states` when doing causal language modeling.
So maybe the causal mask does not work?

```
if self.config.is_decoder and encoder_hidden_states is not None:
    encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.size()
    encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
    if encoder_attention_mask is None:
        encoder_attention_mask = torch.ones(encoder_hidden_shape, device=device)
    encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
else:
    encoder_extended_attention_mask = None
```
Hi @Doragd, BERT is an encoder model, and is therefore ill-suited to the causal language modeling task. Is there a reason you would like to use that model specifically for causal language modeling?<|||||>Hi, @LysandreJik  I just apply causal language modeling as an auxiliary task to lead stable training of our model.  I should have implemented this process myself, but I found this class `BertLMHeadModel`. However, I did not find any code snippet to implement causal mask. I would like to know that if is_decoder=True is set in BERT, can causal language modeling be achieved correctly?<|||||>cc @patrickvonplaten <|||||>Setting `is_decoder=True` automatically creates a causal mask in those lines of code: https://github.com/huggingface/transformers/blob/7fae5350528474c29b664ebb4df5bbc8104b48ec/src/transformers/modeling_utils.py#L266