How to implement seq2seq attention mask conviniently?
BERT's attention mask is square, GPT's attention mask is triangular. How to implement seq2seq attention mask with transformers package conviniently? like the one appears in UniLM, a triangle concatenates a rectangle.
![unilm](https://user-images.githubusercontent.com/49787234/103397155-ff354180-4b71-11eb-8283-1c0f50f5b462.jpg)


The image is also on this link.
https://img-blog.csdnimg.cn/20191025102941935.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9qYWNrY3VpLmJsb2cuY3Nkbi5uZXQ=,size_16,color_FFFFFF,t_70<|||||>Hey @zhizeng8, 

In the future, it would be nice if such questions are posted in the forum: https://discuss.huggingface.co/ as it is not about a bug. 
To answer your question I'd use something like the following

```python
    import torch

    tgt_len = 5
    # make causal mask
    mask = torch.full((tgt_len, tgt_len), float("-inf"))
    mask_cond = torch.arange(mask.size(-1))
    mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
    # attend to encoder part
    mask[:, :3] = 0
```

This mask can however not just be input as an `attention_mask` to transformer models. Because BERT accepts 3d masks however with the 0-th index being the batch_size the above mask could be extended for all batches and input to BERT:

```python
3d_attention_mask = mask[None, :, :]

bert = BertModel.from_pretrained(...)
bert(input_ids, attention_mask=3d_attention_mask)
```<|||||>Thank you very much!
I think in my case the 3d_attention_mask is different for each instance in a batch, due to the different length of source and target sequence.<|||||>I have been reading this article recently. BertModel accepts 3d masks of dimensions [batch_size, from_seq_length, to_seq_length]. 
```
text='I love you' 
attention_mask = [[[1,0,0],[1,1,0],[1,1,1]]]
```
Maybe can help you.
Also I have a question.
```
text='I love you' 
attention_mask = [[[1,1,1],[1,1,1],[1,1,1]]]
```
 I found tensor of the last word 'you' are not same, I don't know the reason.<|||||>I know. Last word's tensor in first attention layer is same, but there are 12 attention layer. Hiddenstate may change.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.