How to define the self-attention layer with transformers
I want to use 'BertLayer'or 'RobertaLayer' to define a self-attention layer,but I can't find the definition of 'RobertaLayer'.Can you provide some relevant documents?Is this API cancelled? Thanks！

“”“
self.basicblocks = nn.ModuleList()
        self.n_layers = 3
        trans_heads = 8
        trans_drop = 0.1
        bert_config = BertConfig(hidden_size=self.config.hidden_size, num_attention_heads=trans_heads, attention_probs_dropout_prob=trans_drop)
        for layer in range(self.n_layers):
            self.basicblocks.append(BertLayer(bert_config))
”“”
Hey @193769981, `BertLayer` is defined here:

https://github.com/huggingface/transformers/blob/e68c3756fea7c811d02b8470539ae17ec3ec0e71/src/transformers/models/bert/modeling_bert.py#L445

`RobertaLayer` is a copy of `BertLayer`, so the two should be identical. 

https://github.com/huggingface/transformers/blob/e68c3756fea7c811d02b8470539ae17ec3ec0e71/src/transformers/models/roberta/modeling_roberta.py#L385<|||||>Hello, I have solved this problem.
I want to know what form the attention_mask should be when I use Roberta_Layer, and what values the <pad> and non-pad should be set.
![image](https://user-images.githubusercontent.com/44664739/147881376-3fb8a558-6030-4dbd-8f46-e0178ad91222.png)
<|||||>HF has a [glossary](https://huggingface.co/docs/transformers/glossary/) that provides explanations and examples of terms like `attention_mask`. I hope this helps!

On a secondary note, I recommend that you check out the [HF discussion forum](https://discuss.huggingface.co), which is more suitable for asking questions. The GitHub issue page is dedicated to bug reports, feature requests, etc.