Masking in Pooling Layer from BERT Output?
In Keras when the embedding layers using masking it is propagate to layer afterwards like Pooling or RNN-Layers. I wonder if this holds for using transformers Bert Models? I.e. in the following are the attention masks used in the following Pooling as masking too? So, the averages do not include padding tokens?

    
```
    id_ = Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
    mask_ = Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
    atn_ = Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32)
    

    bert_model = TFBertModel.from_pretrained('bert-base-uncased')
    
    embedding = bert_model(id_, attention_mask=mask_, token_type_ids=atn_)[0]
    
    x = GlobalAveragePooling1D()(embedding)    #are here attention_mask used as mask?
    x = Dropout(0.2)(x)
    out = Dense(3, activation='softmax')(x)

    model = Model(inputs=[id_, mask_, atn_], outputs=out)
    model.compile(loss='sparse_categorical_crossentropy', optimizer=opt)
```


https://discuss.huggingface.co/t/bert-output-for-padding-tokens/1550
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
