BERT with sequence pairs & padding
## ❓ Questions & Help

I am having trouble understanding how to setup BERT when doing a classification task like STS, for example, inputting two sentences and getting a classification of some sorts. I am using `BertForSequenceClassification` for this purpose. However, what boggles me is how to set up `attention_mask` and `token_type_ids` when using padding.

Let's assume two sentences: `I made a discovery.` and `I discovered something.`

Currently, I'll prepare the input as follows (assume padding).

1. Input IDs (encoded): `[CLS] I made a discovery. [SEP] I discovered something. [SEP] [PAD] [PAD] [PAD]`
2. `token_type_ids`: everything `0` by the first `[SEP]` (also included), **after** which everything will be marked as `1` (padding included).
3. `attention_mask`: `1` for everything but the padding.

And, of course, labels are trivial as they are not affected by padding. Anything wrong with my setup? Am I missing anything?
Hi! Yes, I think your understanding is correct. Your setup seems fine to me!<|||||>regarding the token_type_ids, shall we mark [PAD] token as 0 or 1? [PAD] by default does not belong to any of the two input sequences. Therefore it is ambiguous to determine whether it should be 0 or 1.<|||||>The default padding value for `token_type_ids` is 0 which is defined by `tokenizer._pad_token_type_id`. You can specify it to 1 by `tokenizer._pad_token_type_id=1`.