Can the use of [SEP] reduce the information extraction between the sentences?
Hello. I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. [SEP] can’t stop one sentence from extracting information from another sentence. However, I have a question.
If I have 2 sentences, which are s1 and s2, and our fine-tuning task is the same. In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]. In another, I make the input look like [CLS] + s1 + s2 + [SEP]. When I input them to BERT respectively, what is the difference between them? Will the s1 in second one integrate more information from s2 than the s1 in first one does? Will the token embeddings change a lot between the 2 methods?
Thanks for any help!
I think so. Ultimately you should have s1 and s2 in input, your [CLS] + s1 + s2 + [SEP] will be equivalent to `[CLS] + s1 + [SEP]` in `[CLS] + s1 + [SEP] + s2 + [SEP]` where `s1` now is the concatenation of `s1` and `s2`. I don't think that's what you want to do.