BERT pretraining: [SEP] vs. Segment Embeddings?
I’m confused about the differences between the intent of the [SEP] tokens and Segment Embeddings applied to the input of BERT during pretraining.

As far as I’ve understood, the [SEP] tokens are inserted between sentence A and B to enable the model’s ability to distinguish between the two sentences for BERTs Next-Sentence Prediction pretraining-task. Similarly, the Segment Embeddings are added to the input embeddings to alter the input, creating another opportunity for the model to learn that sentence A and B are distinct things.

However, these seem to be facilitating the same purpose. Why can’t BERT be trained on only Segment Embeddings, omitting [SEP] tokens? What additional information do [SEP] tokens conceptually provide, that the Segment Embeddings don’t?

Furthermore, [SEP] tokens aren’t used directly anyways. NSP is trained on the [CLS] embeddings, which I understand to sort of represent an embedding of sentence continuity.
From the BERT paper: "We differentiate the sentences in two ways. First, we separate them with a special token ([SEP]). Second, we add a learned embedding to every token indicating whether it belongs to sentence A or sentence B."

Deep learning, as you may now, is a lot of experimenting, and in this case, it was a design choice. I guess you could try to omit the [SEP] token, perhaps it doesn't add much information to the model. Or omit the token type embeddings, and check whether the results are significantly different. 

To give another example, people are experimenting with all kinds of position encodings (include absolute ones, as in BERT, relatives ones, as in T5, sinusoidal ones, as in the original Transformer, now there are rotary embeddings, as in the new RoFormer paper)... 

So the question you're asking is a genuine research question :)   <|||||>Thank you for the quick answer, good to know! I was suspecting it might be something along these lines :)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.