@sgugger Is there a bug with the GPT2 Tokenizer when encoding sentence pairs?
tokenizer = AutoTokenizer.from_pretrained('gpt2')
result = tokenizer(text='the needs of the many', text_pair='outweigh the needs of the few')
input_ids = result['input_ids']
print(tokenizer.decode(input_ids))
print(tokenizer.convert_ids_to_tokens(input_ids))

Output:
the needs of the manyoutweigh the needs of the few
['the', 'Ġneeds', 'Ġof', 'Ġthe', 'Ġmany', 'out', 'we', 'igh', 'Ġthe', 'Ġneeds', 'Ġof', 'Ġthe', 'Ġfew']

There should be a separator token between the two sentences. For GPT, the token should be $. It’s stated in the original GPT paper 2, section 3.3:

Textual entailment For entailment tasks, we concatenate the premise p and hypothesis h token
sequences, with a delimiter token ($) in between.

The other tokenizers, e.g. BertTokenizer and XLNetTokenizer, properly introduce separation tokens between sentence pairs.