This actually feels more like a bug than a problem on your end. I suspect that the tokenization between the two is identical (i.e. <s> gets the same ID in both cases), but that means that the special_tokens_mask should also be the same. Best to wait for others who might be more certain.