Thank you for the explanation.
I see the source of my misunderstanding. As I mentioned, the tutorial 36 has this passage:

This is where we will use the offset_mapping from the tokenizer as mentioned above. For each sub-token returned by the tokenizer, the offset mapping gives us a tuple indicating the sub-token’s start position and end position relative to the original token it was split from.

What I didn’t fully understand is that the sentences in that NER example were already pre-split into tokens. I thought that if you pass in non-split sentences into the tokenizer, then it would return offset_mapping with the same values as the pre-split sentence since the tokenizer would still do tokenization.

Note that if you only want to detect the special tokens, you can use the special_tokens_mask the tokenizer can return if you add the flag return_special_tokens_mask=True .

I also want to mask out the sub-token pieces that were split off of tokens. I tried out the special_tokens_mask, and it only marks the added tokens for things like [CLS] but not sub-token pieces.