Hi @Seva,
If you use tokenizers (which are FastTokenizers within transformers.) then you have offsets whenever you call encode (if you use transformers you need to actually call  tokenizer._tokenizer.encode to call the underlying tokenizers object).
tokenizer = AutoTokenizer.from_pretrained('roberta-large')

encoded = tokenizer._tokenizer.encode("This is a test")
encoded.offsets
# [(0, 0), (0, 4), (5, 7), (8, 9), (10, 14), (0, 0)]
# <s>,  "This", "is", "a", "test", </s>

So from that you should be able to recover your information.
This information is available through Fast Tokenizer method calls too.