Hi @fdb,
Thanks for the note. You are perfectly right in your assumptions.
In earlier versions of tokenizers/transformers the offsets of tokens were not necessarily available (because the encode operation was destructive). Because offsets were not available, the API did not send this information.
However thanks to tokenizers @anthony and in the near future 4.0 version of transformers, the encode operation won’t be destructive anymore, so offsets will be available.
There is a PR for transformers for this pipeline https://github.com/huggingface/transformers/pull/8781
And the API inference is already running the fix:
curl -X POST -d 'Hello Sarah Jessica Parker who Jessica lives in New York' https://api-inference.huggingface.co/models/dslim/bert-base-NER

[
{"entity_group":"PER","score":0.9960219860076904,"word":"Sarah Jessica Parker","start":6,"end":26},
{"entity_group":"PER","score":0.9771094918251038,"word":"Jessica","start":31,"end":38},
{"entity_group":"LOC","score":0.9994266927242279,"word":"New York","start":48,"end":56}
]


https://huggingface.co/dslim/bert-base-NER?text=Hello+Sarah+Jessica+Parker+who+Jessica+lives+in+New+York. should soon follow.