Healthcare, Medcat, NLP, Hugging Face, Bert.
hf_tokenizer.convert_ids_to_tokens(i).startswith("##") else 'start' tokenizer = TokenizerNER(hf_tokenizer, id2type=id2type) Using the new tokenizer we can convert the dataset into the required format: encoded_dataset = dataset.map( lambda examples: tokenizer.encode(examples, ignore_subwords=True),