Hi @attari.
In a nutshell what I did was split this problem into two different multi-class problems. In the first task I use a certain set of labels to label each word of the sentence (each word has an exclusive label, such as PoS-tagging). In the second task I use a different set of labels to do the same thing. Finally, I evaluate together the results obtained after the predictions of the two tasks made separately.
Looking at the legacy run_ner 6 example, I basically created two different tasks in the file tasks.py.