I also tried to tackle the imbalance with setting some of the “O” tags to -100 to achieve a more balanced dataset. Similar to what’s going on with the sub-tokens. This helps a bit but performance is still low.
Additionally, after mapping all entities to a single class I called “Entity”  I saw F1 scores around 70%. Though the task is very simplified now.
Would be interesting to know if such a (poor) performance is expected.