run_ner.py huge discrepancy between eval and predict (or "dev" and "test" evaluation modes)
## ❓ Questions & Help

I'm comparing two different pretrained Bert models on the NER task. One is a bert-base-multilingual-cased model, which works fine, consistently, as one would expect. The other is our own pretrained multilingual Bert, which is trained on fewer languages and has so far shown better results on those few languages. However, when running run_ner.py from here, it doesn't evaluate consistently. The `dev.txt` dataset is completely identical to the `test.txt`dataset, so I'd expect identical results, but I get
```
--do_eval
f1 = 0.7603833865814698
loss = 0.06039658671007991
precision = 0.7531645569620253
recall = 0.7677419354838709
```
and
```
--do_predict
f1 = 0.025925925925925925
loss = 0.41404916612165316
precision = 0.030434782608695653
recall = 0.02258064516129032
```
I also tried `--evaluate_during_training` flag and I get "solid" results already from very few steps and identical to `--do_eval` at the end of training, but `--do_predict` are always much worse. Surprisingly, though, this doesn't occur with bert-base-multilingual-cased model even if I save it to disk and point to that folder. Additionally if under the clause if args.predict I change the mode from "test" to "dev" in function evaluate() I get good results. I repeat, dev and test are identical files that differ only in name.

The problem occurs on any max sequence length, after deleting cache, etc. No errors are displayed during training or evaluation/prediction.
Please help if you have any ideas what might be going wrong.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
