Run_ner.py falsely aligns prediction list
## Environment info
- `transformers` version: 3.5.1
- Platform: Linux-5.9.1-kd-cluster-x86_64-with-glibc2.10
- Python version: 3.8.0
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik 
@stefan-it 

## Information

I am using the bert-base-german-cased model, which I trained on an NER task to predict entity labels.
When I run prediction on the test dataset, BERT tells me that the maximum sequence length (300) is exhausted and so no predictions will be made on a number of items.
However, this seems to be a wrong error report. The problem appears to be that in the align_predictions function, the preds_list variable gets a wrong dimension - the predictions are made correctly, but the index is shifted, so that they point to the wrong words. 
An example of what I mean: 
```
Amtsgericht Ort
Leipzig Ort
Abteilung O
für O
Strafsachen O
```
becomes
```
Amtsgericht O
Leipzig O
Abteilung O
für Ort
Strafsachen Ort
```
in the preds_list.
The write_predictions_to_file function then gets tangled up by this shifted index and (I think falsely) declares a sequence length error.

Strangely, another test document works just fine, without any difference between them that I could find. No sequence length error and no false indices there.

The problem arises when using:
* [] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I'm afraid it isn't so easy to reproduce because it requires the trained model and the data.

The align_predictions function:
```
    def align_predictions(predictions: np.ndarray, label_ids: np.ndarray) -> Tuple[List[int], List[int]]:
        preds = np.argmax(predictions, axis=2)

        batch_size, seq_len = preds.shape

        out_label_list = [[] for _ in range(batch_size)]
        preds_list = [[] for _ in range(batch_size)]

        for i in range(batch_size):
            for j in range(seq_len):
                if label_ids[i, j] != nn.CrossEntropyLoss().ignore_index:
                    out_label_list[i].append(label_map[label_ids[i][j]])
                    preds_list[i].append(label_map[preds[i][j]])

        return preds_list, out_label_list
```

Console output:
```
01/21/2021 11:07:01 - INFO - filelock -   Lock 140422091422400 released on /home/IAIS/tschmude/bert_remote/examples/token-classification/Data_processing_scripts/CrossVal_Files/Rotation/Test_file_swap/cached_test_BertTokenizer_340.lock
/home/IAIS/tschmude/anaconda3/envs/bert_env_remote/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:64: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
  0%|                                                     | 0/1 [00:00<?, ?it/s]/home/IAIS/tschmude/anaconda3/envs/bert_env_remote/lib/python3.8/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
01/21/2021 11:07:30 - INFO - run_ner -   
                                    precision    recall  f1-score   support
                             Datum       1.00      1.00      1.00         1
                    Gestaendnis_ja       1.00      1.00      1.00         1
                               Ort       0.50      1.00      0.67         1
                    Schadensbetrag       1.00      1.00      1.00         1
Strafe_Gesamtfreiheitsstrafe_Dauer       1.00      1.00      1.00         1
                 Strafe_Tatbestand       0.00      0.00      0.00         0
             Taeter_Drogenbezug_ja       1.00      1.00      1.00         1
                         micro avg       0.55      1.00      0.71         6
                         macro avg       0.79      0.86      0.81         6
                      weighted avg       0.92      1.00      0.94         6
/tmp/pycharm_project_44/src/transformers/trainer.py:1174: FutureWarning: This method is deprecated, use `Trainer.is_world_process_zero()` instead.
  warnings.warn("This method is deprecated, use `Trainer.is_world_process_zero()` instead.", FutureWarning)
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'Angeklagte'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'trägt'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'die'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'Kosten'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'des'.
01/21/2021 11:08:52 - WARNING - tasks -   Maximum sequence length exceeded: No prediction for 'Verfahrens'.
(and a lot more of these)
```
Config that the model was trained on:
```
Training Arguments:
	Data directory: ...ng_scripts/CrossVal_Files/Rotation/Train_file_swap
	Model: ...bert-base-german-cased
	Epochs: 8
	Seq length: 300
	Learning rate: 5e-05
	Batch size: 16
	Seed: 105
	Do Train: True
	Do Eval: True
	Do Test: True
```

## Expected behavior

That the predictions for the documents are correctly listed and written to file without a sequence length error or shifted indices.

Thank you for your help!
You can check https://github.com/uf-hobi-informatics-lab/ClinicalTransformerNER
We essentially wrap the models in transformers with our NER implementation that can handle sentences longer than the  max_len.<|||||>Hi @Stimmot ,

did you run the `scripts/preprocess.py` to make sure that there are no sentences > 300 subtokens in your final data splits :thinking: 

This should heavily prevent these kind of "maximum sequence length execeeded" errors :)<|||||>Thank you for the response @stefan-it, but I don't think it actually has to do with the sequence lengths. The longest sentences in the documents are around 200 tokens, not even near the maximal length.
Besides, the script crashes at some point with the following error:
```
Traceback (most recent call last):
  File "/tmp/pycharm_project_44/examples/token-classification/run_ner_crossval.py", line 226, in <module>
    main(sys.argv[1])
  File "/tmp/pycharm_project_44/examples/token-classification/run_ner_crossval.py", line 153, in main
    result_string, pred_dict = run_ner.main(json_config)
  File "/tmp/pycharm_project_44/examples/token-classification/run_ner.py", line 359, in main
    token_classification_task.write_predictions_to_file(writer, f, preds_list)
  File "/tmp/pycharm_project_44/examples/token-classification/tasks.py", line 53, in write_predictions_to_file
    elif preds_list[example_id]:
IndexError: list index out of range
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.70it/s]
```
The script doesn't work through the predictions list as it normally would, since the indices are shifted. Any other ideas what this could be?

(Thank you @bugface as well but as said above I don't think it's because of the sequence length)<|||||>@stefan-it I now also used the preprocess.py script just to be sure, but unfortunately it didn't change anything.<|||||>Hi @Stimmot ,

I think I found an interesting information in your provided log: `cached_test_BertTokenizer_340.lock`

That means, that the dataset was initially pre-processed with a sequence length of 340! Then I think you changed the max. sequence length to 300, but your ner script is still using the cached pre-processed test dataset that has a max. sequence length of 340.

Could you try to remove all `cached*` files, so that the dataset features are newly written :thinking: Hope this helps :)<|||||>Thanks @stefan-it, it had indeed to do with the cached models.
I built a script that one by one takes test documents and runs the run_ner.py script on them, however, it seems that it saves a cached version of the model for the first document (which it predicts correctly) and then uses this cached version for all subsequent ones. The cached dimensions of the document don't work on the new ones so, naturally, it runs into errors.

The solution is to let the model be rebuilt after each run, without using cached verisons.

Another quick question on that: is there an option to tell the run_ner.py script not to build cached models, so that it will use new ones each time it predicts?<|||||>Yeah, this can be done via `--overwrite_cache` argument :hugs: <|||||>Thank you very much, worked perfectly!<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.