Tensorboard missing values
## Environment info

- `transformers` version: 4.12.5
- Platform: Ubuntu 18
- Python version: 3.8

### Who can help

Models:
- Wav2Vec2 @patrickvonplaten @anton-l 
## Information

Model I am using is Wav2Vec2ForCTC and it is regarding the eval/loss in Tensorboard.

For some (most) of my experiments I'd notice missing values for the eval/loss:

![image](https://user-images.githubusercontent.com/43335432/149923148-3b82e453-2447-4ba3-b8ff-6143827f1a81.png)

![image](https://user-images.githubusercontent.com/43335432/149923447-7ccb5bd8-2250-43c0-9ee1-2516bd183b6e.png)

I am not doing anything special and I am essentially following the basic fine-tuning examples.

Any idea why I am seeing this? 
That's strange @severo do you maybe have any idea?<|||||>BTW, @stefan-falk feel free to share a link to your training repo here if the 30% WER seems to high for you. There are some hidden tricks on how to evaluate and train wav2vec2-base, *e.g.* https://github.com/pytorch/fairseq/issues/3227<|||||>No idea. @stefan-falk would you be able to share some of these tensorboard trace files (`*.tfevents.*`)?<|||||>@severo I've added the files as I found them in my logging-directory: 

[logs.zip](https://github.com/huggingface/transformers/files/7895914/logs.zip)

@patrickvonplaten 

I am using the same method as in your tutorial with the small change of using my own implementation for the error rates:

```python
def compute_metrics(processor, pred):
    pred_logits = pred.predictions
    pred_ids = np.argmax(pred_logits, axis=-1)

    pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

    pred_str = processor.batch_decode(pred_ids)

    # we do not want to group tokens when computing the metrics
    label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

    wer = error_rate(targets=label_str, predictions=pred_str, tokens="words")
    cer = error_rate(targets=label_str, predictions=pred_str, tokens="characters")

    return {"wer": wer, "cer": cer}
```

The reason is because `load_metric("wer")` uses `jiwer`, a library that constantly breaks on macOS somehow, so I replaced it.

Debugging this method tells me that all decoded labels (`label_str`) appear to be fine:

![image](https://user-images.githubusercontent.com/43335432/150109982-9e6b00db-ba5e-4492-995e-6d42213000a9.png)

<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Unstale<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.