Nan when training LayoutLM_V2 Model
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.13.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU) : 1.10.0+cu111
- Tensorflow version (GPU): 2.7.0
- Flax version: not installed
- Jax version: not installed
- JaxLib version: not installed


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

@NielsRogge




## Information

The model used is LayoutLMv2:

The problem arises when using:
* [x] my own modified scripts:


The tasks I am working on is:
* [x] Document streaming segmentation
In my script, I try to determine when starts a new document with the objective to divide into segments a streaming of folders, so each segment can be interpreted as an independent document.

## To reproduce

Steps to reproduce the behavior:

1. Access to the colab notebook created to train LayoutLM_V2 (https://colab.research.google.com/drive/1MsEkj_WlGYDOs3vFcm1JxmMNLWj_Se78?usp=sharing)
2. Execute every cell in order
3. In the training loop, Accuracy, loss, and output will be printed, and there will be a moment when the output, Accuracy and Loss will become Nan.

## Expected behavior

The model trains, and despite if it accomplishes its task or not, the training loop ends without any Nan.
