Nan when training LayoutLM_V2 Model
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.13.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU) : 1.10.0+cu111
- Tensorflow version (GPU): 2.7.0
- Flax version: not installed
- Jax version: not installed
- JaxLib version: not installed


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

@NielsRogge




## Information

The model used is LayoutLMv2:

The problem arises when using:
* [x] my own modified scripts:


The tasks I am working on is:
* [x] Document streaming segmentation
In my script, I try to determine when starts a new document with the objective to divide into segments a streaming of folders, so each segment can be interpreted as an independent document.

## To reproduce

Steps to reproduce the behavior:

1. Access to the colab notebook created to train LayoutLM_V2 (https://colab.research.google.com/drive/1MsEkj_WlGYDOs3vFcm1JxmMNLWj_Se78?usp=sharing)
2. Execute every cell in order
3. In the training loop, Accuracy, loss, and output will be printed, and there will be a moment when the output, Accuracy and Loss will become Nan.

## Expected behavior

The model trains, and despite if it accomplishes its task or not, the training loop ends without any Nan.

Hi @Asocsar,

For training-related questions, can you use the [forum](https://discuss.huggingface.co/)?

I'm suspecting this has to do with the custom loss function being defined, rather than the model itself.

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.