Cannot load pretrained microsoft's layoutlm
## Environment info
     
- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-112-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help

@julien-c @sshleifer 

## Information

Model I am using: LayoutLM

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import LayoutLMTokenizer, LayoutLMForTokenClassification
import torch

tokenizer = LayoutLMTokenizer.from_pretrained('microsoft/layoutlm-base-uncased')
model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased')
```

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/anaconda3/envs/nlp/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    926             try:
--> 927                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    928             except Exception:

~/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    526         if _is_zipfile(opened_file):
--> 527             with _open_zipfile_reader(f) as opened_zipfile:
    528                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)

~/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/serialization.py in __init__(self, name_or_buffer)
    223     def __init__(self, name_or_buffer):
--> 224         super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
    225 

RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7faec0a9f193 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1f5b (0x7faec3c279eb in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7faec3c28c04 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x6c1ef6 (0x7faf0bb54ef6 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x295928 (0x7faf0b728928 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: PyCFunction_Call + 0x56 (0x5640b0549f76 in /home/user/anaconda3/envs/nlp/bin/python)
frame #6: _PyObject_MakeTpCall + 0x22f (0x5640b050785f in /home/user/anaconda3/envs/nlp/bin/python)
frame #7: <unknown function> + 0x18bfdc (0x5640b0555fdc in /home/user/anaconda3/envs/nlp/bin/python)
frame #8: PyVectorcall_Call + 0x71 (0x5640b0507041 in /home/user/anaconda3/envs/nlp/bin/python)
frame #9: <unknown function> + 0x18c92a (0x5640b055692a in /home/user/anaconda3/envs/nlp/bin/python)
frame #10: _PyObject_MakeTpCall + 0x1a4 (0x5640b05077d4 in /home/user/anaconda3/envs/nlp/bin/python)
frame #11: _PyEval_EvalFrameDefault + 0x4596 (0x5640b058ef56 in /home/user/anaconda3/envs/nlp/bin/python)
frame #12: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #13: _PyObject_FastCallDict + 0x20c (0x5640b055648c in /home/user/anaconda3/envs/nlp/bin/python)
frame #14: _PyObject_Call_Prepend + 0x63 (0x5640b0556733 in /home/user/anaconda3/envs/nlp/bin/python)
frame #15: <unknown function> + 0x18c8ca (0x5640b05568ca in /home/user/anaconda3/envs/nlp/bin/python)
frame #16: _PyObject_MakeTpCall + 0x1a4 (0x5640b05077d4 in /home/user/anaconda3/envs/nlp/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x475 (0x5640b058ae35 in /home/user/anaconda3/envs/nlp/bin/python)
frame #18: _PyEval_EvalCodeWithName + 0x2d2 (0x5640b0554a92 in /home/user/anaconda3/envs/nlp/bin/python)
frame #19: _PyFunction_Vectorcall + 0x1e3 (0x5640b0555943 in /home/user/anaconda3/envs/nlp/bin/python)
frame #20: <unknown function> + 0x10011a (0x5640b04ca11a in /home/user/anaconda3/envs/nlp/bin/python)
frame #21: _PyEval_EvalCodeWithName + 0x7df (0x5640b0554f9f in /home/user/anaconda3/envs/nlp/bin/python)
frame #22: <unknown function> + 0x18bd20 (0x5640b0555d20 in /home/user/anaconda3/envs/nlp/bin/python)
frame #23: <unknown function> + 0x10077f (0x5640b04ca77f in /home/user/anaconda3/envs/nlp/bin/python)
frame #24: _PyEval_EvalCodeWithName + 0x2d2 (0x5640b0554a92 in /home/user/anaconda3/envs/nlp/bin/python)
frame #25: PyEval_EvalCodeEx + 0x44 (0x5640b0555754 in /home/user/anaconda3/envs/nlp/bin/python)
frame #26: PyEval_EvalCode + 0x1c (0x5640b05e3edc in /home/user/anaconda3/envs/nlp/bin/python)
frame #27: <unknown function> + 0x24f083 (0x5640b0619083 in /home/user/anaconda3/envs/nlp/bin/python)
frame #28: <unknown function> + 0x140699 (0x5640b050a699 in /home/user/anaconda3/envs/nlp/bin/python)
frame #29: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #30: _PyGen_Send + 0x149 (0x5640b054edc9 in /home/user/anaconda3/envs/nlp/bin/python)
frame #31: _PyEval_EvalFrameDefault + 0x49a3 (0x5640b058f363 in /home/user/anaconda3/envs/nlp/bin/python)
frame #32: _PyGen_Send + 0x149 (0x5640b054edc9 in /home/user/anaconda3/envs/nlp/bin/python)
frame #33: _PyEval_EvalFrameDefault + 0x49a3 (0x5640b058f363 in /home/user/anaconda3/envs/nlp/bin/python)
frame #34: _PyGen_Send + 0x149 (0x5640b054edc9 in /home/user/anaconda3/envs/nlp/bin/python)
frame #35: <unknown function> + 0x1701cd (0x5640b053a1cd in /home/user/anaconda3/envs/nlp/bin/python)
frame #36: <unknown function> + 0x10075e (0x5640b04ca75e in /home/user/anaconda3/envs/nlp/bin/python)
frame #37: _PyFunction_Vectorcall + 0x10b (0x5640b055586b in /home/user/anaconda3/envs/nlp/bin/python)
frame #38: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #39: _PyFunction_Vectorcall + 0x10b (0x5640b055586b in /home/user/anaconda3/envs/nlp/bin/python)
frame #40: <unknown function> + 0x10075e (0x5640b04ca75e in /home/user/anaconda3/envs/nlp/bin/python)
frame #41: _PyEval_EvalCodeWithName + 0x2d2 (0x5640b0554a92 in /home/user/anaconda3/envs/nlp/bin/python)
frame #42: _PyFunction_Vectorcall + 0x1e3 (0x5640b0555943 in /home/user/anaconda3/envs/nlp/bin/python)
frame #43: <unknown function> + 0x18be79 (0x5640b0555e79 in /home/user/anaconda3/envs/nlp/bin/python)
frame #44: PyVectorcall_Call + 0x71 (0x5640b0507041 in /home/user/anaconda3/envs/nlp/bin/python)
frame #45: _PyEval_EvalFrameDefault + 0x1fdb (0x5640b058c99b in /home/user/anaconda3/envs/nlp/bin/python)
frame #46: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #47: <unknown function> + 0x18bd20 (0x5640b0555d20 in /home/user/anaconda3/envs/nlp/bin/python)
frame #48: <unknown function> + 0x10011a (0x5640b04ca11a in /home/user/anaconda3/envs/nlp/bin/python)
frame #49: <unknown function> + 0x215056 (0x5640b05df056 in /home/user/anaconda3/envs/nlp/bin/python)
frame #50: <unknown function> + 0x1847f3 (0x5640b054e7f3 in /home/user/anaconda3/envs/nlp/bin/python)
frame #51: <unknown function> + 0x140699 (0x5640b050a699 in /home/user/anaconda3/envs/nlp/bin/python)
frame #52: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #53: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #54: _PyFunction_Vectorcall + 0x1e3 (0x5640b0555943 in /home/user/anaconda3/envs/nlp/bin/python)
frame #55: <unknown function> + 0x10075e (0x5640b04ca75e in /home/user/anaconda3/envs/nlp/bin/python)
frame #56: <unknown function> + 0x215056 (0x5640b05df056 in /home/user/anaconda3/envs/nlp/bin/python)
frame #57: <unknown function> + 0x1847f3 (0x5640b054e7f3 in /home/user/anaconda3/envs/nlp/bin/python)
frame #58: <unknown function> + 0x140699 (0x5640b050a699 in /home/user/anaconda3/envs/nlp/bin/python)
frame #59: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #60: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #61: <unknown function> + 0x18bd20 (0x5640b0555d20 in /home/user/anaconda3/envs/nlp/bin/python)
frame #62: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #63: <unknown function> + 0x215056 (0x5640b05df056 in /home/user/anaconda3/envs/nlp/bin/python)


During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-6-29a9c0941587> in <module>
      3 
      4 tokenizer = LayoutLMTokenizer.from_pretrained('microsoft/layoutlm-base-uncased')
----> 5 model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased')
      6 

~/anaconda3/envs/nlp/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    927                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    928             except Exception:
--> 929                 raise OSError(
    930                     "Unable to load weights from pytorch checkpoint file. "
    931                     "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "

OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. 
```
## Expected behavior

Load pretrained layoutlm

Not a bug, when updating to torch 1.6 it works