FlauBERT cannot perform MLM with customized tokenizer (added tokens to vocabulary)
Cannot run `run_mlm.py` for FlauBERT with customized tokenizer

## Environment info

- `transformers` version: 4.12.0.dev0
- Platform: Linux-5.14.7-gentoo-x86_64-x86_64-Intel-R-_Core-TM-_i7-5820K_CPU_@_3.30GHz-with-glibc2.17
- Python version: 3.8.11
- PyTorch version (GPU?): 1.9.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten
@LysandreJik
@sgugger 

Models:

https://huggingface.co/flaubert/flaubert_base_cased

Libraries:

```
\# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
aiohttp                   3.7.4.post0      py38h497a2fe_0    conda-forge
arrow-cpp                 3.0.0            py38h6b21186_4  
astroid                   2.8.2                    pypi_0    pypi
async-timeout             3.0.1                   py_1000    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
aws-c-common              0.4.57               he6710b0_1  
aws-c-event-stream        0.1.6                h2531618_5  
aws-checksums             0.1.9                he6710b0_0  
aws-sdk-cpp               1.8.185              hce553d0_0  
blas                      1.0                         mkl  
boost-cpp                 1.69.0            h11c811c_1000    conda-forge
boto3                     1.18.59                  pypi_0    pypi
botocore                  1.21.59                  pypi_0    pypi
brotli                    1.0.9                h7f98852_5    conda-forge
brotli-bin                1.0.9                h7f98852_5    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.17.1               h27cfd23_0  
ca-certificates           2021.10.8            ha878542_0    conda-forge
certifi                   2021.10.8        py38h578d9bd_0    conda-forge
cffi                      1.14.6           py38ha65f79e_0    conda-forge
chardet                   4.0.0            py38h578d9bd_1    conda-forge
charset-normalizer        2.0.7                    pypi_0    pypi
click                     8.0.3                    pypi_0    pypi
configparser              5.0.2                    pypi_0    pypi
cryptography              3.4.8            py38ha5dfef3_0    conda-forge
cudatoolkit               11.1.74              h6bb024c_0    nvidia
dataclasses               0.8                pyhc8e2a94_3    conda-forge
datasets                  1.12.1                     py_0    huggingface
dill                      0.3.4              pyhd8ed1ab_0    conda-forge
docker-pycreds            0.4.0                    pypi_0    pypi
double-conversion         3.1.5                h9c3ff4c_2    conda-forge
fastcore                  1.3.26                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.3.0              pyhd8ed1ab_0    conda-forge
freetype                  2.10.4               h5ab3b9f_0  
fsspec                    2021.10.0          pyhd8ed1ab_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
gitdb                     4.0.7                    pypi_0    pypi
gitpython                 3.1.24                   pypi_0    pypi
glog                      0.5.0                h48cff8f_0    conda-forge
gmp                       6.2.1                h2531618_2  
gnutls                    3.6.15               he1e5248_0  
grpc-cpp                  1.39.0               hae934f6_5  
huggingface-hub           0.0.19                   pypi_0    pypi
huggingface_hub           0.0.17                     py_0    huggingface
icu                       58.2              hf484d3e_1000    conda-forge
idna                      3.2                      pypi_0    pypi
importlib-metadata        4.8.1            py38h578d9bd_0    conda-forge
importlib_metadata        4.8.1                hd8ed1ab_0    conda-forge
intel-openmp              2021.3.0          h06a4308_3350  
isort                     5.9.3                    pypi_0    pypi
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9b                   h024ee3a_2  
krb5                      1.19.2               hcc1bbae_0    conda-forge
lame                      3.100                h7b6447c_0  
lazy-object-proxy         1.6.0                    pypi_0    pypi
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.35.1               h7274673_9  
libboost                  1.73.0              h3ff78a5_11  
libbrotlicommon           1.0.9                h7f98852_5    conda-forge
libbrotlidec              1.0.9                h7f98852_5    conda-forge
libbrotlienc              1.0.9                h7f98852_5    conda-forge
libcurl                   7.78.0               h0b77cf5_0  
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.3.0               h5101ec6_17  
libgomp                   9.3.0               h5101ec6_17  
libiconv                  1.15                 h63c8f33_5  
libidn2                   2.3.2                h7f8727e_0  
libnghttp2                1.43.0               h812cca2_0    conda-forge
libpng                    1.6.37               hbc83047_0  
libprotobuf               3.17.2               h4ff587b_1  
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.3.0               hd4cf53a_17  
libtasn1                  4.16.0               h27cfd23_0  
libthrift                 0.14.2               he6d91bd_1    conda-forge
libtiff                   4.2.0                h85742a9_0  
libunistring              0.9.10               h27cfd23_0  
libuv                     1.40.0               h7b6447c_0  
libwebp-base              1.2.0                h27cfd23_0  
lz4-c                     1.9.3                h295c915_1  
mccabe                    0.6.1                    pypi_0    pypi
metaflow                  2.4.0                    pypi_0    pypi
mkl                       2021.3.0           h06a4308_520  
mkl-service               2.4.0            py38h7f8727e_0  
mkl_fft                   1.3.0            py38h42c9631_2  
mkl_random                1.2.2            py38h51133e4_0  
multidict                 5.1.0            py38h27cfd23_2  
multiprocess              0.70.12.2        py38h497a2fe_0    conda-forge
ncurses                   6.2                  he6710b0_1  
nettle                    3.7.3                hbbd107a_1  
ninja                     1.10.2               hff7bd54_1  
numpy                     1.21.2                   pypi_0    pypi
numpy-base                1.20.3           py38h74d4b33_0  
olefile                   0.46               pyhd3eb1b0_0  
openh264                  2.1.0                hd408876_0  
openjpeg                  2.4.0                h3ad879b_0  
openssl                   1.1.1l               h7f8727e_0  
orc                       1.6.9                ha97a36c_3  
packaging                 21.0               pyhd8ed1ab_0    conda-forge
pandas                    1.2.5            py38h1abd341_0    conda-forge
pathtools                 0.1.2                    pypi_0    pypi
pillow                    8.3.1            py38h2c7a002_0  
pip                       21.2.4           py38h06a4308_0  
platformdirs              2.4.0                    pypi_0    pypi
promise                   2.3                      pypi_0    pypi
protobuf                  3.18.1                   pypi_0    pypi
psutil                    5.8.0                    pypi_0    pypi
pyarrow                   3.0.0            py38he0739d4_3  
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pylint                    2.11.1                   pypi_0    pypi
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
python                    3.8.11          h12debd9_0_cpython  
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-xxhash             2.0.2            py38h497a2fe_0    conda-forge
python_abi                3.8                      1_cp38    huggingface
pytorch                   1.9.1           py3.8_cuda11.1_cudnn8.0.5_0    pytorch
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1                    pypi_0    pypi
re2                       2021.08.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h27cfd23_0  
regex                     2021.10.8                pypi_0    pypi
requests                  2.26.0             pyhd8ed1ab_0    conda-forge
s3transfer                0.5.0                    pypi_0    pypi
sacremoses                0.0.46                   pypi_0    pypi
sentry-sdk                1.4.3                    pypi_0    pypi
setuptools                58.0.4           py38h06a4308_0  
shortuuid                 1.0.1                    pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_0  
smmap                     4.0.0                    pypi_0    pypi
snappy                    1.1.8                he1b5a44_3    conda-forge
sqlite                    3.36.0               hc218d9a_0  
subprocess32              3.5.4                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tk                        8.6.11               h1ccaba5_0  
tokenizers                0.10.3                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
torchaudio                0.9.1                      py38    pytorch
torchvision               0.10.1               py38_cu111    pytorch
tqdm                      4.62.3                   pypi_0    pypi
transformers              4.12.0.dev0              pypi_0    pypi
typing-extensions         3.10.0.2             hd3eb1b0_0  
typing_extensions         3.10.0.2           pyh06a4308_0  
uriparser                 0.9.3                he1b5a44_1    conda-forge
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
utf8proc                  2.6.1                h27cfd23_0  
wandb                     0.12.4                   pypi_0    pypi
wheel                     0.37.0             pyhd3eb1b0_1  
wrapt                     1.12.1                   pypi_0    pypi
xxhash                    0.8.0                h7f98852_3    conda-forge
xz                        5.2.5                h7b6447c_0  
yarl                      1.6.3            py38h497a2fe_2    conda-forge
yaspin                    2.1.0                    pypi_0    pypi
zipp                      3.6.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.9                haebb681_0  
```

## Information

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I am using the script from https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py for further pretraining (MLM task) a FlauBERT model on my corpus starting from the FlauBERT checkpoint. The issue occurs when adding tokens to the vocabulary of the tokenizer. Otherwise, the default tokenizer works fine. 

## To reproduce

Steps to reproduce the behavior:

1. add tokens to FlauBERT tokenizer

```python
tokenizer = AutoTokenizer.from_pretrained('flaubert/flaubert_base_cased')

tokenizer.vocab_size # should return 68729


additional_tokens = ["locateur", "1er", "1619", "ORDONNE", "CONDAMNE", 
 "TRIBUNAL", "C.c.Q.", "payable", "RÉSILIE", "82.1", 
 "locatrice", "L.R.L.", "CONSIDÉRANT",  "résilié", "REJETTE", 
 "1883", "RÉSERVE", "CONSTATE", "ACCUEILLE", "DÉCLARE", 
 "AUDIENCE", "PERMET", "RÉCLAME", "LOCATEUR", "AUTORISE"]


tokenizer.add_tokens(additional_tokens)

tokenizer.vocab_size # should return 68753

tokenizer.save_pretrained(PATH_TO_SAVING_DIRECTORY)
```

2. select custom tokenizer as argument of `run_mlm.py`. I made a bash script:

```bash
args=(

    --model_name_or_path flaubert/flaubert_base_cased # if not define, training from scratch
    --tokenizer_name PATH_TO_SAVING_DIRECTORY
    #--use_fast_tokenizer

    --train_file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/input/fpt_input_toy_train.txt
    --validation_file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/input/fpt_input_toy_valid.txt

    #--per_device_train_batch_size 8
    #--per_device_eval_batch_size 8

    --preprocessing_num_workers 12
    --mlm_probability 0.15

    #--pad_to_max_length # disable for dynamic padding (pad to the max length in the batch)
    --line_by_line
    --output_dir /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/output_custom-flaubert-base/

    #--overwrite_output_dir #enable if resume from checkpoint

    --do_train
    --do_eval
    --do_predict

    --evaluation_strategy epoch # default is no evaluation at all
    --num_train_epochs 20

    --save_strategy epoch
    --seed 21

    --logging_strategy steps
    --logging_steps 500
    --dataloader_num_workers 12

    --run_name a21_flaubert_mlm

    --dataloader_pin_memory
    )

export CUDA_VISIBLE_DEVICES=0
time python run_mlm.py "${args[@]}"
```



3. The full output is the following :

```
10/15/2021 07:19:32 - WARNING - __main__ - Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False                                            
10/15/2021 07:19:32 - INFO - __main__ - Training/evaluation parameters TrainingArguments(                                                                                            
_n_gpu=1,                                                                                                                                                                            
adafactor=False,                                                                                                                                                                     
adam_beta1=0.9,                                                                                                                                                                      
adam_beta2=0.999,                                                                                                                                                                    
adam_epsilon=1e-08,                                                                                                                                                                  
dataloader_drop_last=False,                                                                                                                                                          
dataloader_num_workers=12,                                                                                                                                                           
dataloader_pin_memory=True,                                                                                                                                                          
ddp_find_unused_parameters=None,                                                                                                                                                     
debug=[],                                                                                                                                                                            
deepspeed=None,                                                                                                                                                                      
disable_tqdm=False,                                                                                                                                                                  
do_eval=True,                                                                                                                                                                        
do_predict=True,                                                                                                                                                                     
do_train=True,                                                                                                                                                                       
eval_accumulation_steps=None,                                                                                                                                                        
eval_steps=None,                                                                                                                                                                     
evaluation_strategy=IntervalStrategy.EPOCH,                                                                                                                                          
fp16=False,                                                                                                                                                                          
fp16_backend=auto,                                                                                                                                                                   
fp16_full_eval=False,                                                                                                                                                                
fp16_opt_level=O1,                                                                                                                                                                   
gradient_accumulation_steps=1,                                                                                                                                                       
gradient_checkpointing=False,                                                                                                                                                        
greater_is_better=None,                                                                                                                                                              
group_by_length=False,                                                                                                                                                               
hub_model_id=None,
hub_strategy=HubStrategy.EVERY_SAVE,                                                                                                                                       [211/1901]
hub_token=<HUB_TOKEN>,                                                                                                                                                               
ignore_data_skip=False,                                                                                                                                                              
label_names=None,                                                                                                                                                                    
label_smoothing_factor=0.0,                                                                                                                                                          
learning_rate=5e-05,                                                                                                                                                                 
length_column_name=length,                                                                                                                                                           
load_best_model_at_end=False,                                                                                                                                                        
local_rank=-1,                                                                                                                                                                       
log_level=-1,                                                                                                                                                                        
log_level_replica=-1,                                                                                                                                                                
log_on_each_node=True,                                                                                                                                                               
logging_dir=/data/rali6/Tmp/salaunol/_NEXT/a21/fpt/output_custom-flaubert-base/runs/Oct15_07-19-32_octal16,                                                                          
logging_first_step=False,                                                                                                                                                            
logging_nan_inf_filter=True,                                                                                                                                                         
logging_steps=500,                                                                                                                                                                   
logging_strategy=IntervalStrategy.STEPS,                                                                                                                                             
lr_scheduler_type=SchedulerType.LINEAR,                                                                                                                                              
max_grad_norm=1.0,                                                                                                                                                                   
max_steps=-1,                                                                                                                                                                        
metric_for_best_model=None,                                                                                                                                                          
mp_parameters=,                                                                                                                                                                      
no_cuda=False,                                                                                                                                                                       
num_train_epochs=20.0,                                                                                                                                                               
output_dir=/data/rali6/Tmp/salaunol/_NEXT/a21/fpt/output_custom-flaubert-base/,                                                                                                      
overwrite_output_dir=False,                                                                                                                                                          
past_index=-1,                                                                                                                                                                       
per_device_eval_batch_size=8,                                                                                                                                                        
per_device_train_batch_size=8,                                                                                                                                                       
prediction_loss_only=False,                                                                                                                                                          
push_to_hub=False,                                                                                                                                                                   
push_to_hub_model_id=None,                                                                                                                                                           
push_to_hub_organization=None,                                                                                                                                                       
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,                                                                                                                                               
remove_unused_columns=True,                                                                                                                                                          
report_to=['wandb'],                                                                                                                                                                 
resume_from_checkpoint=None,                                                                                                                                                         
run_name=a21_flaubert_mlm,                                                                                                                                                           
save_on_each_node=False,                                                                                                                                                             
save_steps=500,                                                                                                                                                                      
save_strategy=IntervalStrategy.EPOCH,                                                                                                                                                
save_total_limit=None,                                                                                                                                                               
seed=21,                                                                                                                                                                             
sharded_ddp=[],                                                                                                                                                                      
skip_memory_metrics=True,                                                                                                                                                            
tpu_metrics_debug=False,  
tpu_num_cores=None,                                                                                                                                                                  
use_legacy_prediction_loop=False,                                                                                                                                                    
warmup_ratio=0.0,                                                                                                                                                                    
warmup_steps=0,                                                                                                                                                                      
weight_decay=0.0,                                                                                                                                                                    
xpu_backend=None,                                                                                                                                                                    
)                                                                                                                                                                                    
10/15/2021 07:19:32 - WARNING - datasets.builder - Using custom data configuration default-df960222b6433548                                                                          
10/15/2021 07:19:32 - INFO - datasets.builder - Overwrite dataset info from restored data version.                                                                                   
10/15/2021 07:19:32 - INFO - datasets.info - Loading Dataset info from /data/rali6/Tmp/salaunol/_NEXT/hf_datasets_cache/text/default-df960222b6433548/0.0.0/e16f44aa1b321ece1f87b0797
7cc5d70be93d69b20486d6dacd62e12cf25c9a5                                                                                                                                              
10/15/2021 07:19:32 - WARNING - datasets.builder - Reusing dataset text (/data/rali6/Tmp/salaunol/_NEXT/hf_datasets_cache/text/default-df960222b6433548/0.0.0/e16f44aa1b321ece1f87b07
977cc5d70be93d69b20486d6dacd62e12cf25c9a5)                                                                                                                                           
10/15/2021 07:19:32 - INFO - datasets.info - Loading Dataset info from /data/rali6/Tmp/salaunol/_NEXT/hf_datasets_cache/text/default-df960222b6433548/0.0.0/e16f44aa1b321ece1f87b0797
7cc5d70be93d69b20486d6dacd62e12cf25c9a5                                                                                                                                              
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 105.12it/s]
[INFO|configuration_utils.py:584] 2021-10-15 07:19:32,706 >> loading configuration file https://huggingface.co/flaubert/flaubert_base_cased/resolve/main/config.json from cache at /d
ata/rali6/Tmp/salaunol/_NEXT/transformers_cache/0b9ef58865bb61b2a44569c51b24b441c7b6b49ba63c659fc4ad5d61ffa011d6.c03a6cc0529664af7ebd7b4b385954d9cd0071c3d965d9377ab407e2eaa06918    
[INFO|configuration_utils.py:621] 2021-10-15 07:19:32,708 >> Model config FlaubertConfig {                                                                                           
  "amp": 1,                                                                                                                                                                          
  "architectures": [                                                                                                                                                                 
    "FlaubertWithLMHeadModel"                                                                                                                                                        
  ],                                                                                                                                                                                 
  "asm": false,                                                                                                                                                                      
  "attention_dropout": 0.1,                                                                                                                                                          
  "bos_index": 0,                                                                                                                                                                    
  "bos_token_id": 0,                                                                                                                                                                 
  "bptt": 512,                                                                                                                                                                       
  "causal": false,                                                                                                                                                                   
  "clip_grad_norm": 5,                                                                                                                                                               
  "dropout": 0.1,                                                                                                                                                                    
  "emb_dim": 768,                                                                                                                                                                    
  "embed_init_std": 0.02209708691207961,                                                                                                                                             
  "encoder_only": true,                                                                                                                                                              
  "end_n_top": 5,                                                                                                                                                                    
  "eos_index": 1,                                                                                                                                                                    
  "fp16": true,                                                                                                                                                                      
  "gelu_activation": true,                                                                                                                                                           
  "group_by_size": true,                                                                                                                                                             
  "id2lang": {                                                                                                                                                                       
    "0": "fr"                                                                                                                                                                        
  },                                                                                                                                                                                 
  "init_std": 0.02,                                                                                                                                                                  
  "is_encoder": true,                                                                                                                                                                
  "lang2id": {                                                                                                                                                                       
    "fr": 0                                                                                                                                                                          
  },    
  "lang_id": 0,                                                                                                                                                            [118/1901]
  "langs": [                                                                                                                                                                         
    "fr"                                                                                                                                                                             
  ],                                                                                                                                                                                 
  "layer_norm_eps": 1e-12,                                                                                                                                                           
  "layerdrop": 0.0,                                                                                                                                                                  
  "lg_sampling_factor": -1,                                                                                                                                                          
  "lgs": "fr",                                                                                                                                                                       
  "mask_index": 5,                                                                                                                                                                   
  "mask_token_id": 0,                                                                                                                                                                
  "max_batch_size": 0,                                                                                                                                                               
  "max_position_embeddings": 512,                                                                                                                                                    
  "max_vocab": -1,                                                                                                                                                                   
  "mlm_steps": [                                                                                                                                                                     
    [                                                                                                                                                                                
      "fr",                                                                                                                                                                          
      null                                                                                                                                                                           
    ]                                                                                                                                                                                
  ],                                                                                                                                                                                 
  "model_type": "flaubert",                                                                                                                                                          
  "n_heads": 12,                                                                                                                                                                     
  "n_langs": 1,                                                                                                                                                                      
  "n_layers": 12,                                                                                                                                                                    
  "pad_index": 2,                                                                                                                                                                    
  "pad_token_id": 2,                                                                                                                                                                 
  "pre_norm": false,                                                                                                                                                                 
  "sample_alpha": 0,                                                                                                                                                                 
  "share_inout_emb": true,                                                                                                                                                           
  "sinusoidal_embeddings": false,                                                                                                                                                    
  "start_n_top": 5,                                                                                                                                                                  
  "summary_activation": null,                                                                                                                                                        
  "summary_first_dropout": 0.1,                                                                                                                                                      
  "summary_proj_to_labels": true,                                                                                                                                                    
  "summary_type": "first",                                                                                                                                                           
  "summary_use_proj": true,                                                                                                                                                          
  "tokens_per_batch": -1,                                                                                                                                                            
  "transformers_version": "4.12.0.dev0",                                                                                                                                             
  "unk_index": 3,                                                                                                                                                         [81/1901]
  "use_lang_emb": true,                                                                                                                                                              
  "vocab_size": 68729,                                                                                                                                                               
  "word_blank": 0,                                                                                                                                                                   
  "word_dropout": 0,                                                                                                                                                                 
  "word_keep": 0.1,                                                                                                                                                                  
  "word_mask": 0.8,                                                                                                                                                                  
  "word_mask_keep_rand": "0.8,0.1,0.1",                                                                                                                                              
  "word_pred": 0.15,                                                                                                                                                                 
  "word_rand": 0.1,                                                                                                                                                                  
  "word_shuffle": 0                                                                                                                                                                  
}                                                                                                                                                                                    
                                                                                                                                                                                     
[INFO|tokenization_utils_base.py:1671] 2021-10-15 07:19:32,932 >> Didn't find file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/custom-flaubert_flaubert_base_cased/tokenizer.json. We won'
t load it.                                                                                                                                                                           
[INFO|tokenization_utils_base.py:1740] 2021-10-15 07:19:32,932 >> loading file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/custom-flaubert_flaubert_base_cased/vocab.json                 
[INFO|tokenization_utils_base.py:1740] 2021-10-15 07:19:32,933 >> loading file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/custom-flaubert_flaubert_base_cased/merges.txt                 
[INFO|tokenization_utils_base.py:1740] 2021-10-15 07:19:32,933 >> loading file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/custom-flaubert_flaubert_base_cased/added_tokens.json          
[INFO|tokenization_utils_base.py:1740] 2021-10-15 07:19:32,933 >> loading file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/custom-flaubert_flaubert_base_cased/special_tokens_map.json    
[INFO|tokenization_utils_base.py:1740] 2021-10-15 07:19:32,933 >> loading file /data/rali6/Tmp/salaunol/_NEXT/a21/fpt/custom-flaubert_flaubert_base_cased/tokenizer_config.json      
[INFO|tokenization_utils_base.py:1740] 2021-10-15 07:19:32,933 >> loading file None                                                                                                  
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,080 >> Adding locateur to the vocabulary                                                                                        
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,080 >> Adding 1619 to the vocabulary                                                                                            
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding ORDONNE to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding CONDAMNE to the vocabulary                                                                                        
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding TRIBUNAL to the vocabulary                                                                                        
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding C.c.Q. to the vocabulary                                                                                          
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding payable to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding RÉSILIE to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding 82.1 to the vocabulary                                                                                            
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding locatrice to the vocabulary                                                                                       
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding L.R.L. to the vocabulary                                                                                          
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding CONSIDÉRANT to the vocabulary                                                                                     
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,081 >> Adding résilié to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding REJETTE to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding 1883 to the vocabulary                                                                                            
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding RÉSERVE to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding CONSTATE to the vocabulary                                                                                        
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding ACCUEILLE to the vocabulary                                                                                       
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding DÉCLARE to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding AUDIENCE to the vocabulary                                                                                        
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding PERMET to the vocabulary                                                                                          
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding RÉCLAME to the vocabulary                                                                                         
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,082 >> Adding LOCATEUR to the vocabulary                                                                                        
[INFO|tokenization_utils.py:414] 2021-10-15 07:19:33,083 >> Adding AUTORISE to the vocabulary                                                                                        
[INFO|modeling_utils.py:1341] 2021-10-15 07:19:33,186 >> loading weights file https://huggingface.co/flaubert/flaubert_base_cased/resolve/main/pytorch_model.bin from cache at /data/
rali6/Tmp/salaunol/_NEXT/transformers_cache/36e988379f6754002245b872381a8b00f6c3a5e3bb887031aef3d85b89cf0122.fa9ba484eef0a0fa228af0c4972ef470103b3e08b15aee50f1a99acf3b33086e        
[INFO|modeling_utils.py:1606] 2021-10-15 07:19:40,271 >> All model checkpoint weights were used when initializing FlaubertWithLMHeadModel.                                                                                                                                                                                

[INFO|modeling_utils.py:1614] 2021-10-15 07:19:40,271 >> All the weights of FlaubertWithLMHeadModel were initialized from the model checkpoint at flaubert/flaubert_base_cased.      
If your task is similar to the task the model of the checkpoint was trained on, you can already use FlaubertWithLMHeadModel for predictions without further training.                
Running tokenizer on dataset line_by_line #0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.78ba/s]
Running tokenizer on dataset line_by_line #1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.89ba/s]
Running tokenizer on dataset line_by_line #2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.35ba/s]
Running tokenizer on dataset line_by_line #3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.45ba/s]
Running tokenizer on dataset line_by_line #4: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.32ba/s]
Running tokenizer on dataset line_by_line #5: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.38ba/s]
Running tokenizer on dataset line_by_line #6: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.37ba/s]
Running tokenizer on dataset line_by_line #7: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.26ba/s]
Running tokenizer on dataset line_by_line #8: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.34ba/s]
Running tokenizer on dataset line_by_line #9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.30ba/s]
Running tokenizer on dataset line_by_line #10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.44ba/s]
Running tokenizer on dataset line_by_line #11: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.50ba/s]
Running tokenizer on dataset line_by_line #0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.40ba/s]
Running tokenizer on dataset line_by_line #1: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.48ba/s]
Running tokenizer on dataset line_by_line #2: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.62ba/s]
Running tokenizer on dataset line_by_line #3: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.41ba/s]
Running tokenizer on dataset line_by_line #4: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.60ba/s]
Running tokenizer on dataset line_by_line #5: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.64ba/s]
Running tokenizer on dataset line_by_line #6: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.65ba/s]
Running tokenizer on dataset line_by_line #7: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.69ba/s]
Running tokenizer on dataset line_by_line #8: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.51ba/s]
Running tokenizer on dataset line_by_line #9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.70ba/s]
Running tokenizer on dataset line_by_line #10: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.90ba/s]
Running tokenizer on dataset line_by_line #11: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.00ba/s]
[INFO|trainer.py:540] 2021-10-15 07:20:26,264 >> The following columns in the training set  don't have a corresponding argument in `FlaubertWithLMHeadModel.forward` and have been ig
nored: special_tokens_mask.                                                                                                                                                          
[INFO|trainer.py:1196] 2021-10-15 07:20:26,296 >> ***** Running training *****█████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.70ba/s]
[INFO|trainer.py:1197] 2021-10-15 07:20:26,296 >>   Num examples = 4937                                                                                                              
[INFO|trainer.py:1198] 2021-10-15 07:20:26,297 >>   Num Epochs = 20████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.52ba/s]
[INFO|trainer.py:1199] 2021-10-15 07:20:26,297 >>   Instantaneous batch size per device = 8                                                                                          
[INFO|trainer.py:1200] 2021-10-15 07:20:26,297 >>   Total train batch size (w. parallel, distributed & accumulation) = 8███████████████████████████████| 1/1 [00:00<00:00,  5.71ba/s]
[INFO|trainer.py:1201] 2021-10-15 07:20:26,297 >>   Gradient Accumulation steps = 1                                                                                                  
[INFO|trainer.py:1202] 2021-10-15 07:20:26,297 >>   Total optimization steps = 12360███████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  5.91ba/s]
[INFO|integrations.py:500] 2021-10-15 07:20:26,298 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"                               
wandb: Currently logged in as: osalaun (use `wandb login --relogin` to force relogin)██████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  6.02ba/s]
wandb: Tracking run with wandb version 0.12.4                                                                                                                                        
wandb: Syncing run a21_flaubert_mlm                                                                                                                                                  
wandb:  View project at https://wandb.ai/osalaun/huggingface                                                                                                                         
wandb:  View run at https://wandb.ai/osalaun/huggingface/runs/1dlneyei                                                                                                               
wandb: Run data is saved locally in /u/salaunol/Documents/_2021_automne/pil0_pretrain_lm/wandb/run-20211015_072026-1dlneyei                                                          
wandb: Run `wandb offline` to turn off syncing.                                                                                                                                      

  0%|                                                                                                                                                      | 0/12360 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_mlm.py", line 552, in <module>
    main()
  File "run_mlm.py", line 501, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/transformers/trainer.py", line 1316, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/transformers/trainer.py", line 1849, in training_step
    loss = self.compute_loss(model, inputs)
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/transformers/trainer.py", line 1881, in compute_loss
    outputs = model(**inputs)
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/transformers/models/xlm/modeling_xlm.py", line 759, in forward
    outputs = self.pred_layer(output, labels)  # (loss, logits) or (logits,) depending on if labels are provided.
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/u/salaunol/anaconda3/envs/fpt/lib/python3.8/site-packages/transformers/models/xlm/modeling_xlm.py", line 664, in forward
    scores.view(-1, self.n_words), y.view(-1), reduction="elementwise_mean"
RuntimeError: shape '[-1, 68729]' is invalid for input of size 45651992

wandb: Waiting for W&B process to finish, PID 1168625... (failed 1). Press ctrl-c to abort syncing.
wandb:                                                                                
wandb: Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Synced a21_flaubert_mlm: https://wandb.ai/osalaun/huggingface/runs/1dlneyei
wandb: Find logs at: ./wandb/run-20211015_072026-1dlneyei/logs/debug.log
wandb: 

```

I am not sure of what the problem is, it seems that the new vocabulary size (68753) did not replace the default one (68729).


## Expected behavior

`run_mlm.py` should be able to perform MLM with FlauBERT with a customized tokenizer (tokens added to default vocabulary). 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey @oliviersalaun,

I think the problem could be that the tokenizer's vocabulary size is not equal to the model's vocabulary size after you add tokens to the tokenizer.

Could you maybe sure that after step 1.) that `model.config.vocab_size = tokenizer.vocab_size`? <|||||>Hey @patrickvonplaten,

Thanks for replying. There is indeed a mismatch between the `vocab_size`s, I noticed something strange :

```python
tokenizer_default = AutoTokenizer.from_pretrained("flaubert/flaubert_base_cased")
tokenizer_custom = AutoTokenizer.from_pretrained(PATH_TO_CUSTOM_TOKENIZER_FOLDER)

print(tokenizer_default.vocab_size)
print(len(tokenizer_default))
print(tokenizer_custom.vocab_size)
print(len(tokenizer_custom))
```

That gives:

```
68729
68729
68729
68753
```

It seems that `tokenizer_custom.vocab_size` ignored the added tokens. Is this normal?<|||||>I think this is expected. If you change the tokenizer (*e.g.* add new tokens), you also have to make sure that the model's input and output embedding matrix is resized correctly. 

E.g. you should call: 

`model.resize_embeddings(len(tokenizer))` on the model before training.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.