RuntimeError: expected dtype Float but got dtype Long - run_lm_finetuning.py
# 🐛 Bug

## Information

I'm using my Camembert-based language model on Italian language (built from scratch).

I'm trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) to fine-tune my language model on a dataset. 

@julien-c suggested my to add  `--line_by_line`  to my launch script, beacuse without that flag, the program blocked on the tokenization of the training set. That advide let the program work. But after some hours, the program crashes with a strange Runtime Error: in the assignment of 10 % of random words to masks at [line 218](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L218) in the _mask_tokens()_ function:
```python
# 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random] #it crashes here
```

The error was this:
`RuntimeError: expected dtype Float but got dtype Long
`
It's a strange error, beacuse it crashes after several minutes or hours after the launch. The time of the proper functioning of program seems random: sometimes the script completes 3/4 epochs and then it crashes, sometimes it crashes before the end of the first epoch.
 
## To reproduce

I launched this:
```bash
python3 run_lm_finetuning.py \
--train_data_file /path/to/train.txt \
--eval_data_file /path/to/eval.txt \
--output_dir /path/to/output \
--mlm \
--do_train \
--do_eval \
--model_type camembert \
--model_name_or_path /path/to/my/model \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--overwrite_output_dir \
--overwrite_cache \
--max_steps 500000 \
--block_size 128 \
--save_steps 50000 \
--eval_all_checkpoints \
--line_by_line
```

I got this error in the middle of 6th epoch:
```
File "run_lm_finetuning.py", line 801, in <module>51:22,  4.94it/s]
    main()
  File "run_lm_finetuning.py", line 750, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 342, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 222, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  55%|█████▍    | 6/11 [20:26:33<17:02:07, 12265.60s/it]
Iteration:  69%|██████▊   | 33378/48603 [1:47:45<49:09,  5.16it/s]
```

I'm managing to run the code anyway, restarting the program using the flag `--model_name_or_path` and giving the last saved checkpoint rather then the original language model every time it crashes.

I printed `inputs[indices_random]` and `random_words[indices_random]` beacause are the two variables in the line in which the program crashes:

- The code crashes with this 2 variables:
```
inputs[indices_random] = tensor([1173.])
Random_words[indices_random] = tensor([4220])

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 797, in <module>
    main()
  File "run_lm_finetuning.py", line 747, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 349, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 229, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  60%|██████    | 3/5 [14:31:21<9:40:54, 17427.18s/it]
```
- while before the crash the code enters the _mask_tokens()_ function corretly and prints lines like these:
```
inputs[indices_random] = tensor([19807,    78,    51,  1204])
Random_words[indices_random] = tensor([14538, 15381, 30255,  3778])
```

In my opinion the only difference is that **tensor([1173.])** in the crash example, contains a not integer value (there is the '.' at the end of the number, while all the other times not. Maybe with a cast of `inputs[indices_random]` it would work.

## Environment info

- `transformers` version: 2.4.1
- Platform: Linux-4.4.0-108-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
@paulthemagno thanks for creating this. My environment is exactly the same, except I am running the lm fine tuner on a Python 3.7.3 environment.  @LysandreJik  asked for more information. A couple of more inputs although I am not sure if this is going to help. This problem is not happening when I subset my dataset and run the code. Neither on training nor evaluation.  So this tells me there is a data problem somewhere.

So I caught the runtime error and output the results of those data objects for my dataset on line 120-122 that is posted above. Only unusual thing I see is that one of the examples for this batch where the code errors out is all zeros for the input tensor. 
```

tensor([[    0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.],
[ 2004.,  2019.,  5587., 10497.,  2819.,  2000.,  1996.,  2194.,  2015.,
          2128.,  8569., 28200., 10896.,  1010.,  3531.,  2421.,  1037.,  2862.,
          1997.,  2169.,  8875.,  2073.,  1996.,  2194.,   103.,  2015.,  2007.,
          1996.,   103.,  1997.,  1996.,  2060.,  4243.,  2000.,   103.,  8946.,
          3388.,  1012.,     0.,     0.,     0.,     0.,     0.,     0.,     0.,
             0.,     0.,     0.,     0.,     0.,     0.],
...
```
My data is split as 1) each sentence in one line, 2) and the documents are split by an empty line as recommended in the documentation. I looked at that particular document but I do not see anything unusual but maybe some of this provide some clues.  My hunch was that there is a non-ascii character which I am cleaning up, or maybe a dash or underscore repeating many many times for that particular example but if my eyes are not failing me, I can't find that in the dataset for that batch.

Thanks you all... 

<|||||>Hi, I'm looking into this problem right now, thank you for providing so much helpful information!
I could set up an experiment where I would get the same error, and I patched it with a cast as @paulthemagno recommended.

It is visible in 1ebfeb7. This should hopefully patch your issue, but as I don't have your particular dataset I can't verify first hand. Do you mind letting me know if it fixes it?<|||||>> Hi, I'm looking into this problem right now, thank you for providing so much helpful information!
> I could set up an experiment where I would get the same error, and I patched it with a cast as @paulthemagno recommended.
> 
> It is visible in [9c67196](https://github.com/huggingface/transformers/commit/9c67196b83a824df577742d32d38e9121d8a9285). This should hopefully patch your issue, but as I don't have your particular dataset I can't verify first hand. Do you mind letting me know if it fixes it?


Thanks, that patch or code block does not reflect the change, maybe a typo on the commit hash? <|||||>Indeed, sorry, edited.<|||||>@LysandreJik happy to confirm that it worked. I patched my own script with the added line for casting the inputs and it ran through the whole corpus 160K records and outputted 5.6 perplexity score. I am assuming it worked.  Thank you very much... 

Ozan<|||||>Fantastic! Thank you for checking!<|||||>> Fantastic! Thank you for checking!

You're welcome.  I am glad I could help.  By the way, out of topic, could you shed some light on why my input tensors are truncated to length = 51 as you can see in my original post above. I don't see where I set that to 51 nor a hard code somewhere. Here are my script arguments:

```
python run_lm_finetuning.py \
 --train_data_file /path/to/data \
  --eval_data_file /path/to/eval_file \
   --output_dir /path/fine_tuned/bert_uncased_lm \
    --mlm \
    --do_train \
     --do_eval \
     --cache_dir  /cache_dir \
     --model_type bert \
     --model_name_or_path bert-base-uncased \
      --per_gpu_train_batch_size 16 \
      --gradient_accumulation_steps 2 \
      --per_gpu_eval_batch_size 16 \
      --block_size 256 \
      --eval_all_checkpoints \
       --line_by_line \
       --fp16
```
As far as I understand Block size is the after tokenization, max seq length is that, where is 51 coming from? This might be a stupid question but I am just trying to avoid making a gross error and get a little bit more of an understanding of the code. 

Ozan
<|||||>That seems weird, indeed, but it's hard for me to debug without having more information about your dataset. Since you're using the `--line_by_line` flag, it should be building tensors according to the line returns in your dataset. Is it possible 51 is the maximum length of a sequence for that specific batch, so it pads up to 51 for the rest of the batch?<|||||>Yes, that must be it, I checked some random batches and the length for the input tensors varies from batch to batch.  I apologize for sidetracking this thread. Seemed like while I had you and the data above, I would get a quick answer. thank you again. <|||||>No worries, glad I could help.<|||||>> Hi, I'm looking into this problem right now, thank you for providing so much helpful information!
> I could set up an experiment where I would get the same error, and I patched it with a cast as @paulthemagno recommended.
> 
> It is visible in [1ebfeb7](https://github.com/huggingface/transformers/commit/1ebfeb79469d544a2bd817aa32c77e0514485ff9). This should hopefully patch your issue, but as I don't have your particular dataset I can't verify first hand. Do you mind letting me know if it fixes it?

Thanks to all. I had already launched the code before you wrote this message, with the additional line `inputs = inputs.type(dtype=torch.long)` without the _clone_ method. It has worked, but I think it is better to restart from 0. Also beacuse re-launching the code from the last saved checkpoint (before the crash), I have noticed that the first new checkpoint had a peek on the perplexity and after that it return to decrease, so better restarting.

Anyway the code worked with my change, so I think also with yours, which is more correct :)