Finetuning my language model
## ❓ Questions & Help

I have a problem about a finetuning of my own language model ([model](https://mxmdownloads.s3.amazonaws.com/umberto/umberto-commoncrawl-cased-v1.tar.gz) and [sentencepiece](https://mxmdownloads.s3.amazonaws.com/umberto/umberto-commoncrawl-cased-v1-sentencepiece.bpe.model)). I'm trying to use [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py).

When it tokenizes the text of the training set, the program enters a loop without giving feedback. 
It blocks at [Line 105](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L105) in the _tokenize_ function. 

```python
            ...
            logger.info("Creating features from dataset file at %s", directory)

            self.examples = []
            with open(file_path, encoding="utf-8") as f:
                text = f.read()

            tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

            for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
                self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
            # Note that we are loosing the last truncated example here for the sake of simplicity (no padding)
            # If your dataset is small, first you should loook for a bigger one :-) and second you
            # can change this behavior by adding (model specific) padding.
            ...
```
I have tried to wait for more than 1 hour, but nothing. Using BERT instead of my model, the finetuning starts (even if after several minutes with no log meanwhile).

I launched this:
```bash
python3 run_lm_finetuning.py \
--train_data_file /path/to/train.txt \
--eval_data_file /path/to/eval.txt \
--output_dir /path/to/output \
--mlm \
--do_train \
--do_eval \
--model_type roberta \
--model_name_or_path /path/to/my/model \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 8 \
--overwrite_output_dir \
--overwrite_cache \
--max_steps 500000 \
--block_size 128\
--save_steps 50000 \
--eval_all_checkpoints
```
My impression is that something is wrong with the tokenization on the model, any ideas?
I am having the same problem when finetuning my own language model with run_lm_finetuning.py on camembert Model. I guess this might be related to the fact that it is reading a big file at once (103M, ~500k lines).
 Since the code reads whole data at once, it requires so much memory to handle huge corpus.

This pull request [2339](https://github.com/huggingface/transformers/pull/2339)  is suggesting:
- read corpus by each lines
- flatten 2-dimension array by itertools.chain, it requies less memory and fast
You can find the code [here](https://github.com/huggingface/transformers/pull/2339/commits/537a1de53d824b5851bce32cb5eafaef3f9ce5ef#diff-713f433a085810c3d63a417486e56a88)

<|||||>> I am having the same problem when finetuning my own language model with run_lm_finetuning.py on camembert Model. I guess this might be related to the fact that it is reading a big file at once (103M, ~500k lines).
> Since the code reads whole data at once, it requires so much memory to handle huge corpus.
> 
> This pull request [2339](https://github.com/huggingface/transformers/pull/2339) is suggesting:
> 
> * read corpus by each lines
> * flatten 2-dimension array by itertools.chain, it requies less memory and fast
>   You can find the code [here](https://github.com/huggingface/transformers/pull/2339/commits/537a1de53d824b5851bce32cb5eafaef3f9ce5ef#diff-713f433a085810c3d63a417486e56a88)

@HendZouari this is interesting.
I thought the same thing, but I'm not sure the reason is the size of data, since I verified that with multlingual BERT, it works. 
<|||||>Can you guys try out the recently-merged-to-master `LineByLineTextDataset`, defined at
https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L124 ?

You can use it by adding the `--line_by_line` flag to your command. (btw: tokenization will soon be _way_ faster everywhere in the library as we are rolling out `tokenizers` integration everywhere)<|||||>For large datasets that don't fit in memory, I use a lazy dataset, modified from [this Github post](https://github.com/pytorch/text/issues/130#issuecomment-333306652).

```python
class LazyTextDataset(Dataset):
    def __init__(self, fin):
        # get absolute path
        # convert to str, linecache doesn't accept Path objects
        self.fin = str(Path(fin).resolve())
        self.num_entries = self._get_n_lines(self.fin)

    @staticmethod
    def _get_n_lines(fin):
        with open(fin, encoding='utf-8') as fhin:
            for line_idx, _ in enumerate(fhin, 1):
                pass

        return line_idx

    def __getitem__(self, idx):
        # linecache starts counting from one, not zero, +1 the given index
        return linecache.getline(self.fin, idx+1)

    def __len__(self):
        return self.num_entries
```

With a bit of work you can modify this to return the tokenized strings. I would advise you to write a custom collate_fn for the dataloader, which can be parallellized by using the `n_workers` argument.

Something like this (untested)

```python
    from torch.utils.data.dataloader import default_collate

    def collate(data):
        data = default_collate(data)
        return tokenizer.encode(data)
```

That being said, it might be easier to just use wait a bit until `tokenizers` is implemented everywhere, as @julien-c mentions.<|||||>@julien-c it's woking now! The flag `--line_by_line` was fundamental for me 🤩.
Thanks also to @HendZouari and @BramVanroy: I guess If I had followed your advices, the program would have worked well in the same way :)<|||||>I am having this issue while using --line_by_line flag
```
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 785, in <module>
    main()
  File "run_lm_finetuning.py", line 730, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "run_lm_finetuning.py", line 147, in load_and_cache_examples
    return LineByLineTextDataset(tokenizer, args, file_path=file_path, block_size=args.block_size)
  File "run_lm_finetuning.py", line 135, in __init__
    self.examples = tokenizer.batch_encode_plus(lines, max_length=block_size)["input_ids"]
AttributeError: 'BertTokenizer' object has no attribute 'batch_encode_plus'
```

Without this flag I run into another loss function related error for which I just opened an issue.<|||||>`AttributeError: 'BertTokenizer' object has no attribute 'batch_encode_plus'`

You need to update `transformers`. `batch_encode_plus` was only introduced recently.<|||||>Yes. Upgraded and now working. Thank you.<|||||>@paulthemagno Can you close this? Thanks <|||||>I have reopened the issue for a stange RuntimeError.
In the middle of the training (after several hours in which it was working with no problem), it crashes with this log:
```
File "finetuning.py", line 801, in <module>51:22,  4.94it/s]
    main()
  File "finetuning.py", line 750, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "finetuning.py", line 342, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "finetuning.py", line 222, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:  55%|█████▍    | 6/11 [20:26:33<17:02:07, 12265.60s/it]
Iteration:  69%|██████▊   | 33378/48603 [1:47:45<49:09,  5.16it/s]
```

It fails in the _mask_tokens()_ function:
```python
# 10% of the time, we replace masked input tokens with random word
    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random]
```

Should I set `dtype=torch.float`? Why does It work fine for so much time and it suddenly gives this error at the 6th epoch on 11? <|||||>I have the same issue, but to me it fails immediately.
```
  File "run_lm_finetuning.py", line 340, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 218, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
```
It is triggered by using `--line_by_line`.<|||||>I have the same issue as @paulthemagno 's. Mine fails while fine-tuning a basic uncased bert model using the new --line_by_line flag about a quarter way through the epoch. 

```Traceback (most recent call last):███████████████████▋                                                                                                     | 2184/8205 [06:29<18:54,  5.31it/s]

  File "run_lm_finetuning.py", line 692, in <module>
    main()
  File "run_lm_finetuning.py", line 641, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 320, in train
    inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
  File "run_lm_finetuning.py", line 193, in mask_tokens
    inputs[indices_random] = random_words[indices_random]
RuntimeError: expected dtype Float but got dtype Long
Epoch:   0%|                                                                                                                                                            | 0/1 [06:29<?, ?it/s]
Iteration:  27%|████████████████████████████████████▋                                                                                                      

2184/8205 [06:29<17:54,  5.61it/s]```<|||||>I tried to set `dtype=torch.float` but it fails immediately after the launch.
The only way I found is to restart from the last saved checkpoint rather than from the original language model. If someone knew how to fix it, I'd appreciate it.<|||||>Can you guys open a new issue for this? (w/ PyTorch version + ideally a small reproduction case)<|||||>> Can you guys open a new issue for this? (w/ PyTorch version + ideally a small reproduction case)

Yes, done now #2728