Hello! Essentially what I want to do is: point the code at a .txt file, and get a trained model out. How can I use run_mlm.py to do this? I’d be satisfied if someone could help me figure out how to even just recreate the EsperBERTo tutorial.
I’m getting bogged down in flags, trying to load tokenizers, errors, etc.
What I’ve done so far: I managed to run through the EsperBERTo tutorial (here 20), and now I’m trying to do the same thing with run_mlm.py.
I went to the examples at transformers/examples/pytorch/language-modeling at master · huggingface/transformers · GitHub 26 first, and I’ve been attempting to adapt those.

First of all, the section at transformers/examples/pytorch/language-modeling at master · huggingface/transformers · GitHub 19 seems the closest to EsperBERTo. So I started with the example for “your own training and validation files”
I wanted to train from scratch, so I edited it to use --model_type

I wanted to just use the oscar.eo.txt, so I omitted --validation_file and --do_eval

I set --max_seq_length 512 to match the tutorial.

I keep running into problems with the tokenizer.


run_mlm.py doesn’t let you not specify a tokenizer when you are training from scratch, apparently.
if you give it --model_type roberta and --tokenizer_name <path to the vocab.json and mergest.txt> it complains about not being able to find a config.json
if you then put a config.json for the Model, not the tokenizer in that folder, the error goes away. But why is it looking in the tokenizer folder for model config? And why does it need a model config at all? I told it model type?

What finally I did was:

made a folder called EsperBERTo. In there I put the vocab.json and merges.txt from the Colab tutorial, and a config.json that I found for RobertaForMaskedLM here 9

ran the following command:

python run_mlm.py \
    --model_type roberta \
    --tokenizer_name /path/to/EsperBERTo/ \
    --train_file /path/to/oscar.eo.txt \
    --max_seq_length 512 \
    --do_train \
    --output_dir ./output/test-mlm

But now I’m getting “IndexError: index out of range in self”. Some Googling lead me to believe this might be to do with the vocab size? I edited config.json to match the tokenizer (52000 vocab size), but no dice.
Here’s config.json
{
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 207,
  "model_type": "roberta",
  "num_attention_heads": 6,
  "num_hidden_layers": 3,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 52000
}

At this point I’m thoroughly confused as to how to proceed, and I’m not even sure I’m going down the right path.
Can anyone give guidance on the proper process to go from .txt file to trained model in the command line? I’m not having much success transferring my knowledge (and my trained tokenizer) from Colab.