Need clarity on training Albert from scratch
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->
Using transformers 2.9.1
I am following EsperBERTo tutorial for Training Albert LM from scratch. I am able to start training for roberta but for Albert getting tokenizer issue:
```
05/15/2020 16:23:29 - INFO - transformers.training_args -   PyTorch: setting up devices
05/15/2020 16:23:29 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: False
05/15/2020 16:23:29 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='./EsperBERTo-small-v1', overwrite_output_dir=False, do_train=True, do_eval=False, do_predict=False, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=0.0001, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=2000, save_total_limit=2, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False)
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   loading configuration file ./EsperBERTo/config.json
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   Model config AlbertConfig {
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 52000
}

05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   loading configuration file ./EsperBERTo/config.json
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   Model config AlbertConfig {
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 52000
}

05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Model name './EsperBERTo' not found in model shortcut name list (albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2). Assuming './EsperBERTo' is a path, a model identifier, or url to a directory containing tokenizer files.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/spiece.model. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/added_tokens.json. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/special_tokens_map.json. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file ./EsperBERTo/tokenizer_config.json
Traceback (most recent call last):
  File "/home/gamut/Downloads/transformers_source/examples/language-modeling/run_language_modeling.py", line 289, in <module>
    main()
  File "/home/gamut/Downloads/transformers_source/examples/language-modeling/run_language_modeling.py", line 188, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 203, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 902, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 1055, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_albert.py", line 155, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/sentencepiece.py", line 275, in _sentencepiece_processor_load
    return self._Load_native(model_file)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/sentencepiece.py", line 75, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
TypeError: not a string
CPU times: user 13.6 ms, sys: 27.4 ms, total: 41.1 ms
Wall time: 1.12 s
```
I guess it was looking for spiece model. So I added spice.model in EsperBERTo folder and it started training.
Now I confuse as vocab it is using now is generated using Huggingface tokenizer and taking spice.model also... can any one clarify whats going on here? 

The Albert original model has a SentencePiece tokenizer, so the `AlbertTokenizer` can only handle SentencePiece vocabularies right now. The `tokenizers` library doesn't handle these vocabularies as of now.

We're still thinking of how we should proceed when training a model from scratch with a tokenizer different to the original, as right now the model-tokenizer pairs have a fixed tokenizer backend. cc @julien-c @thomwolf <|||||>@LysandreJik But  when I added spiece.model it started training. So is it taking vocab created by tokenizer library along with spiece.model?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Does this issue have been solved? When can we train the lm from scratch?<|||||>You can already train an LM from scratch, there are examples in `examples/language-modeling`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Are there any updates on training Albert from scratch? Perhaps via the Trainer?<|||||>Hi, a `DataCollatorForSOP` has been added to the library, which you can indeed use with the `Trainer` object. You can see it's implementation here:

https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/data_collator.py#L201-L206<|||||>> Hi, a `DataCollatorForSOP` has been added to the library, which you can indeed use with the `Trainer` object. You can see it's implementation here:
> 
> https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/data_collator.py#L201-L206

When i try to use this `DataCollatorForSOP` it says that it is deprecated and that i should use `DataCollatorForLanguageModeling`. It is however not clear whether this data collator also produces the objective sentence order prediction as in the original ALBERT paper. Can anyone verify whether it does this or not?