Bug Transformers 2.3.0 - ValueError: invalid literal for int() with base 10: 'pytorch'
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): PT-BR (Multilingual)

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

CUDA_VISIBLE_DEVICES=2,3 nohup python /home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py \
--output_dir=/home/lucasrodrigues/train/transformers/output/multi/250k/ \
--model_type=bert \
--model_name_or_path=/home/lucasrodrigues/train/transformers/model-multi-pytorch/ \
--tokenizer_name=/home/lucasrodrigues/train/transformers/model-multi-pytorch/ \
--config_name=/home/lucasrodrigues/train/transformers/model-multi-pytorch/ \
--block_size=510 \
--do_lower_case \
--train_data_file=/home/lucasrodrigues/datasets/nilc/initial/initial_corpus_train.txt \
--eval_data_file=/home/lucasrodrigues/datasets/nilc/initial/initial_corpus_train.txt \
--do_train \
--do_eval \
--evaluate_during_training \
--logging_steps=50 \
--save_steps=50 \
--per_gpu_train_batch_size=2 \
--per_gpu_eval_batch_size=2 \
--mlm \
> logs/transformers_multi_250k.txt & 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Traceback (most recent call last):
  File "/home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py", line 713, in <module>
    main()
  File "/home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py", line 663, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "/home/lucasrodrigues/code/transformers-2.3.0/examples/run_lm_finetuning.py", line 268, in train
    global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
ValueError: invalid literal for int() with base 10: 'pytorch'

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 
* Python version: 3.6
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 
* Using GPU ? 4x GeForce GTX 1080 Ti
* Distributed or parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

I can't execute the code, could anyone help?
Hi, thank you for raising this issue. Could you please let me know if 27c1b656cca75efa0cc414d3bf4e6aacf24829de fixed this issue by trying the updated script?<|||||>Hello, to solve this problem I added my checkpoint to a folder that has the same Transformer output.

**new folder -> chekpoint-0**

Folders:
|
chekpoint-0
| vocab.txt
| pytorch_model.bin
| config.json

global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])

**Result:
global_step = 0**<|||||>> Hi, thank you for raising this issue. Could you please let me know if [`27c1b65`](https://github.com/huggingface/transformers/commit/27c1b656cca75efa0cc414d3bf4e6aacf24829de) fixed this issue by trying the updated script?

@LysandreJik, your commit fixed the issue for me, thanks!<|||||>> Hi, thank you for raising this issue. Could you please let me know if [27c1b65](https://github.com/huggingface/transformers/commit/27c1b656cca75efa0cc414d3bf4e6aacf24829de) fixed this issue by trying the updated script?

I think this modification is a terrible one since some people maybe download pytorch-pretrained-models like pytorch-model.bin alone in a dir, but when use this "global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])" as a number for global_step, what does it mean?@LysandreJik  like [this_issue](https://github.com/huggingface/transformers/issues/2258) said. Whoever add "global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])" could you please fix this bug? I remember there are nothing about it before.<|||||>@severinsimmler, I agree with @zysNLP, this introduces a bug when you try to use a lm that wasn't from checkpoint folder. I used `run_langauge_modeling.py` to output a lm, which I then feed into `run_glue.py`. This pipeline no longer works because `run_glue.py` is trying to parse a global step number from a folder that doesn't have one. Renaming my folder to checkpoint-0 and then feeding it into `run_glue.py` shouldn't have to be done, surely `args.model_name_or_path.split("-")[-1].split("/")[0]` can be modified slightly; so that, it returns `0` instead of `""`; so that, models in non-checkpoint folders can be added.<|||||>@stefan-it solution in https://github.com/huggingface/transformers/issues/2258 fixes the issue. This or something similar should be added to all the example scripts.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
