run_language_modeling.py line 251: checking if it is a directory
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi, I'm curious why you changed the code to check if model_args.model_name_or_path is a directory. I am trying to train OpenAI GPT in Google Colab, and using arguments as following:

!python /content/transformers/examples/run_language_modeling.py
--output_dir="/content/gdrive/My Drive/output"
--overwrite_output_dir
--model_type=openai-gpt
--model_name_or_path=openai-gpt
--do_train
--train_data_file="/content/gdrive/My Drive/train.txt"
--do_eval
--eval_data_file="/content/gdrive/My Drive/dev.txt"
--num_train_epochs 5
--learning_rate 1.5e-4
--save_steps 1
--save_total_limit 5

but this line causes following error:

Traceback (most recent call last):
File "/content/transformers/examples/run_language_modeling.py", line 283, in
main()
File "/content/transformers/examples/run_language_modeling.py", line 257, in main
trainer.train(model_path=model_path)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 363, in train
self._rotate_checkpoints()
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 458, in _rotate_checkpoints
checkpoints_sorted = self._sorted_checkpoints(use_mtime=use_mtime)
File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 443, in _sorted_checkpoints
regex_match = re.match(".*{}-([0-9]+)".format(checkpoint_prefix), path)
File "/usr/lib/python3.6/re.py", line 172, in match
return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

It seems that openai-gpt is not a directory, thus making model_path variable equal to None. Is this due to my implementation or can this part be fixed?


<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
I also faced the similler issue while the arguments 
--save_steps 200
--save_total_limit 3 
Removing this arguments worked. But need to know what could be the issue<|||||>Did removing them worked through the end completely? Because there is a default value for save_steps, and I just set it to 1 to see if the bug still appears after some manual fixes... Looking at the function names like rotate_checkpoints(), I feel like save_total_limit might be the key, since its default is set to None, but again not sure....<|||||>Yes Removing them worked till the end for me. Total Optimization steps for me was around 1600. so it did save 3 times the checkpoints and correctly loaded them as well. <|||||>Ah okay removing the --save_total_limit make it work for me, but then it would start saving gigabytes of models without top limit. I think the directory checking might come from removing the should_continue argument, but will require us to constantly update where we load the latest model from...?<|||||>I can confirm this is a bug. I'll push a fix on master shortly.