Best weights/models after fine-tuning gpt2
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I am fine-tuning gpt2 on a new dataset and it checkpoints after every 50 iterations and saves the model.  It stretches my local storage to extreme and would want to delete redundant models. So my queries are following:

1) Is it possible to only save the best weights (which gave the lowest perplexity/loss on evaluation data)?
2) When we run_generation.py and passed the directory of our fine-tuned model, which model weights are actually used for generation? ( as there are so many checkpoint folders with model weights).
3) And hence, related to above 2 questions how does the model consider "the best" model/weights from the fine-tuned model directory we pass as arguments? and can we just mention the chekcpoint folder as well in run_generation.py?

Thanks

Hi! Indeed saving checkpoints after every 50 iterations is quite a lot, therefore we've upped this value to 500 yesterday in 335dd5e. Concerning your questions:

1. You can use the `--evaluate_during_training` flag to evaluate the model every `--logging_step`, or you can use the `--evaluate_all_checkpoints` to evaluate all the checkpoints at the end. There is no feature to save only the best model, but you could easily do it by modifying the script to save only when the evaluation yields better results than the previous one.
2. For the `run_generation.py` script, pass it the folder which contains the weights you want to use for generation. The folder must contain a `pytorch_model.bin`, a `config.json`, as well as a tokenizer object.
3. Yes, just mention the checkpoint folder in `run_generation.py`. <|||||>Thank you. My bad i did not pay full attention to the arguments present in the generation script. That answers my queries.