For people interested in tools for logging and comparing different models and training runs in general, Weights & Biases 159 is directly integrated with  Transformers.
You just need to have wandb installed and logged in.
It automatically logs losses, metrics, learning rate, computer ressources, etc.
image1176×718 56.5 KB
Here is another cool example 91 where I ran a sweep to fine-tune GPT-2 on tweets.
image1281×957 678 KB
Finally you can use your runs to create cool reports. See for example my huggingtweets report 52.
See documentation 159 for more details or this colab 123.
At the moment it is integrated with Trainer and TFTrainer.
If you use Pytorch Lightning, you can use WandbLogger. See Pytorch Lightning documentation 25.
Let me know if you have any questions or ideas to make it better!