sigh, it’s the third time that I have had to change my scripts again. Why is that with HF half the time you have to change your entire code? The Trainer API was a gamechanger, until it doesn’t work with some models 
Why do we need to actually use the Flax version of BigBird to run it on TPUs?
I am not asking you give me a 2-line code solution, but the least HuggingFace could strive to do is to not be sooo fragmented as it is now. At this point, Transformers has become so bloated with different tasks, changing APIs, outdated notebooks that its a nightmare (I honestly can’t even comphrend how Open source devs even contribute here).
Honestly, it would rather be easier if it be married to a single framework like pytorch - allowing simple DataLoaders to interface directly with Huggingface Models and be trained easily in a pytorch-style manner with XLA spawn. I think Huggingface tries to take control of too many parts of the expereince - Datasets for making datasets, and it doesn’t always mesh well.
Or perhaps I am just too dumb, and probably unfair to expect pytorch like simplicity from such a huge lib.