Yeah, that’s definitely a bug! Would you mind opening an issue about it on transformers? Here is the link: https://github.com/huggingface/transformers/issues/new/choose -> feel free to ping me (@patrickvonplaten) with a link to this thread here and I’ll take care of it asap. Should be pretty trivial to solve 
Thanks for taking a deeper look into this feature - this should definitely be fully supported by Transformers