Adding fastseq support to more recent version of HF transformers
# 🚀 Feature request

Would it be possible to integrate [fastseq](https://github.com/microsoft/fastseq) with the later version of HuggingFace transformer models.

## Motivation

fastseq is a library that gives speedup on Transformers for text generation. They claim to have pretty large speedups (3-8x) for various Transformer architectures (GPT2, Bart, etc.).
The only caveat is they only support an older version of HF transformers (3.0.2).

Has anyone already looked into making it compatible with the latest API of HuggingFace models?

## Your contribution

I am willing to discuss and can contribute if no one has planned to do so already.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.