Support for Tensorflow (& or Keras)
## 🚀 Feature

pytorch-transformers the best NLP processing library based on the transformer model. However it has only extensive support for PyTorch, just like it's name suggests. It would be really helpful for the entire Machine Learning community to use it in their legacy project which might have been written in Tensorflow and transitioning to PyTorch is either not feasible or company policy.

> NOTE: I'm speaking on behalf of [pytorch-transfomers]() fans who have this same challenge.

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

## Motivation

I work in a company that has being using TensorFlow since its inception and has extensive codebase written in TensorFlow, however, it is not feasible to rewrite or utilize PyTorch in our system. And this isn't just peculiar to my company, I believe many other Machine Learning engineers face this issue as well.

That being said, It would be nice if your API could use some TensorFlow operations and pre-trained model that could utilize [TF-Hub](https://www.tensorflow.org/hub/) at the very least.

Adopting too many toolchain (e.g PyTorch, TensorFlow, Keras, MXNet, etc) isn't something that large codebase does (for easy maintainability amongst teams and whatnot).

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

## Additional context

Saving tensorflow checkpoints isn't just enough, it would be really helpful if you could either add-on the stable [Tensorflow r1.14](https://www.tensorflow.org/api_docs/python/tf) or [TensorFlow 2.0](https://www.tensorflow.org/beta/) beta version.

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">I love pytorch-transformer 🤗 Great job <a href="https://twitter.com/huggingface?ref_src=twsrc%5Etfw">@huggingface</a> <br>Could you maybe support <a href="https://twitter.com/TensorFlow?ref_src=twsrc%5Etfw">@TensorFlow</a> too?</p>&mdash; Victor I. Afolabi (@victor_iyi) <a href="https://twitter.com/victor_iyi/status/1162456581381992452?ref_src=twsrc%5Etfw">August 16, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
Probably not what you want to hear but you should probably look into rebuilding your infrastructure to also allow pytorch models. As someone who also uses tensorflow due to legacy systems, I wouldn't want the huggingface team to waste time struggling with tensorflow idiocracies and the currently in-flux API. <|||||>This was merely a suggestion for the TensorFlow communities as well. Not just the PyTorch community. Plus it's really hard work (next to impossible) to convert thousands of lines of TensorFlow code to PyTorch, in any case.<|||||>Sorry, I didn't mean what I said as an attack on you or anyone using tf. My intention was to present a counterpoint. I do think this is a valid suggestion, even though I disagree with it.<|||||>Gotcha! No hard feelings, so I guess it's not going to be accepted?<|||||>It might.  According to this issue, it seems that 50% are for and 50% are against.<|||||>Hey guys, 

We (mostly @thomwolf) have done some preliminary research into potential (partial) support for TF2, but so far we haven't committed to any specific implementation or timeframe.

Feel free to keep the discussion going in this issue, it's insightful. Thanks!<|||||>Yes, thanks for that. I think it'll go a long way. Not just with me, but other `tensorflow` & `keras` users. _Especially those that aren't really an expert in it._

On the other hand, maybe if it were to be possible _(**DISCLAIMER:** I'm not positive if it has already been implemented and shipped)_, to provide an intuitive API that saves any `pytorch-transformers` models into `tensorflow`'s checkpoints (`chkpt`) _or_ protocol  buffers (`pb`) and _(or)_ `keras`'s HDF5 (`h5`) files. So it can be loaded by the `tf.estimator.Estimator` API or `keras.Model` easily.

I apologize if what I said doesn't make much sense to the developers with years of exporting & importing `tensorflow` & `keras` models. But I think the goal of `pytorch-transformers` is to make life easier for everyone! 😃 

> Suggestion: The work flow could be something like implementing a simple script with the fantastic `pytorch-transformers` API, then either _exporting the trained model_ or _exporting the model architecture to be loaded as a `tf` or `keras` model_, which has been their entire codebase from inception.<|||||>> Just a suggestion

Also, it might be a lot of work switching between frameworks. So I suggest, it's best to either set a backend (`tensorflow`, or `keras`) while working with `pytorch-transformers`, without any change in `pytorch-transformers`'s API.

Although it might be difficult to add-on, but I think this will help individuals and companies that have used `tensorflow` and `keras` their entire "career" and aren't all that willing to integrate `pytorch` into their system. Not because it's not great, but because of their "design decisions" and "company rules" won't allow it.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
