Unable to load UnifiedQA models, tf throws DataLossError
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0a0+b31f58d (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
Probably 
 T5: @patrickvonplaten
 tensorflow: @jplu


## Information

Model I am using UnifiedQA (based on T5):

The problem arises when Using the model loading code provided in the [UnifiedQA Readme](https://github.com/allenai/unifiedqa#using-the-models-in-pytorchhuggingface), shown below. Unable to load models throws DataLossError.


Code:
```
from transformers import T5Config, T5Tokenizer, T5ForConditionalGeneration
from transformers.modeling_t5 import load_tf_weights_in_t5

base_model = "t5-small"
tokenizer = T5Tokenizer.from_pretrained(base_model)
model = T5ForConditionalGeneration(T5Config.from_pretrained(base_model))

load_tf_weights_in_t5(model, None, "./models/unifiedqa-small/")
```  

Error:
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/py_checkpoint_reader.py in NewCheckpointReader(filepattern)
     94   try:
---> 95     return CheckpointReader(compat.as_bytes(filepattern))
     96   # TODO(b/143319754): Remove the RuntimeError casting logic once we resolve the

RuntimeError: Unable to open table file /content/base: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

During handling of the above exception, another exception occurred:

DataLossError                             Traceback (most recent call last)
5 frames
<ipython-input-27-b28dfb350abf> in <module>()
      7 
      8 model_path = './base/' #@param ['./unifiedqa-base/', './base/']
----> 9 load_tf_weights_in_t5(model, None, model_path)
     10 
     11 # tokenizer = T5Tokenizer.from_pretrained('t5-base')

/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py in load_tf_weights_in_t5(model, config, tf_checkpoint_path)
     78     logger.info("Converting TensorFlow checkpoint from {}".format(tf_path))
     79     # Load weights from TF model
---> 80     init_vars = tf.train.list_variables(tf_path)
     81     names = []
     82     tf_weights = {}

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpoint_utils.py in list_variables(ckpt_dir_or_file)
     96     List of tuples `(name, shape)`.
     97   """
---> 98   reader = load_checkpoint(ckpt_dir_or_file)
     99   variable_map = reader.get_variable_to_shape_map()
    100   names = sorted(variable_map.keys())

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpoint_utils.py in load_checkpoint(ckpt_dir_or_file)
     65     raise ValueError("Couldn't find 'checkpoint' file or checkpoints in "
     66                      "given directory %s" % ckpt_dir_or_file)
---> 67   return py_checkpoint_reader.NewCheckpointReader(filename)
     68 
     69 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/py_checkpoint_reader.py in NewCheckpointReader(filepattern)
     97   # issue with throwing python exceptions from C++.
     98   except RuntimeError as e:
---> 99     error_translator(e)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/py_checkpoint_reader.py in error_translator(e)
     42     raise errors_impl.InvalidArgumentError(None, None, error_message)
     43   elif 'Unable to open table file' in error_message:
---> 44     raise errors_impl.DataLossError(None, None, error_message)
     45   elif 'Failed to find the saved tensor slices' in error_message:
     46     raise errors_impl.InternalError(None, None, error_message)
DataLossError: Unable to open table file /path/to/models/unifiedqa-small: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
```

Perhaps might be an issue in the code for loading from tensorflow, [related issue](https://github.com/tensorflow/models/issues/2676).
Also, put an [issue](https://github.com/allenai/unifiedqa/issues/6) on the unifiedQA repo.
As far as I can see, the model you are trying load doesn't have a compliant format. We cannot help more without the full error stack.<|||||>Edit, added the full error stack.
Also, this is fairly easy to reproduce.