Huge bug. TF saved model running in nvidia-docker dose not use GPU.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10
- Platform: Ubutu 18.04
- Python version: 3.6
- PyTorch version (GPU?): latest GPU
- Tensorflow version (GPU?): 2.4 GPU
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@LysandreJik
@Rocketknight1
@sgugger

## Information

Model I am using official 'nateraw/bert-base-uncased-imdb'.

The problem arises when using:
[*] the official example scripts: https://huggingface.co/blog/tf-serving


## To reproduce

Steps to reproduce the behavior:
Follow [official blog](https://huggingface.co/blog/tf-serving), I save pretrained model to saved model, then running model on tensorflow/serving docker (that is CPU docker) and tensorflow/serving:latest-gpu docker (that is GPU docker), then I testing the inference speed:
1. On CPU docker, cost is 0.19s
2. On GPU docker, cost is 2.9s, 10 times slower than CPU docker inference
3. When inferening with GPU docker, GPU-Util is 0 and only one CPU core is working. 
4. I re-test on another machine, got same result.

Tesing machine environment:
#1: GTX 1070
#2: Titan Xp

## Expected behavior

Inference on GPU docker should be much faster than CPU docker.
What's wrong with that?
Have you ever test the offcial blog?
Nobody run saved model with nvidia-docker?

Help me, thx!!!

Code to saved model.

```
from transformers import TFBertForSequenceClassification

model = TFBertForSequenceClassification.from_pretrained("nateraw/bert-base-uncased-imdb", from_pt=True)
# the saved_model parameter is a flag to create a SavedModel version of the model in same time than the h5 weights
model.save_pretrained("my_model", saved_model=True)
```<|||||>Code to do inference.
```
from transformers import BertTokenizerFast, BertConfig
import requests
import json
import numpy as np
import time

sentence = "I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers. I love the new TensorFlow update in transformers."

# Load the corresponding tokenizer of our SavedModel
tokenizer = BertTokenizerFast.from_pretrained("nateraw/bert-base-uncased-imdb")

# Load the model config of our SavedModel
config = BertConfig.from_pretrained("nateraw/bert-base-uncased-imdb")

# Tokenize the sentence
batch = tokenizer(sentence)

# Convert the batch into a proper dict
batch = dict(batch)

# Put the example into a list of size 1, that corresponds to the batch size
batch = [batch]

# The REST API needs a JSON that contains the key instances to declare the examples to process
input_data = {"instances": batch}

t1 = time.time()
# Query the REST API, the path corresponds to http://host:port/model_version/models_root_folder/model_name:method
r = requests.post("http://localhost:8502/v1/models/bert:predict", data=json.dumps(input_data))
print(time.time() - t1)

# Parse the JSON result. The results are contained in a list with a root key called "predictions"
# and as there is only one example, takes the first element of the list
result = json.loads(r.text)["predictions"][0]

# The returned results are probabilities, that can be positive or negative hence we take their absolute value
abs_scores = np.abs(result)

# Take the argmax that correspond to the index of the max probability.
label_id = np.argmax(abs_scores)

# Print the proper LABEL with its index
print(config.id2label[label_id])

```<|||||>Run cpu docker:
`sudo docker run -d -p 8502:8501 --name bert_cpu image_hug_off_cpu`
Run gpu docker:
`sudo docker run -d --gpus '"device=0"' -p 8502:8501 --name bert image_hug_off`<|||||>Hello!  Do you manage to leverage your GPUs using other TensorFlow code than our models? I suspect there is an issue in the setup - we do run some GPU tests so the models should work without issue on GPUs. <|||||>Everything is OK when code runs without signatures, after add signatures, it gets sucked.<|||||>cc @Rocketknight1 <|||||>I'm not sure I understand - can you show an example of a model that runs correctly with the GPU and a model that does not, so we can figure out what the problem is?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.