Fix `TFGroupViT` CI
# What does this PR do?

Fix 3 `TFGroupViT` CI failures.

There is a remaining one `FAILED tests/models/groupvit/test_modeling_tf_groupvit.py::TFGroupViTTextModelTest ::test_saved_model_creation_extended` which I think @Rocketknight1 will know better. 
_The documentation is not available anymore as the PR was closed or merged._<|||||>Hi @Rocketknight1 

There is an test failure `FAILED tests/models/groupvit/test_modeling_tf_groupvit.py::TFGroupViTTextModelTest ::test_saved_model_creation_extended`, see the error below. It happens at

```python
model = tf.keras.models.load_model(saved_model_dir)
outputs = model(class_inputs_dict)
```

It seems we provide arguments in `tf.int32`, but the loaded model expects `tf.int64`. I think you know much better this part.
Could you take a look - maybe open another PR if necessary? Thank you.

**Full Error**
```
2022-10-09T08:01:03.3261432Z __________ TFGroupViTTextModelTest.test_saved_model_creation_extended __________
2022-10-09T08:01:03.3261667Z 
2022-10-09T08:01:03.3261940Z self = <tests.models.groupvit.test_modeling_tf_groupvit.TFGroupViTTextModelTest testMethod=test_saved_model_creation_extended>
2022-10-09T08:01:03.3262235Z 
2022-10-09T08:01:03.3262337Z     @slow
2022-10-09T08:01:03.3262575Z     def test_saved_model_creation_extended(self):
2022-10-09T08:01:03.3262979Z         config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
2022-10-09T08:01:03.3263352Z         config.output_hidden_states = True
2022-10-09T08:01:03.3263664Z         config.output_attentions = True
2022-10-09T08:01:03.3263941Z     
2022-10-09T08:01:03.3264165Z         if hasattr(config, "use_cache"):
2022-10-09T08:01:03.3264499Z             config.use_cache = True
2022-10-09T08:01:03.3264758Z     
2022-10-09T08:01:03.3264989Z         for model_class in self.all_model_classes:
2022-10-09T08:01:03.3265379Z             class_inputs_dict = self._prepare_for_class(inputs_dict, model_class)
2022-10-09T08:01:03.3392097Z             model = model_class(config)
2022-10-09T08:01:03.3392497Z             num_out = len(model(class_inputs_dict))
2022-10-09T08:01:03.3392721Z     
2022-10-09T08:01:03.3392989Z             with tempfile.TemporaryDirectory() as tmpdirname:
2022-10-09T08:01:03.3393340Z                 model.save_pretrained(tmpdirname, saved_model=True)
2022-10-09T08:01:03.3393690Z                 saved_model_dir = os.path.join(tmpdirname, "saved_model", "1")
2022-10-09T08:01:03.3394008Z                 model = tf.keras.models.load_model(saved_model_dir)
2022-10-09T08:01:03.3394308Z >               outputs = model(class_inputs_dict)
2022-10-09T08:01:03.3394467Z 
2022-10-09T08:01:03.3394617Z tests/models/groupvit/test_modeling_tf_groupvit.py:480: 
2022-10-09T08:01:03.3394911Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2022-10-09T08:01:03.3395433Z /usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py:70: in error_handler
2022-10-09T08:01:03.3395756Z     raise e.with_traceback(filtered_tb) from None
2022-10-09T08:01:03.3396038Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2022-10-09T08:01:03.3396187Z 
2022-10-09T08:01:03.3396605Z args = ({'attention_mask': <tf.Tensor 'input_ids:0' shape=(12, 7) dtype=int32>, 'input_ids': <tf.Tensor 'input_ids_1:0' shape=(12, 7) dtype=int32>}, None, None, None, None, None, ...)
2022-10-09T08:01:03.3396964Z kwargs = {}
2022-10-09T08:01:03.3397499Z inputs = (({'attention_mask': <tf.Tensor 'input_ids:0' shape=(12, 7) dtype=int32>, 'input_ids': <tf.Tensor 'input_ids_1:0' shape=(12, 7) dtype=int32>}, None, None, None, None, None, ...), {})
2022-10-09T08:01:03.3397859Z allow_conversion = True
2022-10-09T08:01:03.3398483Z function_name = '__inference_tf_group_vi_t_text_model_36_layer_call_and_return_conditional_losses_370446'
2022-10-09T08:01:03.3399135Z function = <ConcreteFunction tf_group_vi_t_text_model_36_layer_call_and_return_conditional_losses(input_ids, attention_mask=None, position_ids=None, output_attentions=None, output_hidden_states=None, return_dict=None, training=True) at 0x7F8CC81D4D90>
2022-10-09T08:01:03.3400102Z signature_descriptions = ["Option 1:\n  Positional arguments (7 total):\n    * {'attention_mask': TensorSpec(shape=(None, None), dtype=tf.int64...put_ids/input_ids')}\n    * None\n    * None\n    * None\n    * None\n    * None\n    * True\n  Keyword arguments: {}"]
2022-10-09T08:01:03.3400797Z _pretty_format_positional = <function recreate_function.<locals>.restored_function_body.<locals>._pretty_format_positional at 0x7f8d003cb700>
2022-10-09T08:01:03.3401116Z index = 3
2022-10-09T08:01:03.3401601Z concrete_function = <ConcreteFunction tf_group_vi_t_text_model_36_layer_call_and_return_conditional_losses(input_ids, attention_mask=None, position_ids=None, output_attentions=None, output_hidden_states=None, return_dict=None, training=True) at 0x7F8CC81D4D90>
2022-10-09T08:01:03.3402494Z positional = ({'attention_mask': TensorSpec(shape=(None, None), dtype=tf.int64, name='attention_mask'), 'input_ids': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/input_ids')}, None, None, None, None, None, ...)
2022-10-09T08:01:03.3402916Z keyword = {}
2022-10-09T08:01:03.3403035Z 
2022-10-09T08:01:03.3403161Z     def restored_function_body(*args, **kwargs):
2022-10-09T08:01:03.3403636Z       """Calls a restored function or raises an error if no matching function."""
2022-10-09T08:01:03.3403949Z       if not saved_function.concrete_functions:
2022-10-09T08:01:03.3404275Z         raise ValueError("Found zero restored functions for caller function.")
2022-10-09T08:01:03.3404642Z       # This is the format of function.graph.structured_input_signature. At this
2022-10-09T08:01:03.3405012Z       # point, the args and kwargs have already been canonicalized.
2022-10-09T08:01:03.3405274Z       inputs = (args, kwargs)
2022-10-09T08:01:03.3405478Z     
2022-10-09T08:01:03.3405739Z       # First try to find a concrete function that can be called without input
2022-10-09T08:01:03.3406101Z       # conversions. This allows one to pick a more specific trace in case there
2022-10-09T08:01:03.3406417Z       # was also a more expensive one that supported tensors.
2022-10-09T08:01:03.3406708Z       for allow_conversion in [False, True]:
2022-10-09T08:01:03.3407008Z         for function_name in saved_function.concrete_functions:
2022-10-09T08:01:03.3407321Z           function = concrete_functions[function_name]
2022-10-09T08:01:03.3407622Z           if any([inp is None for inp in function.captured_inputs]):
2022-10-09T08:01:03.3407955Z             raise ValueError("Looks like you are trying to run a loaded "
2022-10-09T08:01:03.3408359Z                              "non-Keras model that was trained using "
2022-10-09T08:01:03.3408774Z                              "tf.distribute.experimental.ParameterServerStrategy "
2022-10-09T08:01:03.3409160Z                              "with variable partitioning, which is not currently "
2022-10-09T08:01:03.3409664Z                              "supported. Try using Keras to define your model "
2022-10-09T08:01:03.3409965Z                              "if possible.")
2022-10-09T08:01:03.3410294Z           if _concrete_function_callable_with(function, inputs, allow_conversion):
2022-10-09T08:01:03.3410757Z             return _call_concrete_function(function, inputs)
2022-10-09T08:01:03.3410980Z     
2022-10-09T08:01:03.3411193Z       signature_descriptions = []
2022-10-09T08:01:03.3411414Z     
2022-10-09T08:01:03.3411651Z       def _pretty_format_positional(positional):
2022-10-09T08:01:03.3411939Z         return "Positional arguments ({} total):\n    * {}".format(
2022-10-09T08:01:03.3412206Z             len(positional),
2022-10-09T08:01:03.3412486Z             "\n    * ".join(pprint.pformat(a) for a in positional))
2022-10-09T08:01:03.3412735Z     
2022-10-09T08:01:03.3413063Z       for index, function_name in enumerate(saved_function.concrete_functions):
2022-10-09T08:01:03.3413387Z         concrete_function = concrete_functions[function_name]
2022-10-09T08:01:03.3413721Z         positional, keyword = concrete_function.structured_input_signature
2022-10-09T08:01:03.3414020Z         signature_descriptions.append(
2022-10-09T08:01:03.3414290Z             "Option {}:\n  {}\n  Keyword arguments: {}".format(
2022-10-09T08:01:03.3414622Z                 index + 1, _pretty_format_positional(positional), keyword))
2022-10-09T08:01:03.3414929Z >     raise ValueError(
2022-10-09T08:01:03.3415383Z           "Could not find matching concrete function to call loaded from the "
2022-10-09T08:01:03.3415807Z           f"SavedModel. Got:\n  {_pretty_format_positional(args)}\n  Keyword "
2022-10-09T08:01:03.3416205Z           f"arguments: {kwargs}\n\n Expected these arguments to match one of the "
2022-10-09T08:01:03.3416594Z           f"following {len(saved_function.concrete_functions)} option(s):\n\n"
2022-10-09T08:01:03.3416955Z           f"{(chr(10)+chr(10)).join(signature_descriptions)}")
2022-10-09T08:01:03.3417393Z E     ValueError: Exception encountered when calling layer "tf_group_vi_t_text_model_36" "                 f"(type TFGroupViTTextModel).
2022-10-09T08:01:03.3417731Z E     
2022-10-09T08:01:03.3418041Z E     Could not find matching concrete function to call loaded from the SavedModel. Got:
2022-10-09T08:01:03.3418385Z E       Positional arguments (7 total):
2022-10-09T08:01:03.3418885Z E         * {'attention_mask': <tf.Tensor 'input_ids:0' shape=(12, 7) dtype=int32>,
2022-10-09T08:01:03.3419334Z E      'input_ids': <tf.Tensor 'input_ids_1:0' shape=(12, 7) dtype=int32>}
2022-10-09T08:01:03.3419619Z E         * None
2022-10-09T08:01:03.3419837Z E         * None
2022-10-09T08:01:03.3420048Z E         * None
2022-10-09T08:01:03.3420245Z E         * None
2022-10-09T08:01:03.3420455Z E         * None
2022-10-09T08:01:03.3420668Z E         * False
2022-10-09T08:01:03.3420890Z E       Keyword arguments: {}
2022-10-09T08:01:03.3421125Z E     
2022-10-09T08:01:03.3421420Z E      Expected these arguments to match one of the following 4 option(s):
2022-10-09T08:01:03.3421707Z E     
2022-10-09T08:01:03.3421894Z E     Option 1:
2022-10-09T08:01:03.3422147Z E       Positional arguments (7 total):
2022-10-09T08:01:03.3422648Z E         * {'attention_mask': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/attention_mask'),
2022-10-09T08:01:03.3423198Z E      'input_ids': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/input_ids')}
2022-10-09T08:01:03.3423494Z E         * None
2022-10-09T08:01:03.3423709Z E         * None
2022-10-09T08:01:03.3423922Z E         * None
2022-10-09T08:01:03.3424136Z E         * None
2022-10-09T08:01:03.3424405Z E         * None
2022-10-09T08:01:03.3424613Z E         * False
2022-10-09T08:01:03.3424850Z E       Keyword arguments: {}
2022-10-09T08:01:03.3425065Z E     
2022-10-09T08:01:03.3425242Z E     Option 2:
2022-10-09T08:01:03.3425478Z E       Positional arguments (7 total):
2022-10-09T08:01:03.3425975Z E         * {'attention_mask': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/attention_mask'),
2022-10-09T08:01:03.3426635Z E      'input_ids': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/input_ids')}
2022-10-09T08:01:03.3426902Z E         * None
2022-10-09T08:01:03.3427098Z E         * None
2022-10-09T08:01:03.3427397Z E         * None
2022-10-09T08:01:03.3427576Z E         * None
2022-10-09T08:01:03.3427742Z E         * None
2022-10-09T08:01:03.3427922Z E         * True
2022-10-09T08:01:03.3428119Z E       Keyword arguments: {}
2022-10-09T08:01:03.3428303Z E     
2022-10-09T08:01:03.3428476Z E     Option 3:
2022-10-09T08:01:03.3428729Z E       Positional arguments (7 total):
2022-10-09T08:01:03.3429129Z E         * {'attention_mask': TensorSpec(shape=(None, None), dtype=tf.int64, name='attention_mask'),
2022-10-09T08:01:03.3429670Z E      'input_ids': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/input_ids')}
2022-10-09T08:01:03.3429926Z E         * None
2022-10-09T08:01:03.3430109Z E         * None
2022-10-09T08:01:03.3430288Z E         * None
2022-10-09T08:01:03.3430450Z E         * None
2022-10-09T08:01:03.3430633Z E         * None
2022-10-09T08:01:03.3430814Z E         * False
2022-10-09T08:01:03.3431017Z E       Keyword arguments: {}
2022-10-09T08:01:03.3431193Z E     
2022-10-09T08:01:03.3431371Z E     Option 4:
2022-10-09T08:01:03.3431585Z E       Positional arguments (7 total):
2022-10-09T08:01:03.3431991Z E         * {'attention_mask': TensorSpec(shape=(None, None), dtype=tf.int64, name='attention_mask'),
2022-10-09T08:01:03.3432620Z E      'input_ids': TensorSpec(shape=(None, None), dtype=tf.int64, name='input_ids/input_ids')}
2022-10-09T08:01:03.3432891Z E         * None
2022-10-09T08:01:03.3433073Z E         * None
2022-10-09T08:01:03.3433259Z E         * None
2022-10-09T08:01:03.3433417Z E         * None
2022-10-09T08:01:03.3433589Z E         * None
2022-10-09T08:01:03.3433769Z E         * True
2022-10-09T08:01:03.3433955Z E       Keyword arguments: {}
2022-10-09T08:01:03.3434128Z E     
2022-10-09T08:01:03.3434408Z E     Call arguments received by layer "tf_group_vi_t_text_model_36" "                 f"(type TFGroupViTTextModel):
2022-10-09T08:01:03.3434960Z E       • args=({'input_ids': 'tf.Tensor(shape=(12, 7), dtype=int32)', 'attention_mask': 'tf.Tensor(shape=(12, 7), dtype=int32)'},)
2022-10-09T08:01:03.3435308Z E       • kwargs={'training': 'False'}
```