[WIP] Add tf swiftformer
# What does this PR do?

Adds the TensorFlow version of the "SwiftFormer".

Fixes #22771 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. https://github.com/huggingface/transformers/issues/22771
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @D-Roberts
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23342). All of your documentation changes will be reflected on that endpoint.<|||||>Hi @joaocmd, 
Rapid work on opening the TF port! Let me or @Rocketknight1 know when the PR is ready for review or you experience any issues when porting. <|||||>Hi @Rocketknight1, could I get some pointers as to why I get errors like in most of the tests:

```
E               ValueError: Exception encountered when calling layer 'tf_swift_former_model_18' (type TFSwiftFormerModel).
E               
E               The following keyword arguments are not supported by this model: ['input_ids'].
E               
E               Call arguments received by layer 'tf_swift_former_model_18' (type TFSwiftFormerModel):
E                 • pixel_values={'pixel_values': 'tf.Tensor(shape=(13, 224, 224, 3), dtype=float32)'}
E                 • output_hidden_states=None
E                 • return_dict=None
E                 • training=False

src/transformers/modeling_tf_utils.py:500: ValueError
```

The PyTorch model has this following docstring but I don't see where the input_ids part is being taken care of.
```py
"""
    Here we also overwrite some of the tests of test_modeling_common.py, as SwiftFormer does not use input_ids, inputs_embeds,
    attention_mask and seq_length.
"""
```

Thanks!<|||||>It seems like it is entering in the `else` statement at line 581 of `src/transformers/modeling_tf_utils.py`:

```python
if "args" in output:
    if output["args"] is not None and is_tf_symbolic_tensor(output["args"]):
        tensor_name = output["args"].name.split(":")[0]
        output[tensor_name] = output["args"]
    else:
        # `args` in this case is always the first parameter, then `input_ids`
        output["input_ids"] = output["args"]

    del output["args"]
```

Thus it is injecting the `input_ids` argument into the dictionary.

@amyeroberts @Rocketknight1 How should I get around this? It must be some misconfiguration in my tests or models.

<|||||>@joaocmd Just looking at the error and the CI runs, I think the issue might be a missing `@unpack_inputs` decorator on the call method for the MainLayer class<|||||>> @joaocmd Just looking at the error and the CI runs, I think the issue might be a missing `@unpack_inputs` decorator on the call method for the MainLayer class

Thank you @amyeroberts! It seems like that wasn't causing any issue (yet), but thanks to your comment I found out that I had a duplicate `@unpack_inputs` in one of the models.<|||||>Hi @amyeroberts and @Rocketknight1, can I get some help with the tests that are still failing? I'm getting `ValueError: cannot reshape array of size 10368 into shape (3,3,3,24)` for these two tests:
* `tests/models/swiftformer/test_modeling_tf_swiftformer.py::TFSwiftFormerModelTest::test_compile_tf_model`
* `tests/models/swiftformer/test_modeling_tf_swiftformer.py::TFSwiftFormerModelTest::test_save_load`

But I don't understand exactly what is being reshaped into the wrong shape. Could I get some insight as to what these tests are doing and why it might be failing? Thanks!<|||||>Hi @joaocmd, there's been some large updates to our TF models regarding how they're built - @Rocketknight1 can give you more details :) 

Are these errors happening if you rebase on `main`? <|||||>> Hi @joaocmd, there's been some large updates to our TF models regarding how they're built - @Rocketknight1 can give you more details :)
> 
> Are these errors happening if you rebase on `main`?

Hi @amyeroberts, just rebased the branch. I think it's failing on the same tests but the error on these two tests changed to:
```
NotImplementedError: Could not infer input image shape from config, please override input_signature to specify input shapes.
```

Looking at the stack trace it seems like the image size should have been specified:
```python
if hasattr(vision_config, "image_size"):
    pixel_values_shape[2] = pixel_values_shape[3] = vision_config.image_size
elif hasattr(vision_config, "input_size"):
    pixel_values_shape[2] = pixel_values_shape[3] = vision_config.input_size
else:
   raise NotImplementedError( # <------ this error here
        "Could not infer input image shape from config, please override input_signature to specify input shapes."
    )
```

Shouldn't this also affect the original model?
<|||||>@joaocmd Regarding the error, no, it shouldn't affect the original model. `image_size` is a parameter we add in the configs, even if it's not always used by the model as it's often important for parameterizing other things or understanding. We allow [this here](https://github.com/huggingface/transformers/blob/468aed39afffafe417819a309a4e6d45d2a9e8f4/utils/check_config_attributes.py#L187). It should have been added, and we can add in this PR, but the PT model can do without. 

You'll notice that the error is being raise in `modeling_tf_utils.py`. This is because when constructing a TF model, we have to pass in dummy inputs to build it. In PyTorch this isn't necessary, because we explicitly set the input and output dimensions when creating each layer, so the weight matrices can be created immediately. `image_size` is needed to know the shape of the inputs to pass in. 

As a side note, did you force push after rebasing? From the PR history, it looks like you might not have. As rebasing is a form of "rewriting history" it's necessary to force push.






<|||||>Thanks @amyeroberts, understood. As for the rebase, I had not done one in quite some time and it seems like I did mess it up. I think that is now fixed.

Since I started this PR I have had a fundamental question about huggingface's approach to tensorflow models. The default in TensorFlow is NHWC while in PyTorch it is NCHW, how should I approach this difference in my PR? Based on `modeling_tf_vit.py` I suppose the correct approach is to assume that images are given in PyTorch format and transpose them in the first block, is that so? How does that affect the following blocks?
Also, if we were implementing a model for semantic segmentation, which would return an image with the same size as the original one, would that be returned in the PyTorch format or the default TensorFlow format?

Thank you!<|||||>@joaocmd The pattern we use for the TF vision models is to transpose the NCHW format in the first MainLayer class e.g. [here](https://github.com/huggingface/transformers/blob/868363abb9e72a638b4710d1f5ef1199839b3eec/src/transformers/models/resnet/modeling_tf_resnet.py#L337C18-L337C18) and then transpose back, if pixel values are returned e.g. [here](https://github.com/huggingface/transformers/blob/868363abb9e72a638b4710d1f5ef1199839b3eec/src/transformers/models/resnet/modeling_tf_resnet.py#L348). For some of the older models e.g. ViT this pattern may not have been applied, as these were the first models to be added. 

This pattern means the model is written in the TF compatible NHWC format throughout, but all of our vision models accept and return images in NCHW. <|||||>Thank you @amyeroberts, that makes sense. I've already updated it to match the pattern.

I'm still having some trouble with the `test_compile_tf_model`. Initially it was failing because it was passing a shape `(None, 56, 56, 48)` to a `reshape` (https://github.com/huggingface/transformers/pull/23342/commits/204e216e6047d83775cfb5f0d928b378b73d2e84#diff-7f093399e807b53ca4b63460f610dcc550c2937cb18cd513d71dc49ce6e1b699R385).
I changed the line to use `[-1, width * height, channels]` as shape, which seems like it fixed that case. However, now it is failing because a shape `(None, None, None, 48)` is being passed to that reshape call. Is this expected of this test? According to the stack trace it seems like it's being triggered by a `tf.keras.Model.save()` (https://github.com/joaocmd/transformers/blob/add_tf_swiftformer/tests/test_modeling_tf_common.py#L711).

I've also noticed that there was an overhaul to the serving and dummy_inputs interface (https://github.com/huggingface/transformers/commit/814de8fac7456bd2ce50d1847505da829761bfdc). But maybe @Rocketknight1 can better explain the consequences of this change to mine (and other) PRs.<|||||>@joaocmd Yes, there was a big refactor of the `serving_output` logic. For most models, there's no need to have `serving_output`, `dummy_inputs` or `serving` implemented. You should be able to remove these and have the `test_prepare_serving_output` test pass. 

Looking at the CI run, I don't see `test_compile_tf_model` failing. Were you able to resolve? Or perhaps are you refering to `test_save_load`?