Graphormer model for Graph Classification
# What does this PR do?

Adds the Graphormer model for graph classification in Transformers.

Done:
- [x] Architecture ported
- [x]  Collator (the model has no tokenizer) and preprocessing
- [x] Test results against original implementation, to make sure they are within precision range. Edit: exactly same results :fire: 
- [x] Add checkpoints and make sure they load properly
- [x] Update test suite
- [x] Add model card for the checkpoints (https://huggingface.co/clefourrier/pcqm4mv2_graphormer_base, https://huggingface.co/clefourrier/pcqm4mv1_graphormer_base)
- [x] Update doc

## Dependencies
Cython - this could be ported to Python, but preprocessing will be considerably slower, as well as collation if preprocessing is done on the fly. 

Linked to #20962

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. (Discussed with Thom on Slack)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@sgugger Sorry for not having put my PR in draft and thank you so much for your comments! 

I'll take them into account, keep working on this, and ping you back when I have cleaner code?<|||||>No worries at all, and yes!<|||||>@sgugger I think the code is better now, if you want to take a look.

I have several questions:
- ~should I add more documentation?~
- how can I express the fact that this model does not use a tokenizer, but a collator, so that it appears in the doc ?
- I'm having trouble with the common test suites: for the inputs, for example, we embed nodes and edges of graphs using two different embedding layers > what should "get_input_embeddings" return then? A concatenation of both? Similarly, `input_ids` has no equivalent in our case, as our inputs are both `input_nodes` and `input_edges`, so I'm a bit stuck on what to do here 

Edit: talked to @LysandreJik, will edit tests (though they'll be model specific atm) and doc will stay minimal for now<|||||>@clefourrier

It seems there is an issue with your CircleCI permissions, the tests are not run.
Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?
<|||||>Just did, nothing changed, and I don't have the permission to manually trigger the pipeline on the [CI webpage](https://app.circleci.com/pipelines/github/huggingface/transformers?branch=pull%2F20968).

Do you have other ideas of things I could try? :hugs:<|||||>> Just did, nothing changed

you can do
```bash
git commit --allow-empty -m "Empty commit to trigger CI"
git push
```

> I don't have the permission to manually trigger the pipeline

This usually means you opened that job run page without login.<|||||>Cool, a commit seems to have solved it! :pray: @ydshieh !
<|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>@ydshieh All tests for my model are passing - thank you for your pointers! 
<|||||>Thank you so much for all the time spent reviewing @sgugger @ydshieh ! :pray: 
I hope this time it's finally up to par ^^

Quick question: once merged, how long till it is in a release of transformers? I have a blog post on Graph Classification with this model, when should I plan on publishing it/communicating about it?<|||||>There should be a release of Transformers next week by the way.<|||||>@clefourrier 

There are a few tests failed on daily CI. See [here](https://github.com/huggingface/transformers/actions/runs/3964181931/jobs/6792864795).

I can also help, but I have one question: Does `GraphormerModel` require all the following inputs to be specified?

```python
    def forward(
        ...
        input_nodes,
        input_edges,
        attn_bias,
        in_degree,
        out_degree,
        spatial_pos,
        attn_edge_type,
```<|||||>Hi @clefourrier!

When you get some time, could you take a look on the following failed tests:
```
tests/models/graphormer/test_modeling_graphormer.py::GraphormerModelTest::test_model_from_pretrained
tests/models/graphormer/test_modeling_graphormer.py::GraphormerModelIntegrationTest::test_inference_graph_classification
```
which seems related to the missing or wrong checkpoint link/path.

Regarding the other 3 failed tests (`GraphormerModelTest::test_torchscript_xxx`), I can work on them, but I need a bit of context regarding the inputs for this model 🙏 , see my comment above. Thank you :-)<|||||>@ydshieh Hi! Back from vacations! :wave: 

- Edit: The checkpoint is now here:https://huggingface.co/clefourrier/graphormer-base-pcqm4mv2 (The problem likely came from the wrong dash type, changed the path)
- The model needs as inputs all the inputs you mentioned, which are generated during data preprocessing. If you need more information on the model, I wrote a blog post (still a PR atm https://github.com/huggingface/blog/pull/781) which describes inputs and use and feel free to ask any questions which could help!<|||||>@ydshieh Opened a new PR #21367 to manage the checkpoint path problems. 