Dependency parsing head for pretrained models
# 🚀 Feature request

Add a new classification head for pretrained models, for dependency parsing.

## Motivation

Current heads, such as `AutoModelForTokenClassification`, does not work well for finetuning a pretrained model for dependency parsing. [There are now such heads available](https://openreview.net/pdf?id=Hk95PK9le), so adding such would make it a lot easier to finetune models for such tasks. A `PyTorch` implementation of that paper (and others) can be found [in this repo](https://github.com/yzhangcs/parser).

## Your contribution

I could assist with a PR, but for now I'd like to start the discussion to see if this is something that the HF team and others would be interested in.
Dan, did you start working on this? I would be glad to help with this feature.<|||||>Why me :| @huberemanuel <|||||>Sorry, wrong mention<|||||>@huberemanuel I have a prototype implementation currently, but it probably needs a bunch of tweaking. You can find it at https://github.com/saattrupdan/ScandEval/blob/dependency-parsing/scandeval/dependency_parsing/parser.py#L17.

Right now it's basically a hack which tweaks the `forward` method of the `AutoModelForTokenClassification`. A proper implementation would use the underlying code instead of subclassing it, of course!

If you're interested in testing it and improving the implementation then let me know and we could work it out together 🙂 <|||||>Thanks! I will take a look into it and see how can I contribute to your initiative.<|||||>Hey Dan, just to send an update about this topic. I messed around with the `transformers` code and added an `AutoModelForDependencyParsing` class which for now is just a copy of `AutoModelForTokenClassification`. With that, I ran the NER training script, just to do a sanity test on token classification training. It all worked well and now I know which parts we will need to modify to add this feature. 
I will send more updates as I progress, and when I got to the point to merge your code with the `transformers` internals, we can discuss what design is better. If you advance more, please let me know :) <|||||>@huberemanuel Amazing, great stuff! With the internals in place, it's hopefully doable to tweak it in (roughly) the way I did. Thanks for all the work you're putting into it, and would love to see your code when you're ready to share it! 😊 <|||||>@huberemanuel Any news on this?