Looks like you made a lot of progress, great work! I think having torch-scatter as a soft dependency required for this model is OK. Look at what was done in RAG recently, but basically, you should implement functions like is_scatter_available and requires_scatter in files_utils.py and have a requires_scatter in the init of your models (so it fails with a nice error message if a user doesn’t have the dep).
It looks like you only have to do the final clean-up before opening a PR. Make sure you copy what is done in other modeling files for documentation to have clean docstrings for your models.
One bit of work you have left is to remove imports form the modeling_bert file: for easy customization by researchers, we want all model files to be independent (at the price of dupe code). We have a script that checks copies form other modeling files stay consistent (and can be automatically updated). Check what was done in modeling_roberta.py for an example.