Also wanted to share how we trained huggingface/CodeBERTa-language-id · Hugging Face 18 (trained on code_search_net as well) – see original model card: huggingface/CodeBERTa-small-v1 · Hugging Face 5
That was 1.5 years ago so I think things have changed significantly since then!
In particular, I am super curious which tokenizer type to optimally pick for code (maybe char-based in a better option now, cc @patrickvonplaten )