readme: ---
  language: code
  thumbnail: https://cdn-media.huggingface.co/CodeBERTa/CodeBERTa.png
  datasets:
  - code_search_net
  ---
  
  # CodeBERTa-language-id: The World’s fanciest programming language identification algo 🤯
  
  
  To demonstrate the usefulness of our CodeBERTa pretrained model on downstream tasks beyond language modeling, we fine-tune the [`CodeBERTa-small-v1`](https://huggingface.co/huggingface/CodeBERTa-small-v1) checkpoint on the task of classifying a sample of code into the programming language it's written in (*programming language identification*).
  
  We add a sequence classification head on top of the model.
  
  On the evaluation dataset, we attain an eval accuracy and F1 > 0.999 which is not surprising given that the task of language identification is relatively easy (see an intuition why, below).
  
  ## Quick start: using the raw model
  
  ```python
  CODEBERTA_LANGUAGE_ID = "huggingface/CodeBERTa-language-id"
  
  tokenizer = RobertaTokenizer.from_pretrained(CODEBERTA_LANGUAGE_ID)
  model = RobertaForSequenceClassification.from_pretrained(CODEBERTA_LANGUAGE_ID)
  
  input_ids = tokenizer.encode(CODE_TO_IDENTIFY)
  logits = model(input_ids)[0]
  
  language_idx = logits.argmax() # index for the resulting label
  ```
  
  
  ## Quick start: using Pipelines 💪
  
  ```python
  from transformers import TextClassificationPipeline
  
  pipeline = TextClassificationPipeline(
      model=RobertaForSequenceClassification.from_pretrained(CODEBERTA_LANGUAGE_ID),
      tokenizer=RobertaTokenizer.from_pretrained(CODEBERTA_LANGUAGE_ID)
  )
  
  pipeline(CODE_TO_IDENTIFY)
  ```
  
  Let's start with something very easy:
  
  ```python
  pipeline("""
  def f(x):
      return x**2
  """)
  # [{'label': 'python', 'score': 0.9999965}]
  ```
  
  Now let's probe shorter code samples:
  
  ```python
  pipeline("const foo = 'bar'")
  # [{'label': 'javascript', 'score': 0.9977546}]
  ```
  
  What if I remove the `const` token from the assignment?
  ```python
  pipeline("foo = 'bar'")
  # [{'label': 'javascript', 'score': 0.7176245}]
  ```
  
  For some reason, this is still statistically detected as JS code, even though it's also valid Python code. However, if we slightly tweak it:
  
  ```python
  pipeline("foo = u'bar'")
  # [{'label': 'python', 'score': 0.7638422}]
  ```
  This is now detected as Python (Notice the `u` string modifier).
  
  Okay, enough with the JS and Python domination already! Let's try fancier languages:
  
  ```python
  pipeline("echo $FOO")
  # [{'label': 'php', 'score': 0.9995257}]
  ```
  
  (Yes, I used the word "fancy" to describe PHP 😅)
  
  ```python
  pipeline("outcome := rand.Intn(6) + 1")
  # [{'label': 'go', 'score': 0.9936151}]
  ```
  
  Why is the problem of language identification so easy (with the correct toolkit)? Because code's syntax is rigid, and simple tokens such as `:=` (the assignment operator in Go) are perfect predictors of the underlying language:
  
  ```python
  pipeline(":=")
  # [{'label': 'go', 'score': 0.9998052}]
  ```
  
  By the way, because we trained our own custom tokenizer on the [CodeSearchNet](https://github.blog/2019-09-26-introducing-the-codesearchnet-challenge/) dataset, and it handles streams of bytes in a very generic way, syntactic constructs such `:=` are represented by a single token:
  
  ```python
  self.tokenizer.encode(" :=", add_special_tokens=False)
  # [521]
  ```
  
  <br>
  
  ## Fine-tuning code
  
  <details>
  
  ```python
  import gzip
  import json
  import logging
  import os
  from pathlib import Path
  from typing import Dict, List, Tuple
  
  import numpy as np
  import torch
  from sklearn.metrics import f1_score
  from tokenizers.implementations.byte_level_bpe import ByteLevelBPETokenizer
  from tokenizers.processors import BertProcessing
  from torch.nn.utils.rnn import pad_sequence
  from torch.utils.data import DataLoader, Dataset
  from torch.utils.data.dataset import Dataset
  from torch.utils.tensorboard.writer import SummaryWriter
  from tqdm import tqdm, trange
  
  from transformers import RobertaForSequenceClassification
  from transformers.data.metrics import acc_and_f1, simple_accuracy
  
  
  logging.basicConfig(level=logging.INFO)
  
  
  CODEBERTA_PRETRAINED = "huggingface/CodeBERTa-small-v1"
  
  LANGUAGES = [
      "go",
      "java",
      "javascript",
      "php",
      "python",
      "ruby",
  ]
  FILES_PER_LANGUAGE = 1
  EVALUATE = True
  
  # Set up tokenizer
  tokenizer = ByteLevelBPETokenizer("./pretrained/vocab.json", "./pretrained/merges.txt",)
  tokenizer._tokenizer.post_processor = BertProcessing(
      ("</s>", tokenizer.token_to_id("</s>")), ("<s>", tokenizer.token_to_id("<s>")),
  )
  tokenizer.enable_truncation(max_length=512)
  
  # Set up Tensorboard
  tb_writer = SummaryWriter()
  
  
  class CodeSearchNetDataset(Dataset):
      examples: List[Tuple[List[int], int]]
  
      def __init__(self, split: str = "train"):
          """
          train | valid | test
          """
  
          self.examples = []
  
          src_files = []
          for language in LANGUAGES:
              src_files += list(
                  Path("../CodeSearchNet/resources/data/").glob(f"{language}/final/jsonl/{split}/*.jsonl.gz")
              )[:FILES_PER_LANGUAGE]
          for src_file in src_files:
              label = src_file