readme: # Comcat: A Quality Analysis of Source Code Comments via Comment Categorization
  By Skyler Grandel
  
  ## Overview
  This project aims to automatically provide a qualitative analysis of software comments through a novel classification schema. This schema will make use of recent advancements in natural language processing to systematically classify source code comments by purpose and the information they provide.
  
  ### The Problem
  It is very difficult to qualitatively assess comments (or really anything) in a systematic or objective way. Therefore, we need criteria by which we might determine if a comment is useful or not. A common approach to this problem is to classify our data and then experimentally determine which classes are the most useful. Therefore, we need a classification schema and a tool to automatically classify comments according to this schema.
  
  ### The Solution
  To develop a classification schema, we classified comments by the information they provide about the associated code until we reached saturation. This means that we determined what information the comment contained and placed it in the appropriate category; if no appropriate category existed, we created a new category until we were reasonably sure that no more categories existed (saturation in this case was defined as classifying at least 100 comments without discovering a new category). The resulting categories are listed below:
  
  LABEL_0: Function (A comment that describes a function. Typically a comment is in this cat iff it's javadoc style and it's not a copyright comment)
  
  LABEL_1: Variable (Describe Variable/Constant) 
  
  LABEL_2: Functionality (Describe block functionality; usually just summarizes or describes what the following lines of code do; specifically inline)
  
  LABEL_3: Branch (Describe possible branches and/or preconditions)
  
  LABEL_4: Reasoning (Describe reasoning behind implementation decisions)
  
  LABEL_5: Quirk (Random quirk of the code, authors trying to be funny, niche problems the code can run into, etc)
  
  LABEL_6: Use guidelines (Comments that tell readers how to use a function/container/variable but dont describe their function)
  
  LABEL_7: Source (Describe Code Source; comments like /\* This code was stolen from \[some stack overflow link\]\*/)
  
  LABEL_8: Copyright (Copyright and author info)
  
  LABEL_9: Section (comments like, /\* The following tests are to test foo() \*/ or /\* Setter and Getter methods \*/)
  
  LABEL_10: Code (commented out code)
  
  LABEL_11: Task (TODOs, FIXMEs, etc.)
  
  To automatically classify these comments, we fine tuned a roBERTa based classification transformer on a dataset of manually labeled comments. This utilizes the transformer as a feature selector to vectorize the comment and a NN based classifier head.
  
  ## Usage
  First, it is important to note that this model was trained on C comments, so we cannot guarantee that it will work with comments in other languages. However, languages with similar comment syntax to C (such as C++ or Java) are expected to work well, and other languages are also expected to work somewhat if they are preprocessed. For all comments, you will get the best results if you preprocess them by removing all non-alphanumeric characters other than spaces using the following algorithm (in Python):
  
  `def preprocess(comment):`
  
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`comment = ''.join(e for e in comment if (e.isalnum() or e == ' '))`
      
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`return comment`
  
  It is important not to remove leading, trailing, or duplicate spaces because the model can use those to determine the format of a comment, even when the special characters are removed. This model can easily be loaded using huggingface classes and functions:
  
  `from transformers import RobertaTokenizerFast, AutoModelForSequenceClassification, TextClassificationPipeline`
  
  `import torch`
  
  `tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base')`
  
  `DEVICE = "cuda:1" if torch.cuda.is_available() else "cpu" # use whatever device works for you`
  
  `num_labels = 12`
  
  `model = (AutoModelForSequenceClassification.from_pretrained("skylergrandel/Comcat", num_labels=num_labels).to(DEVICE))`
  
  `classifier = TextClassificationPipeline(model=model, tokenizer=tokenizer, device=DEVICE)`
  
  `prediction = classifier(preprocess('/* This is a comment that we want to classify */'))`
  
  `print('Prediction:', prediction)`
  
  Note that we use the roberta-base tokenizer, because this model was based on roberta-base Example usage of this model can be found in a GitHub repo linked below. 
  
  ## Resource Links
  An example usage of the model can be found on [this GitHub repo](git@github.com:skylergrandel/transformers-2022-final-project.git).
  
  Here are a few papers for some background on comments, comment categorization, and automatic comment generation:
  
  - Steidl et al. classify comments into seven categories using older ML techniques in [“Quality analysis of source code comments”](https://ieeexplore.ieee.org/document/6613836)
  
  - Song et al