readme: ---
  language: 
    - en
  tags:
    - formality
  datasets:
    - GYAFC
    - Pavlick-Tetreault-2016
  ---
  
  The model has been trained to predict for English sentences, whether they are formal or informal. 
  
  Base model: `roberta-base`
  
  Datasets: [GYAFC](https://github.com/raosudha89/GYAFC-corpus) from [Rao and Tetreault, 2018](https://aclanthology.org/N18-1012) and [online formality corpus](http://www.seas.upenn.edu/~nlp/resources/formality-corpus.tgz) from [Pavlick and Tetreault, 2016](https://aclanthology.org/Q16-1005).
  
  Data augmentation: changing texts to upper or lower case; removing all punctuation, adding dot at the end of a sentence. It was applied because otherwise the model is over-reliant on punctuation and capitalization and does not pay enough attention to other features.
  
  Loss: binary classification (on GYAFC), in-batch ranking (on PT data).
  
  Performance metrics on the test data:
  
  | dataset                                      | ROC AUC | precision | recall | fscore | accuracy | Spearman |
  |----------------------------------------------|---------|-----------|--------|--------|----------|------------|
  | GYAFC                                        | 0.9779  | 0.90      | 0.91   | 0.90   | 0.9087   | 0.8233     |
  | GYAFC normalized (lowercase + remove punct.) | 0.9234  | 0.85      | 0.81   | 0.82   | 0.8218   | 0.7294     |
  
  | P&T subset | Spearman R |
  | -     | - |
  news    |	 0.4003
  answers |	 0.7500
  blog    |	 0.7334
  email   |	 0.7606
  