[up]: Update README and load model

README.md

@@ -15,4 +15,112 @@ tags:
 - genereted_text_detection
 - llm_content_detection
 - AI_detection
----
+---
+
+<p align="center">
+<img src="SA_logo.png" alt="SuperAnnotate Logo" width="100" height="100"/>
+</p>
+
+<h1 align="center">SuperAnnotate</h1>
+<h3 align="center">
+LLM Content Detector V2<br/>
+Fine-Tuned RoBERTa Large<br/>
+</h3>
+
+## Description
+
+The model designed to detect generated/synthetic text. \
+At the moment, such functionality is critical for determining the author of the text. It's critical for your training data, detecting fraud and cheating in scientific and educational areas. \
+Couple of articles about this problem: [*Problems with Synthetic Data*](https://www.aitude.com/problems-with-synthetic-data/) | [*Risk of LLMs in Education*](https://publish.illinois.edu/teaching-learninghub-byjen/risk-of-llms-in-education/)
+
+## Model Details
+
+### Model Description
+
+- **Model type:** The custom architecture for binary sequence classification based on pre-trained RoBERTa, with a single output label.
+- **Language(s):** Primarily English.
+- **License:** [SAIPL](https://huggingface.co/SuperAnnotate/roberta-large-llm-content-detector-V2/blob/main/LICENSE)
+- **Finetuned from model:** [RoBERTa Large](https://huggingface.co/FacebookAI/roberta-large)
+
+### Model Sources
+
+- **Repository:** [GitHub](https://github.com/superannotateai/generated_text_detector) for HTTP service
+
+### Training Data
+
+The training data with 'human' label was sourced from three open datasets with equal proportions:
+
+1. [**Wikipedia**](https://huggingface.co/datasets/wikimedia/wikipedia)
+1. [**Reddit ELI5 QA**](https://huggingface.co/datasets/rexarski/eli5_category)
+1. [**Scientific Papers**](https://www.tensorflow.org/datasets/catalog/scientific_papers) extended version with full text of sections
+
+The second half of the dataset was obtained by generating answers to the corresponding human texts.
+For generation, 14 models from 4 different families were used, namely: GPT, LLaMA, Anthropic and Mistral
+
+As a result, the training dataset contained approximately ***36k*** pairs of text-label with an approximate balance of classes. \
+It's worth noting that the dataset's texts follow a logical structure: \
+Human-written and model-generated texts refer to a single prompt/instruction, though the prompts themselves were not used during training.
+
+> [!NOTE]
+> Furthermore, key n-grams (n ranging from 2 to 5) that exhibited the highest correlation with target labels were identified and subsequently removed from the training data utilizing the chi-squared test.
+
+### Peculiarity
+
+During training, one of the priorities was not only maximizing the quality of predictions but also avoiding overfitting and obtaining an adequately confident predictor. \
+We are pleased to achieve the following state of model calibration:
+
+**TODO** Change graph or this section in general.
+
+## Usage
+
+**Pre-requirements**: \
+Install *generated_text_detector* \
+Run following command: ```pip install git+https://github.com/superannotateai/[email protected]```
+
+```python
+from generated_text_detector.utils.model.roberta_classifier import RobertaClassifier
+from transformers import AutoTokenizer
+import torch.nn.functional as F
+
+
+model = RobertaClassifier.from_pretrained("SuperAnnotate/roberta-large-llm-content-detector-V2")
+tokenizer = AutoTokenizer.from_pretrained("SuperAnnotate/roberta-large-llm-content-detector-V2")
+
+text_example = "It's not uncommon for people to develop allergies or intolerances to certain foods as they get older. It's possible that you have always had a sensitivity to lactose (the sugar found in milk and other dairy products), but it only recently became a problem for you. This can happen because our bodies can change over time and become more or less able to tolerate certain things. It's also possible that you have developed an allergy or intolerance to something else that is causing your symptoms, such as a food additive or preservative. In any case, it's important to talk to a doctor if you are experiencing new allergy or intolerance symptoms, so they can help determine the cause and recommend treatment."
+
+tokens = tokenizer.encode_plus(
+text_example,
+add_special_tokens=True,
+max_length=512,
+padding='longest',
+truncation=True,
+return_token_type_ids=True,
+return_tensors="pt"
+)
+
+_, logits = model(**tokens)
+
+proba = F.sigmoid(logits).squeeze(1).item()
+
+print(proba)
+```
+
+## Training Detailes
+
+A custom architecture was chosen for its ability to perform binary classification while providing a single model output, as well as for its customizable settings for smoothing integrated into the loss function.
+
+**Training Arguments**:
+
+- **Base Model**: [FacebookAI/roberta-large](https://huggingface.co/FacebookAI/roberta-large)
+- **Epochs**: 20
+- **Learning Rate**: 5e-05
+- **Weight Decay**: 0.001
+- **Label Smoothing**: 0.27
+- **Warmup Epochs**: 2
+- **Optimizer**: SGD
+- **Gradient Clipping**: 3.0
+- **Scheduler**: Cosine with hard restarts
+
+## Performance
+
+**TODO** RAID Leaderboard should be here

config.json

@@ -0,0 +1,13 @@
+{
+    "pretrain_checkpoint": "FacebookAI/roberta-large",
+    "classifier_dropout": 0.1,
+    "num_labels": 1,
+    "id2label": {
+        "0": "GENERATED"
+    },
+    "label2id": {
+        "GENERATED": 0
+    },
+    "label_smoothing": 0.27,
+    "transformers_version": "4.45.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29f34aac13e42e030bb661c1b60ca6bd9a1f601d53b7b3fe45e0352926741616
+size 1417292644

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50264": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "do_lower_case": true,
+  "eos_token": "</s>",
+  "errors": "replace",
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}