readme: ---
  license: cc-by-sa-4.0
  datasets:
  - cjvt/sentinews
  language:
  - sl
  library_name: transformers
  pipeline_tag: text-classification
  model-index:
    - name: sloberta-sentinews-sentence
      results:
      - task:
          type: text-classification
          name: Sentiment classification
        dataset:
          type: cjvt/sentinews
          name: SentiNews
          config: sentence_level
        metrics:
          - type: f1
            value: 0.6851357247321056
            name: Test macro F1
          - type: accuracy
            value: 0.7158081705150977
            name: Test accuracy
          - type: f1
            value: 0.6934678744913757
            name: Validation macro F1
          - type: accuracy
            value: 0.7207815275310835
            name: Validation accuracy 
  ---
  
  # sloberta-sentinews-sentence
  
  Slovenian 3-class sentiment classifier - [SloBERTa](https://huggingface.co/EMBEDDIA/sloberta) fine-tuned on the sentence-level config of the 
  SentiNews dataset.
  
  The model is intended as:  
  (1) an out-of-the box sentence-level sentiment classifier or   
  (2) a sentence-level sentiment classification baseline.
  
  ## Fine-tuning details
  The model was fine-tuned on a random 90%/5%/5% train-val-test split of the `sentence_level` configuration of the [cjvt/sentinews](https://huggingface.co/datasets/cjvt/sentinews) dataset 
  using the following hyperparameters:  
  ```
  max_length = 79  # 99th percentile of encoded training sequences, sequences are padded/truncated to this length
  batch_size = 128
  optimizer = "adamw_torch"
  learning_rate = 2e-5
  num_epochs = 10
  validation_metric = "macro_f1"
  ```
  
  Feel free to inspect `training_args.bin` for more details.
  
  If you wish to directly compare your model to this one, you should use the same split as this model. To do so, use the following code:  
  ```python
  import json
  import datasets
  
  # You can find split_indices.json in the 'Files and versions' tab 
  with open("split_indices.json", "r") as f_split:
    split = json.load(f_split)
  
  data = datasets.load_dataset("cjvt/sentinews", "sentence_level", split="train")
  train_data = data.select(split["train_indices"])
  dev_data = data.select(split["dev_indices"])
  test_data = data.select(split["test_indices"])
  ```
  
  ## Evaluation results
  
  Best validation set results:
  ```
  {
    "eval_accuracy": 0.7207815275310835,
    "eval_f1_macro": 0.6934678744913757,
    "eval_f1_negative": 0.7042136003337507,
    "eval_f1_neutral": 0.759215853398679,
    "eval_f1_positive": 0.6169741697416974,
    "eval_loss": 0.6337869167327881,
    "eval_precision_negative": 0.6685148514851486,
    "eval_precision_neutral": 0.7752393385552655,
    "eval_precision_positive": 0.6314199395770392,
    "eval_recall_negative": 0.74394006170119,
    "eval_recall_neutral": 0.7438413361169103,
    "eval_recall_positive": 0.6031746031746031
  }
  ```
  
  Test set results:
  ```
  {
    "test_loss": 0.6395984888076782,
    "test_accuracy": 0.7158081705150977,
    "test_precision_negative": 0.6570397111913358,
    "test_recall_negative": 0.7292965271593945,
    "test_f1_negative": 0.6912850812407682,
    "test_precision_neutral": 0.7748017998714377,
    "test_recall_neutral": 0.7418957734919983,
    "test_f1_neutral": 0.7579918247563149,
    "test_precision_positive": 0.6155642023346304,
    "test_recall_positive": 0.5969811320754717,
    "test_f1_positive": 0.6061302681992337,
    "test_f1_macro": 0.6851357247321056,
  }
  ```