readme: ---
  pipeline_tag: text-classification
  
  language:
  
  - ca
  
  license: apache-2.0
  
  tags:
  
  - "catalan"
  
  - "semantic textual similarity"
  
  - "sts-ca"
  
  - "CaText"
  
  - "Catalan Textual Corpus"
  
  datasets:
  
  - "projecte-aina/sts-ca"
  
  metrics:
  
  - "combined_score"
  
  model-index:
  
  - name: roberta-base-ca-v2-cased-sts
    results:
    - task: 
        type: text-classification
      dataset:
        type:   projecte-aina/sts-ca
        name: STS-ca
      metrics:
        - name: Combined score
          type: combined_score
          value: 0.7907
          
  ---
  
  # Catalan BERTa-v2 (roberta-base-ca-v2) finetuned for Semantic Textual Similarity.
  
  ## Table of Contents
  <details>
  <summary>Click to expand</summary>
  
  - [Model description](#model-description)
  - [Intended uses and limitations](#intended-use)
  - [How to use](#how-to-use)
  - [Limitations and bias](#limitations-and-bias)
  - [Training](#training)
    - [Training data](#training-data)
    - [Training procedure](#training-procedure)
  - [Evaluation](#evaluation)
     - [Variable and metrics](#variable-and-metrics)
     - [Evaluation results](#evaluation-results)
  - [Additional information](#additional-information)
    - [Author](#author)
    - [Contact information](#contact-information)
    - [Copyright](#copyright)
    - [Licensing information](#licensing-information)
    - [Funding](#funding)
    - [Citing information](#citing-information)
    - [Disclaimer](#disclaimer)
  </details>
  
  ## Model description
  
  The **roberta-base-ca-v2-cased-sts** is a Semantic Textual Similarity (STS) model for the Catalan language fine-tuned from the [roberta-base-ca-v2](https://huggingface.co/projecte-aina/roberta-base-ca-v2) model, a [RoBERTa](https://arxiv.org/abs/1907.11692) base model pre-trained on a medium-size corpus collected from publicly available corpora and crawlers (check the roberta-base-ca-v2 model card for more details).
  
  ## Intended uses and limitations
  
  **roberta-base-ca-v2-cased-sts** model can be used to assess the similarity between two snippets of text. The model is limited by its training dataset and may not generalize well for all use cases.
  
  ## How to use
  To get the correct<sup>1</sup> model's prediction scores with values between 0.0 and 5.0, use the following code:
  
  ```python
  from transformers import pipeline, AutoTokenizer
  from scipy.special import logit
  
  model = 'projecte-aina/roberta-base-ca-v2-cased-sts'
  tokenizer = AutoTokenizer.from_pretrained(model)
  pipe = pipeline('text-classification', model=model, tokenizer=tokenizer)
  
  def prepare(sentence_pairs):
      sentence_pairs_prep = []
      for s1, s2 in sentence_pairs:
          sentence_pairs_prep.append(f"{tokenizer.cls_token} {s1}{tokenizer.sep_token}{tokenizer.sep_token} {s2}{tokenizer.sep_token}")
      return sentence_pairs_prep
  
  sentence_pairs = [("El llibre va caure per la finestra.", "El llibre va sortir volant."),
                    ("M'agrades.", "T'estimo."),
                    ("M'agrada el sol i la calor", "A la Garrotxa plou molt.")]
  
  predictions = pipe(prepare(sentence_pairs), add_special_tokens=False)
  
  # convert back to scores to the original 0 and 5 interval
  for prediction in predictions:
      prediction['score'] = logit(prediction['score'])
  print(predictions)
  ```
  Expected output:
  ```
  [{'label': 'SIMILARITY', 'score': 2.118301674983813}, 
  {'label': 'SIMILARITY', 'score': 2.1799755855125853}, 
  {'label': 'SIMILARITY', 'score': 0.9511617858568939}]
  ```
  
  <sup>1</sup> _**avoid using the widget** scores since they are normalized and do not reflect the original annotation values._
  
  
  ## Limitations and bias
  At the time of submission, no measures have been taken to estimate the bias embedded in the model. However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this model card will be updated.
  
  ## Training
  
  ### Training data
  We used the STS dataset in Catalan called [STS-ca](https://huggingface.co/datasets/projecte-aina/sts-ca) for training and evaluation.
  
  ### Training procedure
  The model was trained with a batch size of 16 and a learning rate of 5e-5 for 5 epochs. We then selected the best checkpoint using the downstream task metric in the corresponding development set, and then evaluated it on the test set.
  
  ## Evaluation
  
  ### Variable and metrics
  
  This model was finetuned maximizing the average score between the Pearson and Spearman correlations.
  
  ## Evaluation results
  We evaluated the _roberta-base-ca-v2-cased-sts_ on the STS-ca test set against standard multilingual and monolingual baselines:
  
  | Model       | STS-ca (Combined score)   | 
  | ------------|:-------------|
  | roberta-base-ca-v2-cased-sts | 79.07 |
  | roberta-base-ca-cased-sts | **80.19** |
  | mBERT       | 74.26 |
  | XLM-RoBERTa | 61.61 |
  
  For more details, check the fine-tuning and evaluation scripts in the official [GitHub repository](https://github.com/projecte-aina/club).
  
  ## Additional information
  
  ### Author
  Text Mining Unit (TeM