readme: ---
  license: cc-by-4.0
  ---
  ## Aina Project's Catalan-English  machine translation model
  
  ## Table of Contents
  - [Model Description](#model-description)
  - [Intended Uses and Limitations](#intended-use)
  - [How to Use](#how-to-use)
  - [Training](#training)
    - [Training data](#training-data)
    - [Training procedure](#training-procedure)
      - [Data Preparation](#data-preparation)
      - [Tokenization](#tokenization)
      - [Hyperparameters](#hyperparameters)
  - [Evaluation](#evaluation)
     - [Variable and Metrics](#variable-and-metrics)
     - [Evaluation Results](#evaluation-results)
  - [Additional Information](#additional-information)
    - [Author](#author)
    - [Contact Information](#contact-information)
    - [Copyright](#copyright)
    - [Licensing Information](#licensing-information)
    - [Funding](#funding)
    - [Disclaimer](#disclaimer)
    
  ## Model description
  
  This model was trained from scratch using the [Fairseq toolkit](https://fairseq.readthedocs.io/en/latest/) on a combination of Catalan-English datasets, up to 11  million sentences. Additionally, the model is evaluated on several public datasecomprising 5 different domains (general, adminstrative, technology, biomedical, and news).  
  
  ## Intended uses and limitations
  
  You can use this model for machine translation from Catalan to English. 
  
  ## How to use
  
  ### Usage
  Required libraries:
  
  ```bash
  pip install ctranslate2 pyonmttok
  ```
  
  Translate a sentence using python 
  ```python
  import ctranslate2
  import pyonmttok
  from huggingface_hub import snapshot_download
  model_dir = snapshot_download(repo_id="projecte-aina/mt-aina-ca-en", revision="main")
  
  tokenizer=pyonmttok.Tokenizer(mode="none", sp_model_path = model_dir + "/spm.model")
  tokenized=tokenizer.tokenize("Benvingut al projecte Aina!")
  
  translator = ctranslate2.Translator(model_dir)
  translated = translator.translate_batch([tokenized[0]])
  print(tokenizer.detokenize(translated[0][0]['tokens']))
  ```
  
  ## Training
  
  ### Training data
  
  The model was trained on a combination of the following datasets:
  
  | Dataset            | Sentences      |
  |--------------------|----------------|
  | Global Voices      | 21.342         |
  | Memories Lluires   | 1.173.055      |
  | Wikimatrix         | 1.205.908      |
  | TED Talks          | 50.979         |
  | Tatoeba            | 5.500          |
  | CoVost 2 ca-en     | 79.633         |
  | CoVost 2 en-ca     | 263.891        |
  | Europarl           | 1.965.734      |
  | jw300              | 97.081         |
  | Crawled Generalitat| 38.595         |
  | Opus Books         | 4.580          |
  | CC Aligned         | 5.787.682      |
  | COVID_Wikipedia    | 1.531          |
  | EuroBooks          | 3.746          |
  | Gnome              | 2.183          |
  | KDE 4              | 144.153        |
  | OpenSubtitles      | 427.913        |
  | QED                | 69.823         |
  | Ubuntu             | 6.781          |
  | Wikimedia          | 208.073        |
  |--------------------|----------------|
  | **Total**          | **11.558.183** |
  
  ### Training procedure
  
  ### Data preparation
  
   All datasets are concatenated and filtered using the [mBERT Gencata parallel filter](https://huggingface.co/projecte-aina/mbert-base-gencata). Before training, the punctuation is normalized using a modified version of the join-single-file.py script from [SoftCatalà](https://github.com/Softcatala/nmt-models/blob/master/data-processing-tools/join-single-file.py)
  
  
  #### Tokenization
  
   All data is tokenized using sentencepiece, using 50 thousand token sentencepiece model  learned from the combination of all filtered training data. This model is included.  
  
  #### Hyperparameters
  
  The model is based on the Transformer-XLarge proposed by [Subramanian et al.](https://aclanthology.org/2021.wmt-1.18.pdf)
  The following hyperparamenters were set on the Fairseq toolkit:
  
  | Hyperparameter                     | Value                             |
  |------------------------------------|-----------------------------------|
  | Architecture                       | transformer_vaswani_wmt_en_de_big |
  | Embedding size                     | 1024                              |
  | Feedforward size                   | 4096                              |
  | Number of heads                    | 16                                |
  | Encoder layers                     | 24                                |
  | Decoder layers                     | 6                                 |
  | Normalize before attention         | True                              |
  | --share-decoder-input-output-embed | True                              |
  | --share-all-embeddings             | True                              |
  | Effective batch size               | 96.000                            |
  | Optimizer                          | adam                              |
  | Adam betas                         | (0.9, 0.980)                      |
  | Clip norm                          | 0.0                               |
  | Learning rate                      | 1e-3                              