comet translation multilingual af am ar as az be bg bn br bs ca cs cy da de el en eo es et eu fa fi fr fy ga gd gl gu ha he hi hr hu hy id is it ja jv ka kk km kn ko ku ky la lo lt lv mg mk ml mn mr ms my ne nl no om or pa pl ps pt ro ru sa sd si sk sl so sq sr su sv sw ta te th tl tr ug uk ur uz vi xh yi zh arxiv:2502.14429 base_model:FacebookAI/xlm-roberta-large base_model:finetune:FacebookAI/xlm-roberta-large license:apache-2.0 region:us
---
base_model:
- FacebookAI/xlm-roberta-large
language:
- multilingual
- af
- am
- ar
- as
- az
- be
- bg
- bn
- br
- bs
- ca
- cs
- cy
- da
- de
- el
- en
- eo
- es
- et
- eu
- fa
- fi
- fr
- fy
- ga
- gd
- gl
- gu
- ha
- he
- hi
- hr
- hu
- hy
- id
- is
- it
- ja
- jv
- ka
- kk
- km
- kn
- ko
- ku
- ky
- la
- lo
- lt
- lv
- mg
- mk
- ml
- mn
- mr
- ms
- my
- ne
- nl
- 'no'
- om
- or
- pa
- pl
- ps
- pt
- ro
- ru
- sa
- sd
- si
- sk
- sl
- so
- sq
- sr
- su
- sv
- sw
- ta
- te
- th
- tl
- tr
- ug
- uk
- ur
- uz
- vi
- xh
- yi
- zh
library_name: comet
license: apache-2.0
pipeline_tag: translation
---

# COMET-instant-self-confidence

This model is based on [COMET-early-exit](https://github.com/zouharvi/COMET-early-exit), which is a fork but not compatible with original Unbabel's COMET.
To run the model, you need to first install this version of COMET either with:
```bash
pip install "git+https://github.com/zouharvi/COMET-early-exit#egg=comet-early-exit&subdirectory=comet_early_exit"
```
or in editable mode:
```bash
git clone https://github.com/zouharvi/COMET-early-exit.git
cd COMET-early-exit
pip3 install -e comet_early_exit
```

This model specifically makes prediction at each of the 25 layers, both the score and the confidence.
This time, the confidence is the absolute error with respect to the final layer's prediction.
```python
model = comet_early_exit.load_from_checkpoint(comet_early_exit.download_model("zouharvi/COMET-instant-self-confidence"))
data = [
    {
        "src": "Can I receive my food in 10 to 15 minutes?",
        "mt": "Moh bych obdržet jídlo v 10 do 15 minut?",
    },
    {
        "src": "Can I receive my food in 10 to 15 minutes?",
        "mt": "Mohl bych dostat jídlo během 10 či 15 minut?",
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)

# print predictions at 5th, 12th, and last layer
print("scores", model_output["scores"][0][5], model_output["scores"][0][12], model_output["scores"][0][-1])
print("estimated errors", model_output["confidences"][0][5], model_output["confidences"][0][12], model_output["confidences"][0][-1])

# two top-level outputs
assert len(model_output["scores"]) == 2 and len(model_output["confidences"]) == 2
# each output contains prediction per each layer
assert all(len(l) == 25 for l in model_output["scores"]) and all(len(l) == 25 for l in model_output["confidences"])
```
Outputs (formatted):
```
scores            75.60  86.60  85.74
estimated errors  10.48   3.52   0.83
```

This model is based on the work [Early-Exit and Instant Confidence Translation Quality Estimation](http://arxiv.org/abs/2502.14429) which can be cited as:
```
@misc{zouhar2025earlyexitinstantconfidencetranslation,
      title={Early-Exit and Instant Confidence Translation Quality Estimation}, 
      author={Vilém Zouhar and Maike Züfle and Beni Egressy and Julius Cheng and Jan Niehues},
      year={2025},
      eprint={2502.14429},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.14429}, 
}
```