transformers safetensors dataset:lecslab/glosslm-corpus-split arxiv:2403.06399 base_model:google/byt5-base base_model:finetune:google/byt5-base license:apache-2.0 endpoints_compatible region:us
---
base_model:
- google/byt5-base
datasets:
- lecslab/glosslm-corpus-split
library_name: transformers
license: apache-2.0
metrics:
- accuracy
- chrf
- bleu
---

- Repo: https://github.com/foltaProject/glosslm
- Paper: https://arxiv.org/abs/2403.06399

Usage:
```python
import transformers

# Your inputs
transcription = "o sey xtok rixoqiil"
translation = "O sea busca esposa."
lang = "Uspanteco"
metalang = "Spanish"
is_segmented = False

prompt = f"""Provide the glosses for the following transcription in {lang}.

Transcription in {lang}: {transcription}
Transcription segmented: {is_segmented}
Translation in {metalang}: {translation}\n
Glosses: 
"""

model = transformers.T5ForConditionalGeneration.from_pretrained("lecslab/glosslm")
tokenizer = transformers.ByT5Tokenizer.from_pretrained(
    "google/byt5-base", use_fast=False
)

inputs = tokenizer(prompt, return_tensors="pt")
outputs = tokenizer.batch_decode(
    model.generate(**inputs, max_length=1024), skip_special_tokens=True
)
print(outputs[0])
# o sea COM-buscar E3S-esposa
```