transformers safetensors llama text-generation arxiv:1910.09700 autotrain_compatible text-generation-inference endpoints_compatible region:us
---
datasets:
- pauhidalgoo/patufet-pretrain
language:
- ca
library_name: transformers
license: apache-2.0
tags:
- catalan
- language-model
- transformer
model-index:
- name: cucafera
  results:
  - task:
      type: language-understanding
      name: arc_ca_challenge
    dataset:
      name: arc_ca_challenge
      type: catalan_bench
    metrics:
    - type: acc
      value: 0.2295
      name: Accuracy
    - type: acc_norm
      value: 0.2534
      name: Normalized Accuracy
    - type: acc
      value: 0.4238
      name: Accuracy
    - type: acc_norm
      value: 0.4108
      name: Normalized Accuracy
    - type: acc
      value: 0.614
      name: Accuracy
    source:
      url: https://github.com/EleutherAI/lm-evaluation-harness
      name: Eleuther AI LM Evaluation Harness
  - task:
      type: question-answering
      name: catalanqa
    dataset:
      name: catalanqa
      type: catalan_bench
    metrics:
    - type: exact_match
      value: 0.0037
      name: Exact Match
    - type: f1
      value: 0.0991
      name: F1 Score
    source:
      url: https://github.com/EleutherAI/lm-evaluation-harness
      name: Eleuther AI LM Evaluation Harness
  - task:
      type: machine-translation
      name: flores_ca
    dataset:
      name: flores_ca
      type: flores
    metrics:
    - type: bleu
      value: 0.5934
      name: BLEU
    source:
      url: https://github.com/EleutherAI/lm-evaluation-harness
      name: Eleuther AI LM Evaluation Harness
---

# Model Card for cucafera 🔥🐲 (Base Model)

This document describes **cucafera (Base Model)**, a tiny Catalan Large Language Model (LLM) developed for experimental and educational purposes. The model has been pre-trained exclusively on Catalan text using the **pauhidalgoo/patufet-pretrain** dataset, making it a specialized tool for a variety of Catalan NLP applications.

## Model Details

### Model Description

**cucafera (Base Model)** is a 244-million parameter transformer-based language model inspired by the LLAMA architecture (notably LLAMA3). Despite its relatively small size compared to many contemporary models, it is optimized for generating coherent and contextually relevant text in Catalan.

- **Model Size:** 244M parameters  
- **Architecture:** Transformer-based (LLAMA-inspired) with 30 layers  
- **Embedding Size:** 768  
- **Attention Mechanism:** 4 key/value heads and 8 query heads (using Grouped Query Attention - GQA)  
- **Context Length:** 2048 tokens  
- **Tokenizer:** Byte-Pair Encoding (BPE) with a vocabulary size of 65,536  
- **Activation Function:** GeGLU

### Training Data

The base model was pre-trained using the [patufet-pretrain](https://huggingface.co/datasets/pauhidalgoo/patufet-pretrain) dataset. This dataset consists of high-quality Catalan text curated specifically for pre-training language models, ensuring rich linguistic and cultural representation and high-quality text sources.

### Training Procedure

The model was trained using the LLAMA3-like architecture with the following setup:
- **Total training steps:** 11,007
- **Batch size:** 524288
- **Sequence Length:** 2048
- **Learning rate:** 3e-4 with 200 warmup steps and cosine decay up to 3e-5
- **Optimizer:** AdamW
- **Hardware used:** Single A100 80GB GPU from vast.ai
- **Training duration:** 1.5 days
- **Total tokens processed:** 5.5 billion

- **Hardware and Efficiency:**  
  Training was executed on a single A100 80GB GPU, showcasing that effective LLM training can be achieved with accessible resources while maintaining cost efficiency.

### Direct Use

The base model is suitable for general text generation tasks in Catalan, including but not limited to:
- Creative writing
- Summarization

Note that this is the *base* model, so it is trained on a text completion task.

However, due to its limited size, it is not able to provide correct factual information and you must be aware of this fact when using this model.

### Downstream Applications

Users may fine-tune **cucafera (Base Model)** for domain-specific tasks such as:
- General purpose conversational agents
- Catalan language question answering
- Chatbots and interactive applications tailored for Catalan speakers


### Out-of-Scope Uses

- **High-Stakes Applications:**  
  The model is not recommended for uses where extremely high factual accuracy is required or where outputs could have significant real-world consequences.
- **Non-Catalan Tasks:**  
  Since the model is exclusively trained on Catalan text, it is not suited for tasks in other languages without further training or fine-tuning.
- It **does not** have any fine-tuning for safety, so it may generate incorrect or biased outputs.
- It **should not** be used for generating misinformation or offensive content.

## Bias, Risks, and Limitations

- The model has **no instruction tuning**, so it may not follow prompts effectively.
- It **only understands Catalan**, meaning it is unsuitable for multilingual applications.
- Due to its **small size (244M parameters)**, its knowledge and reasoning capabilities are limited.
- It was trained on **a limited dataset**, which may introduce biases in its outputs.

### Recommendations

- The goal of this model is educational. You are encouraged to train your own model.
- If used in production, **human review** of its outputs is recommended.
- Fine-tuning on task-specific data can **improve accuracy** and **mitigate biases**.
- Users should be cautious when using it in **sensitive or high-stakes applications**.

## How to Get Started

Below is an example code snippet to load and utilize **cucafera (Base Model)** using Hugging Face's Transformers library:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "pauhidalgoo/cucafera"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

input_text = "Història sobre un drac i un cavaller. Hi havia una vegada"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Acknowledgements
This model was developed as an experimental project, inspired by Karpathy's [NanoGPT Series](https://github.com/karpathy/nanoGPT).
My colleague [Roger Baiges](https://huggingface.co/baiges) also trained his own [CatGPT](https://huggingface.co/baiges/CatGPT).

For more details, updates, or to contribute to the project, please visit the [GitHub repository](https://github.com/pauhidalgoo/cucafera)