transformers safetensors arxiv:1910.09700 endpoints_compatible region:us
---
library_name: transformers
tags: []
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
This model card contains a LoRa adapter for the Gemma-9b-it model. It aims to improve Machine Translation performance into South African langauges.


## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

The LoRa adapter was trained on a subset of the lelapa/Inkuba-instruct dataset targeting the langauges English, isiZulu and isiXhosa. The data was sampled from the Inkuba-instruct dataset using langauge-ids.

### Model Sources [optional]
- **Developed by:** Sandil Bhikha
- **Model type:** LoRa adapter
- **Language(s) (NLP):** English, isiZulu, isiXhosa
- **Finetuned from model:** Gemma-9b-it

<!-- Provide the basic links for the model. -->

- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Aims to improve Machine Translation performance on South African langauges. Based on Gemma-9b-it, so the model is generally task adaptable.

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The datatset was sampled from  lelapa/Inkuba-instruct dataset. Minor inconsistencies were present in translations from the training data.

## How to Get Started with the Model

Use the code below to get started with the model.
```
base_model_id = "google/gemma-2-9b-it"
adapter_model_id = "sbhikha/Gemma9B_Inkuba_double_quant"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

double_quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config = double_quant_config,
    device_map="auto"
)

config = PeftConfig.from_pretrained(adapter_model_id)
peft_model = PeftModel.from_pretrained(model, adapter_model_id)

text = "Translate this from English to isiZulu: Sawubona, unjani?"

input_ids = tokenizer(text), return_tensors="pt").to("cuda:0")

outputs = peft_model.generate(**input_ids, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

#### English
- sbhikha/Inkuba_english_train_sliced_v1
- sbhikha/Inkuba_english_dev_sliced_v1

#### Zulu
- sbhikha/Inkuba_isizulu_train_sliced_v1
- sbhikha/Inkuba_isizulu_dev_sliced_v1

#### Xhosa
- sbhikha/Inkuba_xhosa_train_sliced_v1
- sbhikha/Inkuba_xhosa_dev_sliced_v1

### Training Hyperparameters

- **Training regime:** bf16 mixed precision <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

LoRa Config:
- Rank = 64
- Alpha = 32
- Target Modules (Linear Modules): ['q_proj', 'gate_proj', 'up_proj', 'k_proj', 'v_proj', 'o_proj', 'down_proj']
- Dropout = 0.05

Training Arguments:
- Learning Rate: 2e-4
- per_device_train_batch_size: 1
- gradient_accumulation_steps: 4
- warmup_steps = 3
- max_steps = 100
- optim="paged_adamw_8bit"

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->


## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** 1x H100 GPU
- **Hours used:** 23.7
- **Cloud Provider:** Lambda Labs
- **Compute Region:** Utah, USA
- **Carbon Emitted:** [More Information Needed]