Update README.md

README.md

@@ -1,199 +1,157 @@
 ---
+language:
+- tn
+- en
 library_name: transformers
-tags: []
+base_model: Davlan/afro-xlmr-large
+datasets:
+- OxxoCodes/Medupe
+- OxxoCodes/Marothodi
+- Magpie-Align/Magpie-Pro-MT-300K-v0.1
+- teknium/OpenHermes-2.5
+- CohereForAI/aya_dataset
+- lelapa/Inkuba-instruct
+- HuggingFaceTB/everyday-conversations-llama3.1-2k
+- castorini/afriberta-corpus
+- allenai/c4
 ---
+# 
 
-# Model Card for Model ID
+<img src="https://huggingface.co/OxxoCodes/Pula-8B/resolve/main/BotsLM.png" >
 
-<!-- Provide a quick summary of what the model is/does. -->
+# Pula-XLMR-Large
+## Model Information
 
+The largest encoder model in the BOTS-LM suite of language models, Pula-XLMR-Large is a highly capable encoder-style langauge model built for Setswana. Based on the Afro-XLMR-Large architecture and fine-tuned on a massive copora of Setswana and English webtext, instruction following, and synthetic data, Pula-XLMR-Large reaches competitive levels of performance compared to existing open models for Setswana and English.
 
-
-## Model Details
+The BOTS-LM suite of language models is fine-tuned on a custom-made massive corpora of Setswana and English web documents ([**Marothodi**](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)), instruction following examples and synthetic data ([**Medupe**](https://huggingface.co/datasets/OxxoCodes/Medupe/settings)), multilingual data (Aya Dataset, Inkuba Instruct), and subsets of OpenHermes 2.5, MagPie Pro MT, and The Tome. The Pula-XLMR models are trained on an additional subset of MC4 and the AfriBERTa Corpus.
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
+- **Developed by:** Nathan Brown ([@OxxoCodes](https://huggingface.co/OxxoCodes)) and Vukosi Marivate ([@vukosi](https://huggingface.co/vukosi))
+- **Funded by:** [More Information Needed]
+- **Model type:** Meta Llama 3.1
+- **Language(s) (NLP):** Setswana, English
 - **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
+- **Finetuned from model:** [Afro-XLMR-Large](https://huggingface.co/Davlan/afro-xlmr-large)
 
-<!-- Provide the basic links for the model. -->
+### Model Sources
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** https://github.com/OxxoCodes/BOTS-LM
+- **Paper:** [[2408.02239] BOTS-LM: Training Large Language Models for Setswana](https://arxiv.org/abs/2408.02239)
+- **Demo:** [More Information Needed]
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
+BOTS-LM models and data are intended for commercial and research use in Setswana and English. All Pula models are trained on predominantly instruction data, and as such are primarily meant for assistant-like chat. However, they may still function for general langauge generation, although this is predominantly untested. The BOTS-LM suite also allows for the use of its outputs to improve other models, including synthetic data generation and distillation.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Downstream Use
 
-[More Information Needed]
+BOTS-LM's Pula LLMs are trained on a wide variety of tasks including (but not limited to) general conversations, Setswana <--> English translation, writing, question answering, tool use/function calling, Named Entity Recognition (NER), and Part of Speech (POS) tagging.
 
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+The use of any model, data, or resource in the BOTS-LM suite that violates applicable laws or regulations, is intended to cause direct or indirect harm, or is otherwise generally considered immoral or unethical, is explicitly prohibited.
 
-[More Information Needed]
+While every model in the BOTS-LM suite has been trained on a small amount of data covering several langauges, use of these models is only officially supported for Setswana and English.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+As with any language model, BOTS-LM models are susceptible to generating inaccurate or biased responses to user prompts. It is recommended the user(s) of BOTS-LM models develop additional safety safeguards or perform additional safety training prior to deployment in customer- or client-facing environments.
 
-[More Information Needed]
-
-### Recommendations
+## How to Get Started with the Model
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Use the code below to get started with the model.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
 
-## How to Get Started with the Model
+tokenizer = AutoTokenizer.from_pretrained("OxxoCodes/Pula-XLMR-large")
+model = AutoModelForMaskedLM.from_pretrained("OxxoCodes/Pula-XLMR-large")
 
-Use the code below to get started with the model.
+# prepare input
+text = "Replace me by any text you'd like."
+encoded_input = tokenizer(text, return_tensors="pt")
 
-[More Information Needed]
+# forward pass
+output = model(**encoded_input)
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+Pula-XLMR was fine-tuned on a corpus of 0.5 billion tokens of Setswana and English text for 3 epochs, totaling of 1.5 Billion tokens, as measured using the Llama 3.1 tokenizer.
 
-[More Information Needed]
+To increase the occurances of high-quality instruction data, all non-OPUS data sources appear twice per epoch, while certain synthetic datasets appear five times per epoch.
 
-### Training Procedure
+We ensure 80% of training tokens are made up of SetsText, SetsText-Instruct, and Hugging Face's Everyday Conversations dataset. The other 20% consists of various high-quality and multilingual data sources.
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+Following these procedures, the final training data distribution is as follows:
 
-#### Preprocessing [optional]
+- **SetsText-Instruct:** 66.43% (336M tokens)
+  
+- **SetsText:** 13.5% (68M tokens)
+  
+- **MagPie Pro MT:** 10% (50M tokens)
+  
+- **OpenHermes 2.5:** 7% (35M tokens)
+  
+- **Aya Dataset:** 2% (10M tokens)
+  
+- **Inkuba Instruct:** 1% (5M tokens)
+  
+- **Everyday Conversations:** 0.07% (0.3M tokens)
 
-[More Information Needed]
+#### Preprocessing
 
+All Pula models in the BOTS-LM series are trained on sequences up to 4096 tokens. For all sequences longer than 4096 tokens, such as large documents or long multi-turn instruction conversations, they are split into chunks of 4096 tokens. In addition, we utilize the Liger Kernel to reduce GPU memory requirements.
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
+- **Training regime:** BF16 mixed precision
+- **Epochs:** 3.0
+- **Learning Rate:** 2e-5
+- **Learning Rate Scheduler:** Cosine
+- **Warmup Ratio:** 5%
+- **LoRA Rank:** 64
+- **LoRA Alpha:** 32
+- **Per Device Train Batch Size:** 1
+- **Gradient Accumulation Steps:** 8
+- **GPUs:** 4x Nvidia H100s 80GB
+- **Sequence Length:** 4096
+- **Effective Batch Size:** 32 (131k tokens)
+
+## Technical Specifications
 
 ### Compute Infrastructure
 
-[More Information Needed]
+Compute is provided by the Clemson University Palmetto Cluster.
 
 #### Hardware
 
-[More Information Needed]
+Pula-XLMR-Large is trained using a single NVIDIA H100 80GB GPU with sixteen CPUs and 64GB RAM.
 
 #### Software
 
-[More Information Needed]
-
-## Citation [optional]
+Training is performed using Hugging Face `transformers`, `accelerate`, `peft`, and `trl` with DeepSpeed and ZeRO-3.
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+## Citation
 
 **BibTeX:**
 
-[More Information Needed]
+```bibtext
+@misc{2408.02239,
+  Author = {Nathan Brown and Vukosi Marivate},
+  Title = {BOTS-LM: Training Large Language Models for Setswana},
+  Year = {2024},
+  Eprint = {arXiv:2408.02239},
+  Note = {Hugging Face repository: \url{https://huggingface.co/collections/OxxoCodes/bots-lm-66af1106ccc0fb38839f39da}}
+}
+```
 
 **APA:**
 
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+Brown, N., & Marivate, V. (2024). *BOTS-LM: Training Large Language Models for Setswana*. arXiv. [[2408.02239] BOTS-LM: Training Large Language Models for Setswana](https://arxiv.org/abs/2408.02239). Hugging Face repository: https://huggingface.co/collections/OxxoCodes/bots-lm-66af1106ccc0fb38839f39da