Update README.md

README.md

@@ -8,6 +8,9 @@ language:
 - id
 - th
 - zh
+tags:
+- multilingual
+- sea
 ---
 <p align="center">
   <img src="seal_logo.png" width="200" />
@@ -53,7 +56,7 @@ The following sections summarize the [Pre-training](#pre-training), [Supervised-
 ## Pre-training
 
 ### Vocabulary Expansion
-Like many English/Latin-dominant LLMs, Llama-2's BPE tokenizer breaks non-European and non-Latin linguistic texts into unsustainably long byte-level sequences that cover much shorter semantic meanings, leading to [degraded performance](https://arxiv.org/abs/2306.11372). For instance, it takes 4.3x more tokens to encode the same sentence in Thai compared to that in English. This leads to the models failing to perform summarization and comprehension tasks without exceeding the context length.
+Like many English/Latin-dominant LLMs, Llama-2's BPE tokenizer breaks non-European and non-Latin linguistic texts into unsustainably long byte-level sequences that cover much shorter semantic meanings, leading to [degraded performance](https://arxiv.org/abs/2306.11372). For instance, it takes 4.3x more tokens to encode the same sentence in Thai compared to that in English (see below table). This leads to the models failing to perform tasks requiring long context modeling (e.g., summarization and comprehension tasks) without exceeding the context length.
 
 Our goal for vocabulary expansion is threefold: (1) the number of newly-added tokens must be minimal and only cover the new languages, (2) the tokens should bring the compression ratios of new languages close to that of English, and (3) minimize the disruption of existing European tokens to preserve Llama-2 knowledge. In the end, we obtain **~11K** new tokens for Vi, Id, Th, and Zh to augment the original 32000-token vocabulary. Details of our expansion technique will be revealed in our upcoming technical report.
 
@@ -71,7 +74,7 @@ As seen in the table below, our new vocabulary reduces the compression ratio fro
 ### Pre-training Data
 
 The pre-training dataset of SeaLLMs is formed by the documents from diverse public sources, including web texts (e.g., [Common Crawl](https://commoncrawl.org/)), 
-news documents (e.g., [CC-News](https://huggingface.co/datasets/cc_news)), academic articles and the texts with expert knowledge (e.g., wikipedia). 
+news documents (e.g., [CC-News](https://huggingface.co/datasets/cc_news)), academic articles, and texts with expert knowledge (e.g., Wikipedia). 
 We firstly employ [FastText language indentifier](https://huggingface.co/facebook/fasttext-language-identification) to filter out the documents that do not belong to Thai, Vietnamese or Indonesian. 
 To further remove harmful or undesirable content, we develop a pipeline with various data cleaning and filtering modules to preprocess the collected data. 
 Meanwhile, to maintain the English performance of SeaLLMs, we also introduce a set of high-quality English texts sampled from [RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data) into pre-training. 
@@ -88,7 +91,7 @@ We pre-train our SeaLLM-base in ~4 weeks on 32gpus, clocking ~150B tokens. We us
 
 ### SFT Data
 
-Our supervised finetuning (SFT) data consists of many categories. The largest of them are public and open-source, such as [OpenORCA](https://huggingface.co/datasets/Open-Orca/OpenOrca) and [Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus). As the aforementioned are monolingual, we employ several established or novel automatic techniques to gather more instruction data for SEA languages. 
+Our supervised finetuning (SFT) data consists of many categories. The largest of them are public and open-source, such as [OpenORCA](https://huggingface.co/datasets/Open-Orca/OpenOrca) and [Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus). As the aforementioned are English only, we employ several established or novel automatic techniques to gather more instruction data for SEA languages. 
 
 Even more noteworthy is that we engaged native speakers to collect a small number of queries used by SEA-language native speakers in natural settings, which helps in adaptation to the local cultural customs, norms, and laws. We also collect country-relevant safety data that cover many culturally and legally sensitive topics in each of these SEA countries - such data tend to be ignored, or may even appear in conflict with Western safety data. Therefore, we believe that our models are more local-friendly and abide by local rules to a higher degree.
 
@@ -103,8 +106,12 @@ We conduct SFT with a relatively balanced mix of SFT data from different categor
 
 One of the most reliable ways to compare chatbot models is peer comparison. 
 With the help of native speakers, we built an instruction test set that focuses on various aspects expected in a user-facing chatbot, namely:
-(1) NLP tasks (e.g. translation & comprehension), (2) Reasoning, (3) Instruction-following and 
-(4) Natural and Informal questions. The test set also covers all languages that we are concerned with.
+(1) task-solving (e.g. translation & comprehension), 
+(2) math-reasoning (e.g., math and logical reasoning questions), 
+(3) general-instruction (e.g., instructions in general domains),
+(4) natural-questions (e.g., questions about local context often written informally), and
+(5) safety-related questions.
+The test set also covers all languages that we are concerned with.
 We use GPT-4 as an evaluator to rate the comparison between our models versus ChatGPT-3.5 and other baselines.
 
 Compared with [PolyLM-13b-chat](https://arxiv.org/pdf/2307.06018.pdf), a recent multilingual model, our model significantly outperforms across all languages and categories.
@@ -117,7 +124,7 @@ Compared with [PolyLM-13b-chat](https://arxiv.org/pdf/2307.06018.pdf), a recent
 
 Compared with Llama-2-13b-chat, our SeaLLM-13b performs significantly better in all SEA languages, 
 despite the fact that Llama-2 was already trained on a decent data amount of Vi, Id, and Th.
-In english, our model is 46% as good as Llama-2-13b-chat, even though it did not undergo complex human-labor intensive RLHF.
+In English, our model is 46% as good as Llama-2-13b-chat, even though it did not undergo complex human-labor intensive RLHF.
 
 
 
@@ -127,7 +134,7 @@ In english, our model is 46% as good as Llama-2-13b-chat, even though it did not
 </div>
 
 Compared with ChatGPT-3.5, our SeaLLM-13b model is performing 45% as good as ChatGPT for Thai. 
-For important aspects such as Safety and Task-Solving, our model nearly on par with ChatGPT across the languages.
+For important aspects such as Safety and Task-Solving, our model is nearly on par with ChatGPT across the languages.
 
 
 <div class="row" style="display: flex; clear: both;">
@@ -156,7 +163,7 @@ Notably, for Thai - a seemingly low-resource language, our model is just 1% behi
 
 ### MMLU - Preserving English-based knowledge
 
-On the 5-shot [MMLU](https://arxiv.org/abs/2009.03300), our SeaLLM models not only preserve but also slightly outperform 13B LLama-2 and Llama-2-chat, despite the fact that optimizing for this English and Chinese dominant test set is not part of our goal.
+On the 5-shot [MMLU](https://arxiv.org/abs/2009.03300), our SeaLLM models not only preserve but also slightly outperform 13B LLama-2 and Llama-2-chat, despite the fact that optimizing for this English dominant test set is not part of our goal.
 
 | MMLU (Acc) | STEM | Humanities | Social | Others | Average
 |-----------| ------- | ------- |  ------- | ------- | ------- | 
@@ -186,7 +193,7 @@ As shown in the table below, the 1-shot reading comprehension performance is sig
 
 For translation tasks, we evaluate our models with the [FloRes-200](https://github.com/facebookresearch/flores/blob/main/flores200/README.md) using [chrF++](https://aclanthology.org/W15-3049/) scores in 4-shot settings.
 
-Similarly observed, our SeaLLM models outperform Llama-2 significantly in the new languages.
+Similarly observed, our SeaLLM model outperforms Llama-2 significantly in the new languages.
 
 
 | FloRes-200 (chrF++) | En-Zh | En-Vi | En-Id | En-Th | En->X | Zh-En | Vi-En | Id-En | Th-En | X->En
@@ -204,7 +211,7 @@ Our models are also performing competitively with ChatGPT for translation betwee
 
 #### Summarization
 
-Lastly, in 2-shot [XL-sum summarization tasks](https://aclanthology.org/2021.findings-acl.413/), our models also achieve a better performance, with substantial gains in Thai.
+Lastly, in 2-shot [XL-sum summarization tasks](https://aclanthology.org/2021.findings-acl.413/), our model also achieves better performance, with substantial gains in Thai.
 
 | XL-Sum (rouge-L) | En | Zh | Vi | Id | Th
 |-------- | ---- | ---- |  ---- | ---- | ---- |
@@ -214,7 +221,7 @@ Lastly, in 2-shot [XL-sum summarization tasks](https://aclanthology.org/2021.fin
 
 ## Acknowledge our linguists
 
-We would like to express our special thanks to our professional and native linguists, who helped build, evaluate and fact-check our supervised finetuning (SFT) dataset as well as evaluating our models across different aspects, especially safety.
+We would like to express our special thanks to our professional and native linguists, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.
 
 ## Citation
 
@@ -228,4 +235,4 @@ If you find our project useful, hope you can star our repo and cite our work as
   title = {SeaLLMs - Large Language Models for Southeast Asia},
   year = 2023,
 }
-```
+```