upload model, tokenizer and readme

README.md

@@ -1,3 +1,85 @@
 ---
+library_name: transformers
+pipeline_tag: translation
+tags:
+- transformers
+- translation
+- pytorch
+- russian
+- kazakh
+
 license: apache-2.0
+language:
+- ru
+- kk
 ---
+
+# KazRush-ru-kk
+
+KazRush-ru-kk is a translation model for translating from Russian to Kazakh.
+
+## Usage
+
+Using the model requires some packages to be installed.
+
+```bash
+pip install numpy==1.26.4 torch~=2.2.2 transformers~=4.39.2 sentencepiece~=0.2.0
+```
+
+After installing necessary dependencies the model can be run with the following code:  
+
+```python
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+import torch
+
+model = AutoModelForSeq2SeqLM.from_pretrained('deepvk/KazRush-ru-kk')
+tokenizer = AutoTokenizer.from_pretrained('deepvk/KazRush-ru-kk')
+
+def generate(text, **kwargs):
+    inputs = tokenizer(text, return_tensors='pt')
+    with torch.no_grad():
+        hypotheses = model.generate(**inputs, num_beams=5, **kwargs)
+    return tokenizer.decode(hypotheses[0], skip_special_tokens=True)
+
+print(generate("Как Кока-Кола может помочь автомобилисту?"))
+```
+
+## Data
+
+This model was trained on the following data (Russian-Kazakh language pairs):  
+[OPUS Corpora](<https://opus.nlpl.eu/results/ru&kk/corpus-result-table>)  
+[kazparc](<https://huggingface.co/datasets/issai/kazparc>)  
+[wmt19 dataset](<https://statmt.org/wmt19/translation-task.html#download>)
+
+Preprocessing of the data included:
+- deduplication;
+- removing trash symbols, special tags, multiple whitespaces etc. from texts;
+- removing texts that were not in Russian or Kazakh (language detection was made via [fasttext](<https://huggingface.co/facebook/fasttext-language-identification>));
+- removing pairs that had low alingment score (comparison was performed via [LaBSE](<https://huggingface.co/sentence-transformers/LaBSE>));
+- filtering the data using [opusfilter](<https://github.com/Helsinki-NLP/OpusFilter>) tools.
+
+## Experiments
+
+Current model was compared to another open-source translation model, NLLB. We compared our model to all version of nllb, excluding nllb-moe-54b due to its size.
+The metrics - BLEU, chrF and COMET - were calculated on `devtest` part of [FLORES+ evaluation benchmark](<https://github.com/openlanguagedata/flores>), most recent evaluation benchmark for multilingual machine translation.
+
+| Model  | Size | BLEU | chrF | COMET |
+|-----------------------------------------|-------|-----------------------------|------------------------|
+| [nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) | 600M   | 13.8  |  48.2  | 0.8684  |
+| [nllb-200-1.3B](https://huggingface.co/facebook/nllb-200-1.3B)                   | 1.3B   | 14.8 | 50.1  | 0.8819  |
+| [nllb-200-distilled-1.3B](https://huggingface.co/facebook/nllb-200-distilled-1.3B)   | 1.3B    | 15.2 | 50.2 | 0.8843   |
+| [nllb-200-3.3B](https://huggingface.co/facebook/nllb-200-3.3B)                    | 3.3B    | 15.6 | 50.7  | 0.8891    |
+| [our model (kzgqkn0f)]()                             | 196 M    | **16.2**  | **51.9**   |   0.8836   |
+
+## Examples of usage:  
+
+```
+print(generate("Каждый охотник желает знать, где сидит фазан."))
+# Әр аңшы қырғауылдың қайда отырғанын білгісі келеді.
+
+print(generate("Местным продуктом-специалитетом с защищённым географическим наименованием по происхождению считается люнебургский степной барашек."))
+# Шығу тегі бойынша қорғалған географиялық атауы бар жергілікті өнім-маман болып люнебург далалық барақ есептеледі.
+
+print(generate("Помогите мне закадрить девушку"))
+# Қызды бауыздауға көмектесіңіз.
+```

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 4096,
+  "d_kv": 64,
+  "d_model": 512,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "initializer_factor": 0.05,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "num_decoder_layers": 12,
+  "num_heads": 8,
+  "num_layers": 12,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.39.3",
+  "use_cache": true,
+  "vocab_size": 8000
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "decoder_start_token_id": 0,
+  "early_stopping": true,
+  "eos_token_id": 1,
+  "max_new_tokens": 127,
+  "num_beams": 3,
+  "pad_token_id": 0,
+  "transformers_version": "4.39.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c76de487e8e2a6007287b2ebc33b86729cf1bcf3d574edca1f78058cedcd178e
+size 787905368

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:377110f5fc7b44c69c6edbf6b0f2638fdb4398a6a942535cb323ca3b41cb99b0
+size 401220

tokenizer_config.json

@@ -0,0 +1,39 @@
+{
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "extra_ids": 0,
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "T5Tokenizer",
+  "unk_token": "<unk>"
+}