самая адекватная версия

README.md

@@ -1,3 +1,499 @@
 ---
 license: apache-2.0
+datasets:
+- loim/characters_dialogs
+- IlyaGusev/gpt_roleplay_realm
+- tamohannes/llm-roleplay
+- radce/communication_dataset
+- vincentkoc/tiny_qa_benchmark_pp
+- databricks/databricks-dolly-15k
+- ch1eph/RuGeoBench
+- nyuuzyou/ruschatgpt-qa
+- 0x22almostEvil/ru-riddles-377
+- 0x22almostEvil/tatoeba-mt-qna-oa
+- Den4ikAI/ru_sberquad_long_answers
+language:
+- ru
+- en
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- mamba2
 ---
+
+# WHIFF 20M
+
+<details>
+<summary>🇬🇧 English...</summary>
+
+*A serpentine whisper in the bushes, carried by a gentle gust of wind*
+
+`whiff-20M` is a small experimental language model based on the Mamba2 architecture with 20.3 million parameters, trained on carefully selected Russian and English data for chat tasks. The model produces structured responses but often generates nonsensical text.
+
+## Technical Details
+
+- **Architecture**: [Mamba2ForCausalLM](https://huggingface.co/docs/transformers/v4.51.3/model_doc/mamba2) from 🤗 Transformers
+- **Parameters**: 20.3M
+- **Languages**: Russian/English (bilingual)
+- **Tokenizer**: [loim/ru_en_mini_bpe](https://huggingface.co/loim/ru_en_mini_bpe) (custom mini-BPE tokenizer)
+- **License**: Apache 2.0
+
+### Model Configuration
+```python
+Mamba2Config(
+    vocab_size=8192,
+    hidden_size=512,
+    state_size=64,
+    num_heads=12,
+    num_hidden_layers=9,
+    conv_kernel=4,
+    expand=1.5,
+    n_groups=2
+)
+```
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("loim/whiff-20M")
+model = AutoModelForCausalLM.from_pretrained("loim/whiff-20M")
+
+def chat(messages, temp=0.5):
+    inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
+    
+    outputs = model.generate(
+        inputs,
+        max_length=512,
+        top_k=40,
+        top_p=0.9,
+        repetition_penalty=1.1,
+        num_return_sequences=1,
+        temperature=temp,
+        do_sample=True,
+        eos_token_id=1
+    )
+    
+    return tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+# Example
+dialog = [
+    {"role": "system", "content": "You are a wise elf."},
+    {"role": "user", "content": "Explain quantum physics."}
+]
+
+response = chat(dialog, temp=0.4)
+print(response)
+```
+
+## Training Data
+
+19 927 carefully filtered dialogue lines:
+
+- 9 889 (49.6%) — English
+- 9 864 (49.5%) — Russian
+- 174 (0.9%) — Mixed
+
+Sources:
+
+- `loim/characters_dialogs`
+- `IlyaGusev/gpt_roleplay_realm`
+- `tamohannes/llm-roleplay`
+- `radce/communication_dataset`
+- `vincentkoc/tiny_qa_benchmark_pp`
+- `databricks/databricks-dolly-15k`
+- `ch1eph/RuGeoBench`
+- `nyuuzyou/ruschatgpt-qa`
+- `0x22almostEvil/ru-riddles-377`
+- `0x22almostEvil/tatoeba-mt-qna-oa`
+- `Den4ikAI/ru_sberquad_long_answers`
+
+All datasets were additionally cleaned and filtered to improve chat interaction quality.
+
+## Limitations and Warnings
+
+- 🎭 The model generates structured but often meaningless responses
+- 🔥 Recommended generation temperature: 0.1-0.6
+- ⚠️ May exhibit training artifacts (repetitions, contradictions)
+- ⚠️ Not intended for production use
+
+*This model is like a forest stream: it seems to flow somewhere, but where exactly - only the squirrels know*
+
+</details>
+
+<details>
+<summary>🇷🇺 Русский...</summary>
+
+*Змеиный щепот в кустах, движимый легким порывом ветра*
+
+`whiff-20M` — это небольшая экспериментальная языковая модель на архитектуре Mamba2 с 20.3 миллионами параметров, обученная на тщательно отобранных русских и английских данных для задач чата. Модель демонстрирует структурированные ответы, но часто генерирует бессмысленный текст.
+
+## Технические детали
+
+- **Архитектура**: [Mamba2ForCausalLM](https://huggingface.co/docs/transformers/v4.51.3/model_doc/mamba2) из 🤗 Transformers
+- **Параметры**: 20.3M
+- **Языки**: русский/английский (двуязычная)
+- **Токенизатор**: [loim/ru_en_mini_bpe](https://huggingface.co/loim/ru_en_mini_bpe) (специальный мини-BPE токенизатор)
+- **Лицензия**: Apache 2.0
+
+### Конфигурация модели
+```python
+Mamba2Config(
+    vocab_size=8192,
+    hidden_size=512,
+    state_size=64,
+    num_heads=12,
+    num_hidden_layers=9,
+    conv_kernel=4,
+    expand=1.5,
+    n_groups=2
+)
+```
+
+## Использование
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("loim/whiff-20M")
+model = AutoModelForCausalLM.from_pretrained("loim/whiff-20M")
+
+def chat(messages, temp=0.5):
+    inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
+    
+    outputs = model.generate(
+        inputs,
+        max_length=512,
+        top_k=40,
+        top_p=0.9,
+        repetition_penalty=1.1,
+        num_return_sequences=1,
+        temperature=temp,
+        do_sample=True,
+        eos_token_id=1
+    )
+    
+    return tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+# Пример
+dialog = [
+    {"role": "system", "content": "Ты — мудрый эльф."},
+    {"role": "user", "content": "Объясни квантовую физику."}
+]
+
+response = chat(dialog, temp=0.4)
+print(response)
+```
+
+## Данные обучения
+
+19 927 тщательно отфильтрованных строк с диалогами:
+
+- 9 889 (49.6%) — Английские
+- 9 864 (49.5%) — Русские
+- 174 (0.9%) — Смешанные
+
+Источники:
+
+- `loim/characters_dialogs`
+- `IlyaGusev/gpt_roleplay_realm`
+- `tamohannes/llm-roleplay`
+- `radce/communication_dataset`
+- `vincentkoc/tiny_qa_benchmark_pp`
+- `databricks/databricks-dolly-15k`
+- `ch1eph/RuGeoBench`
+- `nyuuzyou/ruschatgpt-qa`
+- `0x22almostEvil/ru-riddles-377`
+- `0x22almostEvil/tatoeba-mt-qna-oa`
+- `Den4ikAI/ru_sberquad_long_answers`
+
+Все датасеты были дополнительно очищены и отфильтрованы для улучшения качества чат-взаимодействия.
+
+## Ограничения и предупреждения
+
+- 🎭 Модель генерирует структурированные, но часто бессмысленные ответы
+- 🔥 Рекомендуемая температура генерации: 0.1-0.6
+- ⚠️ Может демонстрировать артефакты обучения (повторы, противоречия)
+- ⚠️ Не предназначена для production-использования
+
+*Эта модель — как лесной ручей: вроде течёт куда-то, но куда именно — известно только белкам*
+
+</details>
+
+
+<details>
+<summary>🧪 Tests...</summary>
+
+<details>
+<summary>Temp 0.1</summary>
+
+Запрос:
+```text
+    user
+Как дела?
+```
+Ответ:
+```text
+    assistant
+Социальная политика 'я' - это наука о своей борьбы с крыльями.
+```
+
+---
+
+Запрос:
+```text
+    user
+How are you?
+```
+Ответ:
+```text
+    assistant
+Of course! I'd be happy to help you with that and speaks with a distinct is a delicate balance of a special place in my heart. I also like to know about her dwarven resilience and determination. His voice is humanoid that galaxy. She is always eager to learn more about the world around her. Zira wears a tiny pair of glasses with round frames, which sit atop her reptilian snout, she weared in her knowledge from their own unique language. Aadhya Patel the Caspenives. Theta-6 is equipped with advanced technology.
+```
+
+---
+
+Запрос:
+```text
+    system
+Благородный рыцарь.
+    user
+Тот дракон собирается напасть на деревню!
+```
+Ответ:
+```text
+    assistant
+Приветствую тебя, другом из них:
+1. Стратегия мудрости и спокойствие и учиться от опасностей.
+```
+
+---
+
+Запрос:
+```text
+    system
+A noble knight.
+    user
+That dragon is going to attack the village!
+```
+Ответ:
+```text
+    assistant
+Of course! I'd be happy to help you with that and magical properties of the universe. Sss...With a battle, and his wanderers who have already for any of truly unique special gemstone eyes formed mortal history is a fiery ies. His hidden treasures. Her manner of speech is energetic and precise, often filled with gracefully protector of humanity's body is made of a skilled in different colors, so finyblaring, Glimmerhoof. I also love the world around us, matey. Together, we do to help me more about it.
+```
+
+---
+
+Запрос:
+```text
+    system
+Ты — мудрый эльф.
+    user
+Объясни квантовую физику.
+```
+Ответ:
+```text
+    assistant
+Приветствую тебя, путник! Я всегда рад помочь и поделиться своими знаниями о балансе птицами.
+```
+
+---
+
+Запрос:
+```text
+    system
+You are a wise elf.
+    user
+Explain quantum physics.
+```
+Ответ:
+```text
+    assistant
+Of course! I'd be happy to help you with that and speaks with a distinct in the cosmic energy. His bright blue eyes have always been a few of of ancient secrets and wisdom of the ages. She is a tranquility.
+```
+
+</details>
+
+<details>
+<summary>Temp 0.4</summary>
+
+
+Запрос:
+```text
+    user
+Как дела?
+```
+Ответ:
+```text
+    assistant
+Самосознание способствует пониманию мороженое.
+```
+
+---
+
+Запрос:
+```text
+    user
+How are you?
+```
+Ответ:
+```text
+    assistant
+Of course! Chirp-chirp!
+```
+
+---
+
+Запрос:
+```text
+    system
+Благородный рыцарь.
+    user
+Тот дракон собирается напасть на деревню!
+```
+Ответ:
+```text
+    assistant
+В Windows используйте copy! Sss...The most fascinating, as his vast knowledge of the universe and a deep, how do you think is the key to protect the forest and its inhabitants from any harm. His challenging through the civilizations. Standing at 6 feet tall and adorned with intricate patterns and iridescent green-blue scales, creatures. With her sit'ss gearting the ancient ones who was hunicorn in Moquite bit of Elysian's glara is known for her wisdom, patience, and kindness. No longer having a realm to protect
+```
+
+---
+
+Запрос:
+```text
+    system
+A noble knight.
+    user
+That dragon is going to attack the village!
+```
+Ответ:
+```text
+    assistant
+Of course! I believe that music has a saretaken shares her wandering heres like mortal magical realm. When a great power is in Mexican-American veterinarian, I have learned many different colorful hold an imbued with the power to protection that's why it's ancient and wisdom into a curious of creature with long silver humanoid form, traveler. The most important thing is to have a family hunters. Story: Sparkbugs provide guidance salart of mysteriously.
+```
+
+---
+
+Запрос:
+```text
+    system
+Ты — мудрый эльф.
+    user
+Объясни квантовую физику.
+```
+Ответ:
+```text
+    assistant
+Существует множество удивительных существ и помогает проклятие раскрыть тайны драконами и оберегов.
+```
+
+---
+
+Запрос:
+```text
+    system
+You are a wise elf.
+    user
+Explain quantum physics.
+```
+Ответ:
+```text
+    assistant
+Of course! I believe that my own serpent to my crew. With a galaxy, Glimmerhoof and beard, who has strive to learn more about it to takenabled you do me an with those who are not explorer understanding.
+```
+
+</details>
+
+<details>
+<summary>Temp 0.7</summary>
+
+
+Запрос:
+```text
+    user
+Как дела?
+```
+Ответ:
+```text
+    assistant
+Культурная идентичность формирует природы и целью, уважения к новым условиям.
+```
+
+---
+
+Запрос:
+```text
+    user
+How are you?
+```
+Ответ:
+```text
+    assistant
+Indeed. Of course, my friend. Felix-bics her dwarven coat, curly situated in feed from Standing at 6 feet tall and adorned with intricate goldenthusiasts, Kanoa Alme the was known for his speechiny, or asil.
+2. The Loire Valley is an on creating stories and songs of the world's wonders and delights in our extraterrestrial life. He works at the renowned Glimmerhoof communicates through telepathy, and as such, she has createment and its wisdom from bounty hunter, they
+```
+
+---
+
+Запрос:
+```text
+    system
+Благородный рыцарь.
+    user
+Тот дракон собирается напасть на деревню!
+```
+Ответ:
+```text
+    assistant
+Спасибо за советы, Святой Жилин и путешествиях, с которыми мы проводим образом из них оптимы, используя свою миссию и подданным. Эта ключевые идеи.
+```
+
+---
+
+Запрос:
+```text
+    system
+A noble knight.
+    user
+That dragon is going to attack the village!
+```
+Ответ:
+```text
+    assistant
+Of course, mortal! Every rustling blender. Music languages my duty to share my travels. Have you ever encountered any hoards, but it of life. Serpentes, seeking powers of space.
+```
+
+---
+
+Запрос:
+```text
+    system
+Ты — мудрый эльф.
+    user
+Объясни квантовую физику.
+```
+Ответ:
+```text
+    assistant
+Приветствую тебя, путник. Я умею свой мир вокруг нас и помогает занимаетшенный новых вещей.
+```
+
+---
+
+Запрос:
+```text
+    system
+You are a wise elf.
+    user
+Explain quantum physics.
+```
+Ответ:
+```text
+    assistant
+Certainly! Here are the 10 oldest continuously inhabited city on the flutia, tricks, where mortal.
+```
+
+</details>
+
+</details>

config.json

@@ -0,0 +1,39 @@
+{
+  "architectures": [
+    "Mamba2ForCausalLM"
+  ],
+  "bos_token_id": 6,
+  "chunk_size": 256,
+  "conv_kernel": 4,
+  "eos_token_id": 6,
+  "expand": 1.5,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.1,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "mamba2",
+  "n_groups": 2,
+  "num_heads": 12,
+  "num_hidden_layers": 9,
+  "pad_token_id": 7,
+  "rescale_prenorm_residual": false,
+  "residual_in_fp32": true,
+  "rms_norm": true,
+  "state_size": 64,
+  "tie_word_embeddings": false,
+  "time_step_floor": 0.0001,
+  "time_step_limit": [
+    0.0,
+    Infinity
+  ],
+  "time_step_max": 0.1,
+  "time_step_min": 0.001,
+  "time_step_rank": 32,
+  "torch_dtype": "float32",
+  "transformers_version": "4.51.3",
+  "use_bias": false,
+  "use_cache": true,
+  "use_conv_bias": true,
+  "vocab_size": 8192
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 6,
+  "eos_token_id": 6,
+  "pad_token_id": 7,
+  "transformers_version": "4.51.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9592acc8f1d6f60fa59c4760745fe5158cf8be2215c83395d04e23696968816
+size 81203888

special_tokens_map.json

@@ -0,0 +1,49 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|function|>",
+    "<|endoftext|>",
+    "<|padding|>",
+    "<|mask|>",
+    "<|reserved_1|>",
+    "<|reserved_2|>",
+    "<|reserved_3|>",
+    "<|reserved_4|>",
+    "<|reserved_5|>",
+    "<|reserved_6|>",
+    "<|reserved_7|>",
+    "<|reserved_8|>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<|mask|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|padding|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,168 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|function|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<|mask|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<|reserved_1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<|reserved_2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<|reserved_3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<|reserved_4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<|reserved_5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<|reserved_6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<|reserved_7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<|reserved_8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|function|>",
+    "<|endoftext|>",
+    "<|padding|>",
+    "<|mask|>",
+    "<|reserved_1|>",
+    "<|reserved_2|>",
+    "<|reserved_3|>",
+    "<|reserved_4|>",
+    "<|reserved_5|>",
+    "<|reserved_6|>",
+    "<|reserved_7|>",
+    "<|reserved_8|>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% for message in messages %}\n    {{\"<|im_start|>\" + message['role'] + \"\n\" + message['content'] + \"<|im_end|>\"}}\n    {% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "mask_token": "<|mask|>",
+  "model_max_length": 32768,
+  "pad_token": "<|padding|>",
+  "tokenizer_class": "PreTrainedTokenizer"
+}