README.md

---
license: other
license_name: hyperclovax-seed
license_link: LICENSE
---

![image/png](https://cdn-uploads.huggingface.co/production/uploads/65265ab8f8db96cffcb969dc/TSOdcOQ7qgu6ubVFMMo1R.png)

## Overview

HyperCLOVAX-SEED-Text-Instruct-1.5B is a model developed by NAVER that can understand and generate text. It demonstrates competitive performance on major benchmarks related to Korean language and culture. In addition, it supports a context length of up to 16k tokens, enabling it to handle a wide range of tasks.

## Basic Information

- Model Architecture: Transformer-based architecture (Dense Model)
- Number of Parameters: 1.5B
- Input/Output Format: Text / Text (both input and output are in text format)
- Context Length: 16k
- Knowledge Cutoff Date: The model was trained on data prior to August 2024.


## Training and Data

The training data for HyperCLOVAX-Seed-Instruct-1.5B consists of diverse sources, including high-quality datasets. The training process was carried out in four main stages: Pretraining Stage 1, where the model learns from a large volume of documents; Pretraining Stage 2, which focuses on additional training with high-quality data; Rejection sampling Fine-Tuning (RFT), aimed at enhancing the model’s knowledge across various domains and its complex reasoning abilities; and Supervised Fine-Tuning (SFT), which improves the model’s instruction-following capabilities. Furthermore, due to the characteristics of smaller models, vulnerability to long-context handling was observed. To address this, reinforcement for long-context understanding was incorporated from the pretraining stages through to the SFT stage, enabling the model to stably support context lengths of up to 16k tokens.

## Benchmark

| **Model**                      | **KMMLU (5-shot, acc)** | **HAE-RAE (5-shot, acc)** | **CLiCK (5-shot, acc)** | **KoBEST (5-shot, acc)** | 
| --------------------------------- | --------------------------- | --------------------------- | ------------------------- | -------------------------- | 
| **HyperCLOVAX-SEED-Text-Base-1.5B** | 0.4181 | 0.6370 | 0.5373 | 0.6963 |
| **HyperCLOVAX-SEED-Text-Instruct-1.5B**      | 0.3933                       | 0.5674                       | 0.4947                     | 0.6490                      |
| **Qwen2.5-1.5B-instruct**        | 0.3696                       | 0.5160                       | 0.4772                     | 0.5968                      |
| **gemma-3-1b-it**                | 0.3075                       | 0.3648                       | 0.3724                     | 0.5869                      | 


## Huggingface Usage Example

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("/path/to/ckpt")
tokenizer = AutoTokenizer.from_pretrained("/path/to/ckpt")

chat = [
  {"role": "tool_list", "content": ""},
  {"role": "system", "content": "- AI 언어모델의 이름은 \"CLOVA X\" 이며 네이버에서 만들었다.\n- 오늘은 2025년 04월 24일(목)이다."},
  {"role": "user", "content": "슈뢰딩거 방정식과 양자역학의 관계를 최대한 자세히 알려줘."},
]

inputs = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_dict=True, return_tensors="pt")
output_ids = model.generate(**inputs, max_length=1024, stop_strings=["<|endofturn|>", "<|stop|>"], tokenizer=tokenizer)
print(tokenizer.batch_decode(output_ids))
```