FronyAI/frony-natural2sql-ko-v0.1.0

FronyAI sLLM for AI Agent (NL2SQL)

The sLLM accelerates inference efficiency and reduces the workload of the main LLM (e.g., LLaMA) in AI Agent workflows. This specialized sLLM delivers performance comparable to that of the main LLM. frony-natural2sql-ko-v0.1.0 is optimized to process Natural Language to SQL Query task in AI Agent workflows.

Dataset

Source	AI 허브 - 자연어 기반 질의(NL2SQL) 검색 생성 데이터
Train Samples	90,000
Valid Samples	10,000
Number of Shemas	1 Database / 95% has 1 tables and 5% has 2 tables

Training

• Base model: meta-llama/Llama-3.2-3B-Instruct
• Framework: Unsloth
• Training Configuration:

  learning Rate	1e-4
  weight Decay	1e-3
  epochs	3
  batch Size	4
  quantization	4bit
  target_modules	"q_proj", "k_proj", "v_proj", "o_proj"
  r	8
  lora_alpha	16
  lora_dropout	0.0

Prompt Template

Below prompt must be used

데이터베이스 정보를 참고하여 질문에 대한 SQL 쿼리를 작성해 주세요.

## 데이터베이스 정보
{context}

## 질문
{utterance}

Prompt example

데이터베이스 정보를 참고하여 질문에 대한 SQL 쿼리를 작성해 주세요.

## 데이터베이스 정보
{
  "tables": [
    {
      "name": "SEOUL_PUBLIC_HYGIENE_BIZ",
      "description": "서울시 기타 위생용품 제조업 현황",
      "columns": [
        {
          "name": "CGG_CODE",
          "description": "시군구코드",
          "type": "number"
        },
        ...,
        {
          "name": "SITE_ADDR_RD",
          "description": "소재지도로명",
          "type": "text"
        }
      ]
    }
  ]
}

## 질문
태창위생지의 행정동 이름이 뭐야

Inference

CPU inference with OpenVINO

Intel OpenVINO GenAI provides fast inference in CPU

# for llm inference
pip install openvino-genai
# for OV format conversion
pip install nncf
pip install git+https://github.com/huggingface/optimum-intel.git
# for supporting latest models
pip install -U transformers

# Before using OpenVINO, You should convert the huggingface model to OV model format
optimum-cli export openvino --model HF_MODEL_PATH --task text-generation-with-past --weight-format int4 OV_MODEL_PATH

# load model
pipe = ov_genai.LLMPipeline(OV_MODEL_PATH, "CPU")
tokenizer = pipe.get_tokenizer()

# create prompt
with open(PROMPT_TEMPLATE_PATH, "r") as f:
    prompt_template = f.read()
prompt = prompt_template.format(context=schema, utterance=query)
prompt = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True)
"""
|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

<|eot_id|><|start_header_id|>user<|end_header_id|>

데이터베이스 정보를 참고하여 질문에 대한 SQL 쿼리를 작성해 주세요.
## 데이터베이스 정보
{
"tables": [
{
"name": "TB_PHARMACY_OPERATE_INFO",
"description": "서울시 약국 운영시간 정보",
"columns": [
{
"name": "DUTYNAME",
"description": "약국명",
"type": "text"
},
{
  "name": "DUTYTIME1C",
  "description": "월요일 진료 마감 시간",
  "type": "number"
},
{
  "name": "DUTYTIME2C",
  "description": "화요일 진료 마감 시간",
  "type": "number"
},
... (other columns)
{
"name": "DUTYTIME1S",
"description": "월요일 진료 시작 시간",
"type": "number"
},
{
"name": "DUTYTIME2S",
"description": "화요일 진료 시작 시간",
"type": "number"
},
... (other columns)
]
}
]
}
## 질문
대한약국 월요일 진료 시작 시간 알려줘<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

# generation
generation_config = ov_genai.GenerationConfig(
    max_new_tokens=512,
    temperature=0.0,
)
pipe.generate(prompt, generation_config=generation_config)
"""
SELECT DUTYTIME1S FROM TB_PHARMACY_OPERATE_INFO WHERE DUTYNAME = '대한약국'
"""

GPU inference with transformers

# for supporting latest models
pip install -U transformers

import torch
from transformers import pipeline

model_id = "flash659/frony-natural2sql-ko-v0.1.0"
pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

with open(PROMPT_TEMPLATE_PATH, "r") as f:
    prompt_template = f.read()
schema = ...
query = ...
messages = [{"role": "user", "content": prompt_template.format(context=schema, utterance=query)}]
outputs = pipe(
    messages,
    max_new_tokens=512,
    temperature=0.0,
)
print(outputs[0]["generated_text"][-1])

Limitation

Given that all samples contain only one database and very few tables, high quality is essential for deployment in a production environment.