Qwen2.5
Collection
11 items
•
Updated
Qwen2.5-1.5B-Instruct
This version of Qwen2.5-1.5B-Instruct has been converted to run on the Axera NPU using w8a16 and w4a16 quantization.
This model has been optimized with the following LoRA:
Compatible with Pulsar2 version: 4.1
Feature
- Support for longer contexts, in this sample it's 2.5k
- Support context dialogue
- System prompt kvcache is supported
Convert tools links:
For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int8
Pulsar2 Link, How to Convert LLM from Huggingface to axmodel
AXera NPU AXEngine LLM Runtime
Convert script
The follow show how to convert Qwen2.5-1.5B-Instruct-GPTQ-Int8
pulsar2 llm_build --input_path Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int8 \
--output_path Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int8-ctx-ax650 \
--hidden_state_type bf16 --kv_cache_len 2047 --prefill_len 128 \
--last_kv_cache_len 128 \
--last_kv_cache_len 256 \
--last_kv_cache_len 384 \
--last_kv_cache_len 512 \
--last_kv_cache_len 640 \
--last_kv_cache_len 768 \
--last_kv_cache_len 896 \
--last_kv_cache_len 1024 \
--chip AX650 -c 1 --parallel 8
Support Platform
- AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card
- AX630C
- TBD
| Chips | w8a16 | w4a16 | DDR | Flash |
|---|---|---|---|---|
| AX650 | 12 tokens/sec | 17 tokens/sec | 2.3GB | 2.3GB |
How to use
Download all files from this repository to the device
root@ax650:/mnt/qtang/llm-test/Qwen2.5-1.5B-Instruct# tree -L 1
.
├── main_api
├── main_ax650
├── main_axcl_aarch64
├── main_axcl_x86
├── post_config.json
├── qwen2.5-1.5b-ctx-ax650
├── qwen2.5-1.5b-ctx-int4-ax650
├── qwen2.5_tokenizer
├── qwen2.5_tokenizer_uid.py
├── run_qwen2.5_1.5b_ctx_ax650_api.sh
├── run_qwen2.5_1.5b_ctx_ax650.sh
├── run_qwen2.5_1.5b_ctx_axcl_aarch64.sh
├── run_qwen2.5_1.5b_ctx_axcl_x86.sh
└── run_qwen2.5_1.5b_ctx_int4_ax650.sh
Start the Tokenizer service
root@ax650:/mnt/qtang/llm-test/Qwen2.5-1.5B-Instruct# python qwen2.5_tokenizer_uid.py
Server running at http://0.0.0.0:12345
System prompt cache
- The System prompt can be preset through the configuration file from
--system_prompt - The System prompt can be cached in the form of kv cache to a specified folder for quick loading at the next run time from
--kvcache_path - This folder needs to be created manually before running, for example
mkdir kvcache
root@ax650:/mnt/qtang/llm-test/qwen2.5-1.5b-ctx# cat run_qwen2.5_1.5b_ctx_ax650.sh
./main_ax650 \
--template_filename_axmodel "qwen2.5-1.5b-ctx-ax650/qwen2_p128_l%d_together.axmodel" \
--axmodel_num 28 \
--tokenizer_type 2 \
--url_tokenizer_model "http://0.0.0.0:12345" \
--filename_post_axmodel "qwen2.5-1.5b-ctx-ax650/qwen2_post.axmodel" \
--filename_tokens_embed "qwen2.5-1.5b-ctx-ax650/model.embed_tokens.weight.bfloat16.bin" \
--tokens_embed_num 151936 \
--tokens_embed_size 1536 \
--use_mmap_load_embed 1 \
--live_print 1
#--system_prompt "你的名字叫小智（allen）,你是一个人畜无害的AI助手。深圳市今天（4月1日）阴天，愚人节，气温在14°C至19°C之间，微风。" \
#--kvcache_path "./kvcache" \
Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board
Open another terminal and run run_qwen2.5_1.5b_ctx_ax650.sh
root@ax650:/mnt/qtang/llm-test/qwen2.5-1.5b-ctx# ./run_qwen2.5_1.5b_ctx_ax650.sh
[I][ Init][ 110]: LLM init start
[I][ Init][ 34]: connect http://0.0.0.0:12345 ok
[I][ Init][ 57]: uid: 1d0fadb4-1aa1-44d2-9587-e27badcd2ebf
bos_id: -1, eos_id: 151645
3% | ██ | 1 / 31 [4.80s<148.95s, 0.21 count/s] tokenizer init ok
[I][ Init][ 26]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ | 31 / 31 [24.90s<24.90s, 1.24 count/s] init post axmodel ok,remain_cmm(7477 MB)
[I][ Init][ 188]: max_token_len : 2047
[I][ Init][ 193]: kv_cache_size : 256, kv_cache_num: 2047
[I][ Init][ 201]: prefill_token_num : 128
[I][ Init][ 205]: grp: 1, prefill_max_token_num : 1
[I][ Init][ 205]: grp: 2, prefill_max_token_num : 128
[I][ Init][ 205]: grp: 3, prefill_max_token_num : 256
[I][ Init][ 205]: grp: 4, prefill_max_token_num : 384
[I][ Init][ 205]: grp: 5, prefill_max_token_num : 512
[I][ Init][ 205]: grp: 6, prefill_max_token_num : 640
[I][ Init][ 205]: grp: 7, prefill_max_token_num : 768
[I][ Init][ 205]: grp: 8, prefill_max_token_num : 896
[I][ Init][ 205]: grp: 9, prefill_max_token_num : 1024
[I][ Init][ 209]: prefill_max_token_num : 1024
[I][ load_config][ 282]: load config:
{
"enable_repetition_penalty": false,
"enable_temperature": false,
"enable_top_k_sampling": false,
"enable_top_p_sampling": false,
"penalty_window": 20,
"repetition_penalty": 1.2,
"temperature": 0.9,
"top_k": 10,
"top_p": 0.8
}
[I][ Init][ 218]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
[I][ GenerateKVCachePrefill][ 271]: input token num : 21, prefill_split_num : 1 prefill_grpid : 2
[I][ GenerateKVCachePrefill][ 308]: input_num_token:21
[I][ main][ 230]: precompute_len: 21
[I][ main][ 231]: system_prompt:
prompt >> who are you?
[I][ SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:21 input_num_token:12
[I][ SetKVCache][ 534]: current prefill_max_token_num:896
[I][ Run][ 660]: input token num : 12, prefill_split_num : 1
[I][ Run][ 686]: input_num_token:12
[I][ Run][ 829]: ttft: 306.20 ms
I am Qwen, a large language model created by Alibaba Cloud. I am here to assist you with your questions and provide helpful information. How may I assist you today?
[N][ Run][ 943]: hit eos,avg 12.20 token/s
[I][ GetKVCache][ 500]: precompute_len:68, remaining:956
prompt >> q
root@ax650:/mnt/qtang/llm-test/qwen2.5-1.5b-ctx#
- Downloads last month
- 40
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support