ๅบๆผ lianghsun/Llama-3.2-Taiwan-3B ๆจกๅ๏ผ้้ไธญ่ฏๆฐๅๅฐ็ฃๆณๅพๆขๆๅๅคๆฑบๆธ็ญ็ธ้่ณๆ้้ฒ่กๅพฎ่ชฟใ
| Update Date | Model Version | Key Changes |
|---|---|---|
| 2024-11-13 | v2024.11.13 | Fine-tuned version v2024.11.13 of lianghsun/Llama-3.2-Taiwan-3B. This instruction version initiates experimental integration of non-Chinese instructions to enhance the modelโs robustness and mitigate risks of overfitting. |
| 2024-11-06 | v2024.11.6 | Starting with this release, fine-tuning is based on the foundation model lianghsun/Llama-3.2-Taiwan-3B v2024.10.27 , and versioning has been updated to use the YYYY-mm-dd format. |
| 2024-10-17 | v1.1.0 (v2024.10.17) | (Model collapsed ๐ฅ) Experimental fine-tuning on v1.0.0 with added legal code data from the Republic of China (Taiwan) |
| 2024-10-10 | v1.0.0 (v2024.10.10) | Full model training completed, but missing legal code data for the Republic of China (Taiwan) |
| 2024-09-27 | v0.1.0 (v2024.09.27) | Model v0.1.0 released, but training was interrupted after 3 epochs due to lack of compute resources |
ๅบๆผ lianghsun/Llama-3.2-Taiwan-3B ๆจกๅ๏ผๆญคๅพฎ่ชฟ้็จไฝฟ็จไบไพ่ชไธญ่ฏๆฐๅๅฐ็ฃ็ๆณๅพๆขๆ่็ธ้ๅคๆฑบๆธ่ณๆ้๏ผไปฅๆๅๆจกๅๅจๆณๅพ้ ๅ็ๅฐๆฅญ็ฅ่ญ่ๆ็จ่ฝๅใ้ไบ่ณๆ้ๆถต่ไบๆณๅพๆขๆ็็ตๆงใๅคๆฑบๆธ็ๆ ผๅผ๏ผๆณๅบญไธๅธธ่ฆ็ๆณๅพ่ช่จ่่ก่ช๏ผไธฆๅ ๅซไบ้จๅๆณๅพ่ณๆ็งๅญธไปปๅ็ๆ็จ๏ผไฝฟๆจกๅ่ฝๅค ๆดๆบ็ขบๅฐ็่งฃๅ่็่ๅฐ็ฃๆณๅพ้ซ็ณป็ธ้็ๅ้กใ็ถ้้ไบๅพฎ่ชฟ๏ผๆจกๅๅฐ่ฝๅค ๆดๅฅฝๅฐ็บๆณๅพๅฐๆฅญไบบๅฃซๆไพๅนซๅฉ๏ผไธฆๅจๅฐ็ฃๆณๅถๆกๆถๅ งๆไพๆด็ฒพๆบ็ๅๆ่ๅปบ่ญฐใ
ๆญคๆจกๅๅฏไปฅ็ดๆฅ็จๆผ็่งฃๅ็ๆ็น้ซไธญๆๆณๅพๆๆฌ๏ผ้ฉๅ้่ฆ่็ๅฐ็ฃๆณๅพ็ธ้ๅ้ก็ๆ็จๅ ดๆฏใๆจกๅ้ ่จญ็ๆไปคๅๅๆ่ฝๅค ๆๆๆไพๆณๅพ่ณ่จใ้ๆธ ๆณๅพๆขๆใไธฆ็ๆ็ฌฆๅๆณๅพๅฐๆฅญ็ๅๆใๅ ถ็ดๆฅไฝฟ็จ็ฏๅๅ ๆฌไฝไธ้ๆผๆณๅพ่ณ่จๆฅ่ฉขใๆณๅพๆๆฌๆ่ฆใๅๅบๆฌ็ๆณๆขๅฐ่ฉฑใ
็ถ้ๅพฎ่ชฟๅพ๏ผ่ฉฒๆจกๅๅฏ็จๆผๆดๅ ท้ซ็ๆณๅพไปปๅ๏ผๅฆ่ชๅๅคๆฑบๆธๅๆใๆณๅพๅฏฆ้ซ่ญๅฅ๏ผNER๏ผใๆณ่ฆ็ทจ่่ฝๆ๏ผไปฅๅๆณๅพๅ่ฆๅฏฉๆฅ่ผๅฉใๆญคๆจกๅๅฏไปฅ็ก็ธซ้ๆ่ณๆณๅพๆธๆ็งๅญธๆ็จๆๆณๅพๆ่ก๏ผLegalTech๏ผ็ณป็ตฑไธญ๏ผๅนซๅฉๆณๅพๅฐๆฅญไบบๅฃซๆไผๆฅญๆๅๅทฅไฝๆ็ใ
่ฉฒๆจกๅไธฆไธ้ฉ็จๆผ้ๆณๅพ็ธ้้ ๅ็็ๆไปปๅ๏ผไธไธๆ็จๆผ้ฒ่กๅฏ่ฝๆถๅ่ชคๅฐๆ้ฏ่ชค็ๆณๅพๅปบ่ญฐ๏ผๅฐคๅ ถๆฏๅจๆช็ถๅฐๆฅญๅฏฉๆฅ็ๆ ๆณไธใ้ฟๅ ๅฐๆจกๅ็จๆผๆช็ถๆๆฌๆ้ๆณ็จ้๏ผๅฆ็ๆๅ ท็ญ่ญฐๆงๆๅ ทๅ่ฆ็ๆณๅพๅปบ่ญฐใ
ๆจกๅๅจ็ๆๆณๅพๆขๆๅๅคๆฑบๆธๅ งๅฎนๆ๏ผๅฏ่ฝๆ็ๆ่ๆงๆไธๅญๅจ็ๆณๆขๆๅคๆฑบๆธๅ งๅฎน๏ผ้ๆฏๆจกๅ็ๅ งๅจ้ๅถไนไธใไฝฟ็จ่ ๅจๅ่้ไบ่ณๆๆ๏ผๆ่ฌนๆ ๆชขๆฅ็ๆ็ๅ งๅฎน๏ผไธฆ้ฟๅ ๅฐๆจกๅ่ผธๅบ่ฆ็บๆณๅพไพๆใๅปบ่ญฐๅจๅฏฆ้ๆ็จไธญ๏ผๅฐๆจกๅ็ๆ็็ตๆ่ๅฏ้ ็ๆณๅพ่ฆ่งฃๅไพๆบ้ฒ่กๆฏๅฐ๏ผ็ขบไฟๆบ็ขบๆงใๅๆณๆงๅ้ฉ็จๆงใ
ๆญคๆจกๅ้็ถ็ถ้ๆณๅพๆๆฌ็ๅพฎ่ชฟ๏ผไฝๅจๆผๆณๅพๆๆฌ็ๆธ้ๅๅบ็คๆจกๅ็บ SLM๏ผๆจกๅ่ฝๅไปๆๆฅต้๏ผไฝฟ็จ่ ๆๆณจๆไปฅไธ้ขจ้ช่้ๅถ๏ผ
ๅ่ฆ้ขจ้ช๏ผ ๆจกๅๅฏ่ฝๆๅๆ ๅ ถ่จ็ทด่ณๆไธญ็ๆฝๅจๅ่ฆใ็ฑๆผๆณๅพๆๆฌ็็นๅฎๆง๏ผๆจกๅๅฏ่ฝๆด็ๆๆไบๆณ่ฆใๆขๆๆๅคๆฑบๆกไพ๏ผ่ๅจๅ ถไป้ ๅ่กจ็พ่ผๅผฑใ็นๅฅๆฏๅจ่็ไธๅธธ่ฆ็ๆณๅพๅ้กๆๆช่ขซ่จ็ทด้็ๆฐๆณ่ฆๆ๏ผๆจกๅ็่ผธๅบๅฏ่ฝๆๅธถๆๅ่ฆใ
ๆ่ก้ๅถ๏ผ ้็ถๆจกๅ่ฝๅค ่็ๅคง้จๅ็ๆณๅพๆๆฌ๏ผไฝๅฐๆผ็ตๆงๆฅตๅ ถ่ค้ๆ่ช่จๆจกๆฃฑๅ ฉๅฏ็ๆณๅพๆขๆ๏ผๆจกๅๅฏ่ฝ็กๆณ็ข็็ฒพ็ขบ็ๅ็ญใไฝฟ็จ่ ๆ้ฟๅ ๅฎๅ จไพ่ณดๆจกๅ็่ผธๅบ๏ผๅฐคๅ ถๅจๆณๅพๆฑบ็ญ้็จไธญ๏ผๅปบ่ญฐ้ฒ่ก้กๅค็ๅฐๆฅญๆชขๆฅใ
ๆณๅพ่ฒฌไปป๏ผ ๆจกๅไธฆ้ๅฐๆฅญๆณๅพ้กงๅ๏ผๅ ๆญคๅ ถ็ๆ็ๅๆไธๆ่ขซ่ฆ็บๆญฃ็ขบ็ๆณๅพๅปบ่ญฐใไฝฟ็จ่ ๆ็ขบไฟๅจ็ๆงไธๅฐๆฅญ่ๆฏไธ้ฒ่กๆจกๅ็ๆ็จ๏ผไธฆ้ฟๅ ๅจ้้ตๆฑบ็ญไธญ้ๅบฆไพ่ณดๆจกๅใ
่ชค็จ้ขจ้ช๏ผ ไธ็ถไฝฟ็จๆจกๅ้ฒ่ก้ฏ่ชคๆ่ชคๅฐๆง็ๆณๅพๅปบ่ญฐ๏ผๅฏ่ฝๅฐๅไบบๆไผๆฅญ้ ๆ่ฒ ้ขๅฝฑ้ฟใไฝฟ็จ่ ๆ่ฌนๆ ๆ็จๆจกๅๆผๅ่ฆๆๆณๅพ็ธ้ไปปๅไธญ๏ผไธฆไฟๆๅฐๅ ถ่ผธๅบ็ๆชข่ฆๅๆ กๆญฃใ
็บไบๆธๅฐ้ไบ้ขจ้ช๏ผๅปบ่ญฐไฝฟ็จ่ ๅจๆ็จๆจกๅ่ผธๅบๆ้ฒ่กไบๆฌกๆชขๆฅ๏ผ็นๅฅๆฏๅจๆถๅๆณๅพๆฑบ็ญ็ๆ ๅขไธญใๆฌๆจกๅ็พ้ๆฎต็บๆไพๆณๅพ็งๆ้ ๅ้ฒ่กๅคง่ช่จๆจกๅ็ ็ฉถ๏ผไธฆ้ๅไปฃๅฐๆฅญๆณๅพๅทฅไฝ่ ไนๅฐๆฅญๅปบ่ญฐใ
่ฆไฝฟ็จ vLLM Docker image ไพๅๅๆญคๆจกๅ๏ผๆจๅฏไปฅๆ็ งไปฅไธๆไฝ๏ผ
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct
่ซๆณจๆ๏ผๅฆๆๆณ่ฆไฝฟ็จไธๅ็ๆฌ็ checkpoint๏ผ่ซๅ ไธ --revision <tag_name> ๏ผ
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=<secret>" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model lianghsun/Llama-3.2-Taiwan-Legal-3B-Instruct --revision <tag_name>
ๆญคๆจกๅๅจ v2024.11.06 ็ๆฌๅพๆนๆก็จ lianghsun/Llama-3.2-Taiwan-3B ๅ็บๅบ็คๆจกๅ๏ผfoundation model๏ผใ Tokenizer ไป่ๆๅๅ ็ meta-llama/Llama-3.2-3B ็ธๅ๏ผๅจๆชไพ็ๆฌๅ่ฆๆณๆดๅ ไธญๆๅญใ
The following hyperparameters were used during training:
Note: ..(WIP)..
Note: ..(WIP)..
Note: ..(WIP)..
Note: ..(WIP)..
Note: ..(WIP)..
Note: ..(WIP)..
Note: ..(WIP)..
Note: ..(WIP)..
ๆฌๆจกๅๅบๆผ lianghsun/Llama-3.2-Taiwan-3B ๏ผไฝฟ็จ่ชๅๆญธ Transformer ๆถๆง้ฒ่ก่ช่จๅปบๆจกใ่ฉฒๆจกๅ็ไธป่ฆ็ฎๆจๆฏๆๅๅฐๅฐ็ฃๆณๅพๆๆฌ็็่งฃ่็ๆ่ฝๅ๏ผๅฐคๅ ถๆฏ้ๅฐๅคๆฑบๆธใๆณๆข็ๅฐๆฅญ่็่ๆ็จใ้้ๅฐ้่จญ่จ็ๆณๅพๆๆฌ้้ฒ่กๅพฎ่ชฟ๏ผๆจกๅ่ฝๆด็ฒพ็ขบๅฐๅ็ญๆณๅพๅ้กไธฆๆไพ็ธ้ๅปบ่ญฐใ
็กใ
็กใ
ๅ็ฎกๆๅๅทฒๆบๅไบ่จฑๅค้ๆผไธญ่ฏๆฐๅๅฐ็ฃๆณๅพ้ ๅ็่ณๆ้๏ผไฝ็ฑๆผ็ฎๅ่ณๆบๆ้๏ผ็กๆณๅฐๆๆ่ณๆ้้ฒ่กๅฎๆด่จ็ทด๏ผๆฏ็๏ผๆๅไธฆๆฒๆๅฐๅ จ้จ่ณๆ้้ฝ้ฒ่ก่จ็ทด๏ผๅ ๅๅบ่ขซ่ช็บๆๅบ็ค็ๆณๅพๆๆฌ๏ผ๏ผๅฐ่ดๆจกๅๅฐๆช้ๅฐๆไฝณ่กจ็พใๅ ๆญค๏ผ็ฎๅ็ checkpoint ๆฏๅบๆผๆ้่ณๆบ็็ๆฌใๅฆๆๆจๆๆ้ก่ดๅฉ็ฎๅ๏ผๆญก่ฟ่ๆ่ฏ็นซใๆ็ธไฟก่ฅ่ฝๅฐๆดๅคๅทฒๆบๅไฝๅฐๆช็ดๅ ฅ่จ็ทด็ๆณๅพ่ชๆ้ฒ่กๅพฎ่ชฟ๏ผ่ฉฒๆจกๅๅฐ่ฝ้ๅฐ็น้ซไธญๆๆณๅพ้ ๅ็ๆไฝณ่กจ็พใ
ๆญคๆจกๅๅฆๆ้ฒไธๆญฅ่ณๆบ๏ผๅฐๆไธๅฎๆๆดๆฐ๏ผๆ้ๆจกๅๆๆฐๆถๆฏ่ซ่ฆ Model Update History ็ซ ็ฏใ