nielsr HF Staff commited on
Commit
a342ec1
Β·
verified Β·
1 Parent(s): 326401f

Add metadata

Browse files

This PR ensures a "Use this model" button appears at the top right.

Files changed (1) hide show
  1. README.md +61 -59
README.md CHANGED
@@ -1,59 +1,61 @@
1
- ---
2
- license: llama3.1
3
- datasets:
4
- - BAAI/Infinity-Instruct
5
- base_model:
6
- - meta-llama/Llama-3.1-8B-Instruct
7
- tags:
8
- - Instruct_Tuning
9
- ---
10
-
11
- # Shadow-FT
12
-
13
- <a href="https://arxiv.org/pdf/2505.12716"><b>[πŸ“œ Paper]</b></a> β€’
14
- <a href="https://huggingface.co/collections/taki555/shadow-ft-683288b49e1e5e1edcf03135"><b>[πŸ€— HF Models]</b></a> β€’
15
- <a href="https://github.com/wutaiqiang/Shadow-FT"><b>[🐱 GitHub]</b></a>
16
-
17
- This repo contains the weights from our paper: <a href="https://arxiv.org/pdf/2505.12716" target="_blank">Shadow-FT: Tuning Instruct via Base</a> by <a href="https://wutaiqiang.github.io" target="_blank">Taiqiang Wu*</a> <a href="https://rummyyang.github.io/" target="_blank">Runming Yang*</a>, Jiayi Li, Pengfei Hu, Ngai Wong and Yujiu Yang.
18
-
19
- \* for equal contributions.
20
-
21
-
22
-
23
- ## Overview
24
-
25
- <img src="framework.png" width="100%" />
26
-
27
- Observation:
28
-
29
- - Directly tuning the INSTRUCT (i.e., instruction tuned) models often leads to marginal improvements and even performance degeneration.
30
-
31
- - Paired BASE models, the foundation for these INSTRUCT variants, contain highly similar weight values (i.e., less than 2% on average for Llama 3.1 8B).
32
-
33
- $\Rightarrow$ We propose the Shadow-FT framework to tune the INSTRUCT models by leveraging the corresponding BASE models. The key insight is to fine-tune the BASE model, and then _directly_ graft the learned weight updates to the INSTRUCT model.
34
-
35
-
36
- ## Performance
37
-
38
- This repository contains the Llama-3.1-8B tuned on BAAI-2k subsets using Shadow-FT.
39
-
40
- <img src="performance.png" width="100%" />
41
-
42
- please refer to [our paper](https://arxiv.org/pdf/2505.12716) for details.
43
-
44
-
45
-
46
- ## β˜•οΈ Citation
47
-
48
- If you find this repository helpful, please consider citing our paper:
49
-
50
- ```
51
- @article{wu2025shadow,
52
- title={Shadow-FT: Tuning Instruct via Base},
53
- author={Wu, Taiqiang and Yang, Runming and Li, Jiayi and Hu, Pengfei and Wong, Ngai and Yang, Yujiu},
54
- journal={arXiv preprint arXiv:2505.12716},
55
- year={2025}
56
- }
57
- ```
58
-
59
- For any questions, please pull an issue or email at `[email protected]`
 
 
 
1
+ ---
2
+ license: llama3.1
3
+ datasets:
4
+ - BAAI/Infinity-Instruct
5
+ base_model:
6
+ - meta-llama/Llama-3.1-8B-Instruct
7
+ tags:
8
+ - Instruct_Tuning
9
+ library_name: transformers
10
+ pipeline_tag: text-generation
11
+ ---
12
+
13
+ # Shadow-FT
14
+
15
+ <a href="https://arxiv.org/pdf/2505.12716"><b>[πŸ“œ Paper]</b></a> β€’
16
+ <a href="https://huggingface.co/collections/taki555/shadow-ft-683288b49e1e5e1edcf03135"><b>[πŸ€— HF Models]</b></a> β€’
17
+ <a href="https://github.com/wutaiqiang/Shadow-FT"><b>[🐱 GitHub]</b></a>
18
+
19
+ This repo contains the weights from our paper: <a href="https://arxiv.org/pdf/2505.12716" target="_blank">Shadow-FT: Tuning Instruct via Base</a> by <a href="https://wutaiqiang.github.io" target="_blank">Taiqiang Wu*</a> <a href="https://rummyyang.github.io/" target="_blank">Runming Yang*</a>, Jiayi Li, Pengfei Hu, Ngai Wong and Yujiu Yang.
20
+
21
+ \* for equal contributions.
22
+
23
+
24
+
25
+ ## Overview
26
+
27
+ <img src="framework.png" width="100%" />
28
+
29
+ Observation:
30
+
31
+ - Directly tuning the INSTRUCT (i.e., instruction tuned) models often leads to marginal improvements and even performance degeneration.
32
+
33
+ - Paired BASE models, the foundation for these INSTRUCT variants, contain highly similar weight values (i.e., less than 2% on average for Llama 3.1 8B).
34
+
35
+ $\Rightarrow$ We propose the Shadow-FT framework to tune the INSTRUCT models by leveraging the corresponding BASE models. The key insight is to fine-tune the BASE model, and then _directly_ graft the learned weight updates to the INSTRUCT model.
36
+
37
+
38
+ ## Performance
39
+
40
+ This repository contains the Llama-3.1-8B tuned on BAAI-2k subsets using Shadow-FT.
41
+
42
+ <img src="performance.png" width="100%" />
43
+
44
+ please refer to [our paper](https://arxiv.org/pdf/2505.12716) for details.
45
+
46
+
47
+
48
+ ## β˜•οΈ Citation
49
+
50
+ If you find this repository helpful, please consider citing our paper:
51
+
52
+ ```
53
+ @article{wu2025shadow,
54
+ title={Shadow-FT: Tuning Instruct via Base},
55
+ author={Wu, Taiqiang and Yang, Runming and Li, Jiayi and Hu, Pengfei and Wong, Ngai and Yang, Yujiu},
56
+ journal={arXiv preprint arXiv:2505.12716},
57
+ year={2025}
58
+ }
59
+ ```
60
+
61
+ For any questions, please pull an issue or email at `[email protected]`