|
--- |
|
base_model: |
|
- openchat/openchat-3.5-0106 |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
license: apache-2.0 |
|
--- |
|
<p align="center"> |
|
<a href="https://ko-fi.com/pretergeek">Buy me a Ko-Fi</a> • |
|
<a href="https://patreon.com/Pretergeek">Support my work using Patreon</a> |
|
</p> |
|
|
|
# OpenChat-3.5-0106_10.7B_48Layers-Interleaved |
|
|
|
This is NOT your usual frankenmerge created using [mergekit](https://github.com/cg123/mergekit). |
|
|
|
## Merge Details |
|
### Merge Method |
|
|
|
This model was merged using the passthrough merge method, but employing the Block Expansion method described in the paper [LLaMA Pro: Progressive LLaMA with Block Expansion](https://arxiv.org/abs/2401.02415). |
|
|
|
The authors of the paper added new layers interleaved in between the original layers of the model, setting the parameters of the o_proj and down_proj layers to zero. This effectively adds layers that will just output their input (as if they were "transparent") allowing the model to remain functional even without further training. These new layers can then be targeted during training or fine-tuning without risking catastrophic forgetting, if you follow the author's training method to freeze the original layers and only train the new layers. |
|
|
|
This model has not yet received additional training, so it should perform close to the original model. |
|
|
|
### Models Merged |
|
|
|
The following models were included in the merge: |
|
* [openchat/openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) |
|
|
|
### Configuration |
|
|
|
The following YAML configuration was used to produce this model: |
|
|
|
```yaml |
|
slices: |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [0, 2] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [1, 2] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [2, 4] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [3, 4] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [4, 6] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [5, 6] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [6, 8] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [7, 8] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [8, 10] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [9, 10] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [10, 12] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [11, 12] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [12, 14] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [13, 14] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [14, 16] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [15, 16] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [16, 18] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [17, 18] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [18, 20] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [19, 20] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [20, 22] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [21, 22] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [22, 24] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [23, 24] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [24, 26] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [25, 26] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [26, 28] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [27, 28] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [28, 30] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [29, 30] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [30, 32] |
|
- sources: |
|
- model: openchat/openchat-3.5-0106 |
|
layer_range: [31, 32] |
|
parameters: |
|
scale: |
|
- filter: o_proj |
|
value: 0.0 |
|
- filter: down_proj |
|
value: 0.0 |
|
- value: 1.0 |
|
merge_method: passthrough |
|
dtype: bfloat16 |
|
|
|
``` |
|
|
|
## Citation |
|
``` |
|
@misc{wu2024llamaproprogressivellama, |
|
title={LLaMA Pro: Progressive LLaMA with Block Expansion}, |
|
author={Chengyue Wu and Yukang Gan and Yixiao Ge and Zeyu Lu and Jiahao Wang and Ye Feng and Ying Shan and Ping Luo}, |
|
year={2024}, |
|
eprint={2401.02415}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2401.02415}, |
|
} |
|
``` |
