license: apache-2.0
tags:
- merge
- mergekit
- lazymergekit
- mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
- VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct
Llama-3.1-8b-instruct_4bitgs64_hqq_calib-Llama-3.1-SauerkrautLM-8b-Instruct-ties-merge
Llama-3.1-8b-instruct_4bitgs64_hqq_calib-Llama-3.1-SauerkrautLM-8b-Instruct-ties-merge is a sophisticated model resulting from the strategic merging of two powerful models: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib and VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct. This merge was accomplished using mergekit, a specialized tool that facilitates precise model blending to optimize performance and synergy between the merged architectures.
🧩 Merge Configuration
slices:
- sources:
- model: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
layer_range: [0, 31]
- model: VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct
layer_range: [0, 31]
merge_method: ties
base_model: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: float16
Model Features
This merged model combines the advanced capabilities of the HQQ quantized version of Llama-3.1-8B-Instruct with the fine-tuned prowess of the SauerkrautLM variant. The result is a versatile model that excels in both generative tasks and nuanced understanding of multilingual contexts, particularly in German and English. The integration of Spectrum Fine-Tuning from the SauerkrautLM model enhances the efficiency of the model while preserving its extensive knowledge base.
Evaluation Results
The performance of the parent models provides a solid foundation for the merged model. Here are some evaluation results from the original models:
mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
- ARC (25-shot): 60.49
- HellaSwag (10-shot): 80.16
- MMLU (5-shot): 68.98
- Average Performance: 69.51
VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct
- Fine-tuned on German-English data, showcasing significant improvements in multilingual capabilities.
These results indicate that the merged model is likely to inherit and enhance the strengths of both parent models, particularly in text generation and comprehension tasks.
Limitations
While the merged model benefits from the strengths of both parent models, it may also carry over some limitations. For instance, the potential for uncensored content remains a concern, as noted in the SauerkrautLM documentation. Additionally, the model's performance may vary depending on the specific task and the languages involved, particularly in less common languages or dialects. Users should be aware of these factors when deploying the model in real-world applications.