--- base_model: - PocketDoc/Dans-DangerousWinds-V1.1.0-12b - mergekit-community/MN-Hekate-Geneteira-17B - LatitudeGames/Wayfarer-12B - PocketDoc/Dans-SakuraKaze-V1.0.0-12b - mistralai/Mistral-Nemo-Base-2407 - elinas/Chronos-Gold-12B-1.0 - PygmalionAI/Eleusis-12B library_name: transformers tags: - mergekit - merge --- # merge This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). ## Merge Details ### Merge Method This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [mergekit-community/MN-Hekate-Geneteira-17B](https://huggingface.co/mergekit-community/MN-Hekate-Geneteira-17B) as a base. ### Models Merged The following models were included in the merge: * [PocketDoc/Dans-DangerousWinds-V1.1.0-12b](https://huggingface.co/PocketDoc/Dans-DangerousWinds-V1.1.0-12b) * [LatitudeGames/Wayfarer-12B](https://huggingface.co/LatitudeGames/Wayfarer-12B) * [PocketDoc/Dans-SakuraKaze-V1.0.0-12b](https://huggingface.co/PocketDoc/Dans-SakuraKaze-V1.0.0-12b) * [mistralai/Mistral-Nemo-Base-2407](https://huggingface.co/mistralai/Mistral-Nemo-Base-2407) * [elinas/Chronos-Gold-12B-1.0](https://huggingface.co/elinas/Chronos-Gold-12B-1.0) * [PygmalionAI/Eleusis-12B](https://huggingface.co/PygmalionAI/Eleusis-12B) ### Configuration The following YAML configuration was used to produce this model: ```yaml dtype: float32 out_dtype: bfloat16 merge_method: model_stock base_model: mergekit-community/MN-Hekate-Geneteira-17B slices: - sources: - model: mergekit-community/MN-Hekate-Geneteira-17B layer_range: [0, 20] - sources: - model: mergekit-community/MN-Hekate-Geneteira-17B layer_range: [20, 36] parameters: weight: [0.4, 0.2] - model: mistralai/Mistral-Nemo-Base-2407 layer_range: [16, 32] parameters: weight: [2, 1.5] - model: PygmalionAI/Eleusis-12B layer_range: [16, 32] - model: LatitudeGames/Wayfarer-12B layer_range: [16, 32] parameters: weight: [1.5, 1.49, 1.46, 1.4, 1.33, 1.25, 1.15, 1.05, 1] - model: PocketDoc/Dans-DangerousWinds-V1.1.0-12b layer_range: [16, 32] parameters: weight: [1.5, 1.49, 1.46, 1.4, 1.33, 1.25, 1.15, 1.05, 1] - model: elinas/Chronos-Gold-12B-1.0 layer_range: [16, 32] parameters: weight: [1.0, 1.1, 1.2, 1.29, 1.37, 1.43, 1.48, 1.5, 1.5] - model: PocketDoc/Dans-SakuraKaze-V1.0.0-12b layer_range: [16, 32] parameters: weight: [1.0, 1.1, 1.2, 1.29, 1.37, 1.43, 1.48, 1.5, 1.5] - sources: - model: mergekit-community/MN-Hekate-Geneteira-17B layer_range: [36, 56] tokenizer: source: union tokens: "[INST]": source: mergekit-community/MN-Hekate-Geneteira-17B force: true "[/INST]": source: mergekit-community/MN-Hekate-Geneteira-17B force: true "<|im_start|>": source: mergekit-community/MN-Hekate-Geneteira-17B force: true "<|im_end|>": source: mergekit-community/MN-Hekate-Geneteira-17B force: true ```