|
--- |
|
license: apache-2.0 |
|
tags: |
|
- Roleplay |
|
- Solar |
|
- Mistral |
|
- Text Generation |
|
--- |
|
 |
|
|
|
### Premise |
|
|
|
So this is a basic slerp merge between a smart model and a good prose model. Prose and smarts. What we all want in an uncensored RP model right? I feel like Solar has untapped potential, in any case. |
|
|
|
Sao10K's Frostwind finetune is a key component of the mixture, its smarts are impressive. NyxKrage's Frostmaid experiment, which merges Frostwind with a frankenmerge of Noromaid and a mystery medical model, delivers quite impressive prose. His model creatively incorporates long-range context and instructions too, despite being slightly incoherent due to the fraken merging. |
|
|
|
So those are the main ingredients. Thanks to Nyx for sorting out the pytorch files btw. |
|
|
|
### Recipe |
|
|
|
So, the recipe. I basically just gradient SLERP'd Frostwind into Frostmaid with these params: |
|
|
|
- filter: self_attn |
|
value: [0.9, 0.6, 0.3, 0, 0] |
|
- filter: mlp |
|
value: [0.3, 0.6] |
|
- value: 0.5 # fallback for rest of tensors |
|
|
|
|
|
### Tentative Dozen or So Test Conclusion |
|
|
|
This made a model that was actually pretty much everything I was looking for - NEARLY as smart as Frostwind but with MOST of Frostmaids punchy prose. I tried doing TIES merges and DARE ties merges, but they actually came out worse, because both models have major weaknesses - one is very dry and gpt-ish, the other is a little loose with what's going on. The ties merges tended to bring out those qualities, dare even worse. So I stuck with this. It's not AS smart as Frostwind, so you maybe have to regen a little, but it's pretty smart, and quite creative. A sweet spot hopefully. Maybe someone merge wiser than I can do more with this recipe, but I'm very pleased with it, it did what I was hoping for - a smaller model I can mobile dgpu and produces pretty outsized quality responses (it's fairly zealous tho, be warned). I've only played with it a TINY bit, so there may be qualities or flaws I've missed. |
|
|
|
Cheers to all the finetuners, mergers and developers without which open source models wouldn't be half of what they are. |
|
|
|
Resources used: |
|
|
|
https://huggingface.co/NyxKrage/FrostMaid-10.7B-TESTING-pt |
|
|
|
https://huggingface.co/Sao10K/Frostwind-10.7B-v1 |
|
|
|
https://github.com/cg123/mergekit/tree/main |
