Body: <p>Is there research on why models mixes work? One would expect that averaging the weights of two models would produce garbage, but many models mixes created by amateurs show that they not only work, but also produce results that, for example, interpolate between two image styles when two image models with different styles are mixed. In mixes of language models, the interpolation is harder to see, but mixing the models also seems to work, and is used by people trying to improve model performance by mixing good models.</p>
  <p>If you want to see examples, visit <a href="https://civitai.com/models" rel="nofollow noreferrer">Civitai</a> and set the filters (in the upper right corner) to &quot;type=checkpoint&quot; and &quot;checkpoint_type=merge&quot; and you can find, e.g., models that merge weights for anime style with weights for realistic style and get a semi-realistic style as a result.</p>
  <p>Here is the <a href="https://github.com/hako-mikan/sd-webui-supermerger?tab=readme-ov-file#merge-models" rel="nofollow noreferrer">documentation</a> of one of the softwares that allow to merge stable diffusion checkpoints.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-22T20:43:57.810Z
Comments.Text: if you mean neural networks as models, and "mixed models" as a linear combination of their weights, pretty sure this is not the case
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-22T20:45:37.620Z
Comments.Text: @Alberto People, for example, average the weights of two stable diffusion models and get out another working model. You can see a lot of examples on civitai, that show how well it works. I don't think any of these people know why it works, but they know how to use it in a way that it produces high-quality models.
Comments.UserDisplayName: allo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-22T20:49:30.913Z
Comments.Text: can you please link to a blogpost/page/whatever where they talk about this? usually this works for latent spaces, not for model weights
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-22T20:55:52.363Z
Comments.Text: @Alberto I added two links to the question.
Comments.UserDisplayName: allo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T06:47:10.717Z
Comments.Text: Diffusion models are essentially Markovian hierarchical VAEs so model merges are nothing but hierarchical latent spaces averaging which certainly can make sense.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T23:02:33.360Z
Comments.Text: https://arxiv.org/abs/2305.03053 this is the closest thing i can find, but it's far from a linear combination of weights
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T23:06:42.920Z
Comments.Text: @cinch DDPM are Markovian, definitely not hierarchical, and for sure are not VAE. No matter what kind of model you are dealing with, given a nonconvex loss, and two solutions on different local minima, there is FOR SURE a linear combination of the two that have a higher loss than the worst of the two solution
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-23T23:09:20.940Z
Comments.Text: At most, we can argue that merging _latent spaces_ will lead to mixed solutions, and this was already been observed many years ago with Word2Vec
Comments.UserDisplayName: Alberto
Tags: <neural-networks><generative-model><large-language-models><diffusion-models>
Title: Why do mix models work?
UserDisplayName: allo
ThreadText: # Why do mix models work?
  
  (Asked by: username_0 on 2024-02-22)
  
  Is there research on why models mixes work? One would expect that averaging the weights of two models would produce garbage, but many models mixes created by amateurs show that they not only work, but also produce results that, for example, interpolate between two image styles when two image models with different styles are mixed. In mixes of language models, the interpolation is harder to see, but mixing the models also seems to work, and is used by people trying to improve model performance by mixing good models.
  
  If you want to see examples, visit [Civitai](https://civitai.com/models) and set the filters (in the upper right corner) to "type=checkpoint" and "checkpoint\_type=merge" and you can find, e.g., models that merge weights for anime style with weights for realistic style and get a semi-realistic style as a result.
  
  Here is the [documentation](https://github.com/hako-mikan/sd-webui-supermerger?tab=readme-ov-file#merge-models) of one of the softwares that allow to merge stable diffusion checkpoints.
  
  **Tags:** neural-networks, generative-model, large-language-models, diffusion-models
  > **Comments:**
  > **username_1**: if you mean neural networks as models, and "mixed models" as a linear combination of their weights, pretty sure this is not the case  
  > **username_0**: @Alberto People, for example, average the weights of two stable diffusion models and get out another working model. You can see a lot of examples on civitai, that show how well it works. I don't think any of these people know why it works, but they know how to use it in a way that it produces high-quality models.  
  > **username_1**: can you please link to a blogpost/page/whatever where they talk about this? usually this works for latent spaces, not for model weights  
  > **username_0**: @Alberto I added two links to the question.  
  > **username_2**: Diffusion models are essentially Markovian hierarchical VAEs so model merges are nothing but hierarchical latent spaces averaging which certainly can make sense.  
  > **username_1**: https://arxiv.org/abs/2305.03053 this is the closest thing i can find, but it's far from a linear combination of weights  
  > **username_1**: @cinch DDPM are Markovian, definitely not hierarchical, and for sure are not VAE. No matter what kind of model you are dealing with, given a nonconvex loss, and two solutions on different local minima, there is FOR SURE a linear combination of the two that have a higher loss than the worst of the two solution  
  > **username_1**: At most, we can argue that merging _latent spaces_ will lead to mixed solutions, and this was already been observed many years ago with Word2Vec  
  
  ---
  
  ## 0 answers
  
  