Hugging Face's logo

shuvom
/
my-mixtral-2x7B

Text Generation
Transformers
Safetensors
mixtral
Mixture of Experts
Merge
mergekit
lazymergekit
mistralai/Mistral-7B-Instruct-v0.2
OpenPipe/mistral-ft-optimized-1218
conversational
text-generation-inference
Model card Files Community
1

Question

#1
by streamerbtw1002 - opened
Discussion
streamerbtw1002

I have a question: How did you combine parts of other pre-trained models into MoE? I'm trying to find a way how.

shuvom
 Owner
•
edited Jan 19, 2024

using merge kit,

and it is just the first or weak MOE, further new and strong MOEs are coming.

shuvom changed discussion status to closed
streamerbtw1002

@shuvom Thanks, I am gonna train a 3b AI model and make them into 4 experts.

streamerbtw1002 changed discussion status to open
shuvom
 Owner

great!

Sign up or log in to comment