Base version

#1
by lazyDataScientist - opened

Would it be possible to have a finetune of your model without instruction finetuning?

All the models, including the "base" model in the MOE are instruct.
All would have to be tuned from base/chat Llama 3.2 -> then re-"MOEed" so to speak.

Is this what you mean?
or do you mean without "gating" in the MOE ?

For the latter; the "reg" version would work.
Please clarify if I am off base here ;

Sign up or log in to comment