I found this paper to be thought-provoking: "Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling" by Bansal, Hosseini, Agarwal,  Tran, and Kazemi.
text
I found this paper to be thought-provoking: "Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling" by Bansal, Hosseini, Agarwal,  Tran, and Kazemi.
new_line
https://arxiv.org/abs/2408.16737
https://arxiv.org/abs/2408.16737
link
new_line
The direct implication is that smaller models could be used to create cost-effective synthetic datasets. And on that note, in the Gemma terms of use, Google explicitly claims no rights on outputs generated from those models, which means one is free to synthgen from the Gemma line. Meta's Llama 3 licence forbids synthetic generation of outputs if used to improve other models. Relevant Mistral, Qwen, and Yi models under the Apache 2.0 license are unrestricted for this purpose.
text
The direct implication is that smaller models could be used to create cost-effective synthetic datasets. And on that note, in the Gemma terms of use, Google explicitly claims no rights on outputs generated from those models, which means one is free to synthgen from the Gemma line. Meta's Llama 3 licence forbids synthetic generation of outputs if used to improve other models. Relevant Mistral, Qwen, and Yi models under the Apache 2.0 license are unrestricted for this purpose.
I found this paper to be thought-provoking: "Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling" by Bansal, Hosseini, Agarwal,  Tran, and Kazemi.
https://arxiv.org/abs/2408.16737
The direct implication is that smaller models could be used to create cost-effective synthetic datasets. And on that note, in the Gemma terms of use, Google explicitly claims no rights on outputs generated from those models, which means one is free to synthgen from the Gemma line. Meta's Llama 3 licence forbids synthetic generation of outputs if used to improve other models. Relevant Mistral, Qwen, and Yi models under the Apache 2.0 license are unrestricted for this purpose.
👀
John6666 anakin87 AtAndDev tachyon-beep tommulder victor gghfez louisbrulenaudet djuna
👍
trollek AymaneElfirdo ajibawa-2023
🔥
aaditya tommulder
630f3e4002ce39336c411048
https://cdn-avatars.huggingface.co/v1/production/uploads/630f3e4002ce39336c411048/H281TzWT-Uxo-Owk_HlXC.png
alkinun
AtAndDev
user
65acc58c14d782df067f759b
/avatars/52a153d04d325469e1be69bce610ebe5.svg
Tan Hong Kai
ecyht2
user