hi everyone!
text
hi everyone!
new_line
new_line
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
text
i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
new_line
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into 
text
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into 
https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
HuggingFaceFW/fineweb-edu
dataset
resource
https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
 could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
text
 could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
new_line
new_line
check it out! 
text
check it out! 
https://huggingface.co/Fizzarolli/phi3-4x4b-v1
Fizzarolli/phi3-4x4b-v1
model
resource
https://huggingface.co/Fizzarolli/phi3-4x4b-v1
hi everyone!

i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)

check it out! https://huggingface.co/Fizzarolli/phi3-4x4b-v1
👍
Hastagaras