content.raw: hi everyone!
content.type: text
content.value: hi everyone!
content.type: new_line
content.type: new_line
content.raw: i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
content.type: text
content.value: i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
content.type: new_line
content.raw: while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into 
content.type: text
content.value: while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into 
content.raw: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
content.id: HuggingFaceFW/fineweb-edu
content.type: dataset
content.type: resource
content.url: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu
content.raw:  could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
content.type: text
content.value:  could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
content.type: new_line
content.type: new_line
content.raw: check it out! 
content.type: text
content.value: check it out! 
content.raw: https://huggingface.co/Fizzarolli/phi3-4x4b-v1
content.id: Fizzarolli/phi3-4x4b-v1
content.type: model
content.type: resource
content.url: https://huggingface.co/Fizzarolli/phi3-4x4b-v1
rawContent: hi everyone!
  
  i wanted to share an experiment i did with upcycling phi-3 mini into an moe recently.
  while benchmarks are definitely within a margin of error and they performed similarly, i think it's an interesting base to try and see if you can improve phi's performance! (maybe looking into https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu could be interesting, i also left some other notes if anyone with more compute access wants to try it themselves)
  
  check it out! https://huggingface.co/Fizzarolli/phi3-4x4b-v1
reactions.reaction: 👍
reactions.users: Hastagaras