content.raw: Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
content.type: text
content.value: Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
content.type: new_line
content.type: new_line
content.raw: And now, everything is crumbling!
content.type: text
content.value: And now, everything is crumbling!
content.type: new_line
content.type: new_line
content.raw: The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
content.type: text
content.value: The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
content.type: new_line
content.type: new_line
content.raw: The model card in the Transformers behaves as Llama 3 and not 3.1.
content.type: text
content.value: The model card in the Transformers behaves as Llama 3 and not 3.1.
content.type: new_line
content.type: new_line
content.raw: While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
content.type: text
content.value: While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
content.type: new_line
content.type: new_line
content.raw: And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
content.type: text
content.value: And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
content.type: new_line
content.type: new_line
content.raw: But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
content.type: text
content.value: But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
content.type: new_line
content.type: new_line
content.raw: So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
content.type: text
content.value: So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
content.type: new_line
content.type: new_line
content.raw: And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
content.type: text
content.value: And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
content.type: new_line
content.type: new_line
content.raw: I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
content.type: text
content.value: I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
content.type: new_line
content.type: new_line
content.raw: I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
content.type: text
content.value: I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
content.type: new_line
content.raw: - "Think Before You Speak: Training Language Models With Pause Tokens"
content.type: text
content.value: - "Think Before You Speak: Training Language Models With Pause Tokens"
content.type: new_line
content.raw: - "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
content.type: text
content.value: - "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
content.type: new_line
content.type: new_line
content.raw: PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
content.type: text
content.value: PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
content.type: new_line
content.type: new_line
content.raw: Great YT video: 
content.type: text
content.value: Great YT video: 
content.href: https://youtu.be/Xtr_Ll_A9ms
content.raw: https://youtu.be/Xtr_Ll_A9ms
content.type: link
content.type: new_line
content.type: new_line
content.raw: Hugging Face Clem Delangue 🤗?
content.type: text
content.value: Hugging Face Clem Delangue 🤗?
content.type: new_line
content.raw: Can you please help here if possible? This will be the pinnacle of open-source!
content.type: text
content.value: Can you please help here if possible? This will be the pinnacle of open-source!
rawContent: Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
  
  And now, everything is crumbling!
  
  The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
  
  The model card in the Transformers behaves as Llama 3 and not 3.1.
  
  While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
  
  And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
  
  But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
  
  So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
  
  And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
  
  I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
  
  I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
  - "Think Before You Speak: Training Language Models With Pause Tokens"
  - "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
  
  PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
  
  Great YT video: https://youtu.be/Xtr_Ll_A9ms
  
  Hugging Face Clem Delangue 🤗?
  Can you please help here if possible? This will be the pinnacle of open-source!
reactions.reaction: 👀
reactions.users: John6666 comarproject danielus louisbrulenaudet
reactions.reaction: 😔
reactions.users: Forbu14
commentators._id: 64b795b275e4fe20e9fc5aca
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b795b275e4fe20e9fc5aca/v5AWl4XUNF54_0hyRXqBc.jpeg
commentators.fullname: Ayaan Sharif
commentators.name: Ayaan-Sharif
commentators.type: user