content.raw: Yesterday we just released Qwen1.5. Maybe someday I can tell more about the experience. But this is is at least a good release even if it is not yet SOTA. There is not so many SOTA by the way. This time, we actually fixed a lot of problems. 
content.type: text
content.value: Yesterday we just released Qwen1.5. Maybe someday I can tell more about the experience. But this is is at least a good release even if it is not yet SOTA. There is not so many SOTA by the way. This time, we actually fixed a lot of problems. 
content.type: new_line
content.type: new_line
content.raw: 1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).
content.type: text
content.value: 1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).
content.type: new_line
content.type: new_line
content.raw: 2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.
content.type: text
content.value: 2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.
content.type: new_line
content.type: new_line
content.raw: 3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better. 
content.type: text
content.value: 3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better. 
content.type: new_line
content.type: new_line
content.raw: 4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice  performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.
content.type: text
content.value: 4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice  performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.
content.type: new_line
content.type: new_line
content.raw: 5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too. 
content.type: text
content.value: 5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too. 
content.type: new_line
content.type: new_line
rawContent: Yesterday we just released Qwen1.5. Maybe someday I can tell more about the experience. But this is is at least a good release even if it is not yet SOTA. There is not so many SOTA by the way. This time, we actually fixed a lot of problems. 
  
  1. Context lengths are finally unified for all sizes. Previously, a lot of users kept telling us that 14B only supports 2K (Yeah even dynamic NTK does not work that well and it can only be extended to around 4-5K. Let alone those know nothing about how to use dynamic NTK).
  
  2. If you carefully use our base language models, you will find that they understand special tokens of ChatML, which means that you can directly use LoRA to train on data with ChatML format. Why you can't do this before? This is because if the base language model does not understand the special tokens, you need to make them trained, which means that you should turn on the training of embedding. This is disgusting and it often leads to problems when you use ZeRO3.
  
  3. We did strengthen our base language models except for 72. You should find better base language models, especially for 7 and 14. Why not 72? Nah, hard to say, but will make it better. 
  
  4. About the multilingual capabilities. Yes we finally build up our multilingual evaluation system and find out that our new base language models have nice  performance in multilingual evaluation for base language models. This tells us that we should pay more attention to the post-training with multilingual data. And we did that too. This is why this time we tell you something about multilingual performance. It is for sure much much better than our models before this release.
  
  5. Chat models are the most promising stuff. Before this release, we gave you the SFT models. But this time, we had very nice SFT+DPO models. Yeah not only annotators like them but also users like them. I am sure you developers will feel that way too. 
  
  
reactions.reaction: ❤️
reactions.users: s3nh osanseviero Pclanglais andysalerno hf-delta huodon KnutJaegersberg cin-hubert xu3kev hiyouga merve Stopwolf Dlbk ShushengYang lewtun Ji-Ha xianbao euclaise smerchi AdinaY Sangmin philschmid joaogante victor nateraw dlimeng bowiehsu jkeisling HDiffusion EditaZ
reactions.reaction: 🤯
reactions.users: osanseviero merve xianbao AdinaY philschmid
reactions.reaction: 👍
reactions.users: krumeto xianbao AdinaY philschmid
commentators._id: 64ce091a9e9ca8123d7a42b0
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ce091a9e9ca8123d7a42b0/OEPggp82RwigxNLL35LgT.jpeg
commentators.fullname: Pierre-Carl Langlais
commentators.name: Pclanglais
commentators.type: user
commentators._id: 61caeda441f9432649f03ab6
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61caeda441f9432649f03ab6/IazJBCi7Cr34DgZXZeI4k.jpeg
commentators.fullname: s3nh
commentators.name: s3nh
commentators.type: user
commentators._id: 63732ebbbd81fae2b3aaf3fb
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669551186189-63732ebbbd81fae2b3aaf3fb.jpeg
commentators.fullname: Knut Jägersberg
commentators.name: KnutJaegersberg
commentators.type: user
commentators._id: 620760a26e3b7210c2ff1943
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
commentators.fullname: Junyang Lin
commentators.name: JustinLin610
commentators.type: user
commentators._id: 64137e2150358a805203cbac
commentators.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64137e2150358a805203cbac/w9RQx8Q07UvgFyIZ3ce_k.jpeg
commentators.fullname: Jade
commentators.name: euclaise
commentators.type: user