content.raw: We built a new small language model SmolLM2-MedIT-Upscale-2B, based on SmolLM2-1.7B-Instruct from Hugging Face. The premise was simple - increasing the vector in attention layers would positively impact the model's capabilities.
content.type: text
content.value: We built a new small language model SmolLM2-MedIT-Upscale-2B, based on SmolLM2-1.7B-Instruct from Hugging Face. The premise was simple - increasing the vector in attention layers would positively impact the model's capabilities.
content.type: new_line
content.type: new_line
content.raw: What did we prove?
content.type: text
content.value: What did we prove?
content.type: new_line
content.raw: In total, not much really, since we don't have the original trained under the same conditions as our upscale. However...
content.type: text
content.value: In total, not much really, since we don't have the original trained under the same conditions as our upscale. However...
content.type: new_line
content.type: new_line
content.raw: 1. We scaled up the model without losing its quality
content.type: text
content.value: 1. We scaled up the model without losing its quality
content.type: new_line
content.raw: 2. We confirmed that the method we devised works
content.type: text
content.value: 2. We confirmed that the method we devised works
content.type: new_line
content.raw: 3. After extremely short fine-tuning, the model achieved much better results in IFEval compared to the original (53.68 vs 64.29) and a higher overall average score in Open LLM Leaderboard (14.75 vs 15.17)
content.type: text
content.value: 3. After extremely short fine-tuning, the model achieved much better results in IFEval compared to the original (53.68 vs 64.29) and a higher overall average score in Open LLM Leaderboard (14.75 vs 15.17)
content.type: new_line
content.type: new_line
content.raw: I consider this a big success 😇, since surpassing the original in metrics is often very time-consuming, generates high costs, and doesn't always work out.
content.type: text
content.value: I consider this a big success 😇, since surpassing the original in metrics is often very time-consuming, generates high costs, and doesn't always work out.
content.type: new_line
content.type: new_line
content.raw: Meanwhile, we're moving forward, training SmolLM2 400M Instruct as an upscale of 136M.
content.type: text
content.value: Meanwhile, we're moving forward, training SmolLM2 400M Instruct as an upscale of 136M.
content.type: new_line
content.type: new_line
content.raw: We're curious about how increasing the base and intermediate vectors will affect the model's quality. We'll compare it to the original and the 360M Instruct version released by Hugging Face.
content.type: text
content.value: We're curious about how increasing the base and intermediate vectors will affect the model's quality. We'll compare it to the original and the 360M Instruct version released by Hugging Face.
content.type: new_line
content.type: new_line
content.raw: License: Apache 2.0​​​​​​​​​​​​​​​​
content.type: text
content.value: License: Apache 2.0​​​​​​​​​​​​​​​​
content.type: new_line
content.type: new_line
content.raw: https://huggingface.co/meditsolutions/SmolLM2-MedIT-Upscale-2B
content.id: meditsolutions/SmolLM2-MedIT-Upscale-2B
content.type: model
content.type: resource
content.url: https://huggingface.co/meditsolutions/SmolLM2-MedIT-Upscale-2B
rawContent: We built a new small language model SmolLM2-MedIT-Upscale-2B, based on SmolLM2-1.7B-Instruct from Hugging Face. The premise was simple - increasing the vector in attention layers would positively impact the model's capabilities.
  
  What did we prove?
  In total, not much really, since we don't have the original trained under the same conditions as our upscale. However...
  
  1. We scaled up the model without losing its quality
  2. We confirmed that the method we devised works
  3. After extremely short fine-tuning, the model achieved much better results in IFEval compared to the original (53.68 vs 64.29) and a higher overall average score in Open LLM Leaderboard (14.75 vs 15.17)
  
  I consider this a big success 😇, since surpassing the original in metrics is often very time-consuming, generates high costs, and doesn't always work out.
  
  Meanwhile, we're moving forward, training SmolLM2 400M Instruct as an upscale of 136M.
  
  We're curious about how increasing the base and intermediate vectors will affect the model's quality. We'll compare it to the original and the 360M Instruct version released by Hugging Face.
  
  License: Apache 2.0​​​​​​​​​​​​​​​​
  
  https://huggingface.co/meditsolutions/SmolLM2-MedIT-Upscale-2B
reactions.reaction: 🔥
reactions.users: YaTharThShaRma999 John6666 Norod78 rjmalagon