Hey, it has been a while... I was busy participating in 💎 𝐆𝐞𝐦𝐦𝐚 𝐜𝐨𝐦𝐩𝐞𝐭𝐢𝐭𝐢𝐨𝐧!
text
Hey, it has been a while... I was busy participating in 💎 𝐆𝐞𝐦𝐦𝐚 𝐜𝐨𝐦𝐩𝐞𝐭𝐢𝐭𝐢𝐨𝐧!
new_line
new_line
Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.
text
Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.
new_line
new_line
My submission: 💎🌍🇮🇹 𝐍𝐞𝐨𝐠𝐞𝐧𝐞𝐬𝐢𝐬 - 𝐏𝐨𝐬𝐭-𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐆𝐞𝐦𝐦𝐚 𝐟𝐨𝐫 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐚𝐧𝐝 𝐛𝐞𝐲𝐨𝐧𝐝
text
My submission: 💎🌍🇮🇹 𝐍𝐞𝐨𝐠𝐞𝐧𝐞𝐬𝐢𝐬 - 𝐏𝐨𝐬𝐭-𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐆𝐞𝐦𝐦𝐚 𝐟𝐨𝐫 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐚𝐧𝐝 𝐛𝐞𝐲𝐨𝐧𝐝
new_line
📓 Kaggle notebook: 
text
📓 Kaggle notebook: 
https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond
link
new_line
new_line
In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
text
In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
new_line
I believe this method is adaptable to other languages and model sizes.
text
I believe this method is adaptable to other languages and model sizes.
new_line
new_line
𝘒𝘦𝘺 𝘚𝘵𝘦𝘱𝘴
text
𝘒𝘦𝘺 𝘚𝘵𝘦𝘱𝘴
new_line
📊 Choose reference metrics
text
📊 Choose reference metrics
new_line
🧑‍🔬 Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
text
🧑‍🔬 Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
new_line
🏋️‍♂️ Efficient Instruction Fine Tuning with Spectrum
text
🏋️‍♂️ Efficient Instruction Fine Tuning with Spectrum
new_line
🧑‍🔬 Data curation for Preference Tuning: identify existing datasets + generate synthetic data
text
🧑‍🔬 Data curation for Preference Tuning: identify existing datasets + generate synthetic data
new_line
👍👎 Efficient Direct Preference Optimization with Spectrum
text
👍👎 Efficient Direct Preference Optimization with Spectrum
new_line
📈 Evaluation
text
📈 Evaluation
new_line
new_line
new_line
🤗 Hugging Face collection (with models and datasets): 
text
🤗 Hugging Face collection (with models and datasets): 
https://huggingface.co/collections/anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
collection
resource
https://huggingface.co/collections/anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e
new_line
new_line
I'm also planning a 🎁 Gemma Giveaway (on LinkedIn - 
text
I'm also planning a 🎁 Gemma Giveaway (on LinkedIn - 
https://www.linkedin.com/in/stefano-fiorucci
https://www.linkedin.com/in/stefano-fiorucci
link
) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! 📻
text
) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! 📻
Hey, it has been a while... I was busy participating in 💎 𝐆𝐞𝐦𝐦𝐚 𝐜𝐨𝐦𝐩𝐞𝐭𝐢𝐭𝐢𝐨𝐧!

Here's the idea: Gemma open models have a large vocabulary size (256K), so improving them for a specific language or cultural context should be pretty affordable - no need for continued pre-training.

My submission: 💎🌍🇮🇹 𝐍𝐞𝐨𝐠𝐞𝐧𝐞𝐬𝐢𝐬 - 𝐏𝐨𝐬𝐭-𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐆𝐞𝐦𝐦𝐚 𝐟𝐨𝐫 𝐈𝐭𝐚𝐥𝐢𝐚𝐧 𝐚𝐧𝐝 𝐛𝐞𝐲𝐨𝐧𝐝
📓 Kaggle notebook: https://www.kaggle.com/code/anakin87/post-training-gemma-for-italian-and-beyond

In this notebook, I show how I improve the performance of Gemma 2 2B on Italian via Post-Training.
I believe this method is adaptable to other languages and model sizes.

𝘒𝘦𝘺 𝘚𝘵𝘦𝘱𝘴
📊 Choose reference metrics
🧑‍🔬 Data curation for Instruction Fine Tuning: identify existing datasets + generate synthetic data
🏋️‍♂️ Efficient Instruction Fine Tuning with Spectrum
🧑‍🔬 Data curation for Preference Tuning: identify existing datasets + generate synthetic data
👍👎 Efficient Direct Preference Optimization with Spectrum
📈 Evaluation


🤗 Hugging Face collection (with models and datasets): https://huggingface.co/collections/anakin87/gemma-neogenesis-67824b7bf13ac9cfe091fe2e

I'm also planning a 🎁 Gemma Giveaway (on LinkedIn - https://www.linkedin.com/in/stefano-fiorucci) in the next few days - sharing techniques, datasets, and models I used for my project... so stay tuned! 📻
👍
John6666