"How expensive is it actually to teach a #LanguageModel German through #finetuning 💰💰💰? We get asked this quite often.
text
"How expensive is it actually to teach a #LanguageModel German through #finetuning 💰💰💰? We get asked this quite often.
new_line
new_line
There is no one-size-fits-all answer to this question, as among other factors:
text
There is no one-size-fits-all answer to this question, as among other factors:
new_line
⏹ each fine-tuning is different,
text
⏹ each fine-tuning is different,
new_line
⏹ the hardware used can be a major cost driver,
text
⏹ the hardware used can be a major cost driver,
new_line
⏹ the amount and type of training data can extend the process,
text
⏹ the amount and type of training data can extend the process,
new_line
⏹ and the skills to be trained can increase the difficulty of fine-tuning.
text
⏹ and the skills to be trained can increase the difficulty of fine-tuning.
new_line
new_line
However, we have broken down the costs incurred for our latest fine-tune ( 
text
However, we have broken down the costs incurred for our latest fine-tune ( 
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
VAGOsolutions/SauerkrautLM-Qwen-32b
model
resource
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
)
text
)
new_line
new_line
new_line
Base model: 
text
Base model: 
https://huggingface.co/Qwen/Qwen1.5-32B
Qwen/Qwen1.5-32B
model
resource
https://huggingface.co/Qwen/Qwen1.5-32B
new_line
Fine-Tuning Goal: Train German language
text
Fine-Tuning Goal: Train German language
new_line
Training dataset size: 160,000 SFT data / 110,000 DPO data
text
Training dataset size: 160,000 SFT data / 110,000 DPO data
new_line
Training duration: 72.5 hours (2 epochs SFT / 1 epoch DPO)
text
Training duration: 72.5 hours (2 epochs SFT / 1 epoch DPO)
new_line
GPU: 2x A100 SXM
text
GPU: 2x A100 SXM
new_line
New model: 
text
New model: 
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
VAGOsolutions/SauerkrautLM-Qwen-32b
model
resource
https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b
new_line
new_line
Total cost: 312 euros 💶
text
Total cost: 312 euros 💶
new_line
new_line
These are quite reasonable training costs considering the model now speaks passable German (previously very broken). Depending on the use case and process requirements, this can even be a real alternative to the costly continuous pre-training of foreign language models.
text
These are quite reasonable training costs considering the model now speaks passable German (previously very broken). Depending on the use case and process requirements, this can even be a real alternative to the costly continuous pre-training of foreign language models.
"How expensive is it actually to teach a #LanguageModel German through #finetuning 💰💰💰? We get asked this quite often.

There is no one-size-fits-all answer to this question, as among other factors:
⏹ each fine-tuning is different,
⏹ the hardware used can be a major cost driver,
⏹ the amount and type of training data can extend the process,
⏹ and the skills to be trained can increase the difficulty of fine-tuning.

However, we have broken down the costs incurred for our latest fine-tune ( https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b)


Base model: https://huggingface.co/Qwen/Qwen1.5-32B
Fine-Tuning Goal: Train German language
Training dataset size: 160,000 SFT data / 110,000 DPO data
Training duration: 72.5 hours (2 epochs SFT / 1 epoch DPO)
GPU: 2x A100 SXM
New model: https://huggingface.co/VAGOsolutions/SauerkrautLM-Qwen-32b

Total cost: 312 euros 💶

These are quite reasonable training costs considering the model now speaks passable German (previously very broken). Depending on the use case and process requirements, this can even be a real alternative to the costly continuous pre-training of foreign language models.
🔥
jed-tiotuico osanseviero clem victor fpreiss philipp-zettl joennlae Kearm