Bielik 11B v2 Technical Report
We present Bielik 11B v2, a state-of-the-art language model optimized for
Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to
11B parameters using depth up-scaling, this model demonstrates exceptional
performance across Polish language benchmarks while maintaining strong
cross-lingual capabilities. We introduce two key technical innovations:
Weighted Instruction Cross-Entropy Loss, which optimizes learning across
diverse instruction types by assigning quality-based weights to training
examples, and Adaptive Learning Rate, which dynamically adjusts based on
context length. Comprehensive evaluation across multiple benchmarks
demonstrates that Bielik 11B v2 outperforms many larger models, including those
with 2-6 times more parameters, and significantly surpasses other specialized
Polish language models on tasks ranging from linguistic understanding to
complex reasoning. The model's parameter efficiency and extensive quantization
options enable deployment across various hardware configurations, advancing
Polish language AI capabilities and establishing new benchmarks for
resource-efficient language modeling in less-represented languages.
6819f19e5c7ea9f74284d3a3
Krzysztof Ociepa
claimed_verified
2025-05-12T09:03:52.265Z
63ecbccac8827dd0f0f59579
https://cdn-avatars.huggingface.co/v1/production/uploads/63ecbccac8827dd0f0f59579/kz-2F9Z0QKllifgZmr8tH.jpeg
Chris Ociepa
user
chrisociepa
6819f19e5c7ea9f74284d3a4
Łukasz Flis
claimed_verified
2025-05-14T07:36:13.128Z
6609d7a8362a1d713aa4b260
/avatars/5ed9fbc9352e8664787aa56b89b96611.svg
Lukasz Flis
user
lflis
6819f19e5c7ea9f74284d3a5
Krzysztof Wróbel
claimed_verified
2025-05-12T09:03:50.340Z
5e47d3eb178ca95365287400
https://cdn-avatars.huggingface.co/v1/production/uploads/1633554153914-5e47d3eb178ca95365287400.png
Krzysztof Wróbel
user
djstrong
6819f19e5c7ea9f74284d3a6
Adrian Gwoździej
claimed_verified
2025-05-12T12:56:22.907Z
636b9b006890dd5a450081a5
/avatars/2b4ba89894d952f95ecd1e9926580608.svg
AG
user
adgw
6819f19e5c7ea9f74284d3a7
Remigiusz Kinas
claimed_verified
2025-05-12T06:51:13.426Z
61786d0b038518aa2827c6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/61786d0b038518aa2827c6b7/d1UnfivoVreYebS5JM3P9.jpeg
Remek Kinas
user
Remek
Bielik 11B v2 Technical Report
We present Bielik 11B v2, a state-of-the-art language model optimized for
Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to
11B parameters using depth up-scaling, this model demonstrates exceptional
performance across Polish language benchmarks while maintaining strong
cross-lingual capabilities. We introduce two key technical innovations:
Weighted Instruction Cross-Entropy Loss, which optimizes learning across
diverse instruction types by assigning quality-based weights to training
examples, and Adaptive Learning Rate, which dynamically adjusts based on
context length. Comprehensive evaluation across multiple benchmarks
demonstrates that Bielik 11B v2 outperforms many larger models, including those
with 2-6 times more parameters, and significantly surpasses other specialized
Polish language models on tasks ranging from linguistic understanding to
complex reasoning. The model's parameter efficiency and extensive quantization
options enable deployment across various hardware configurations, advancing
Polish language AI capabilities and establishing new benchmarks for
resource-efficient language modeling in less-represented languages.
Bielik 11B v2, a scaled language model with 11B parameters, excels on Polish benchmarks through Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate, outperforming larger models and demonstrating resource-efficient deployment.
Weighted Instruction Cross-Entropy Loss Adaptive Learning Rate depth up-scaling parameter efficiency quantization