title: Bielik 11B v2 Technical Report
summary: We present Bielik 11B v2, a state-of-the-art language model optimized for
  Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to
  11B parameters using depth up-scaling, this model demonstrates exceptional
  performance across Polish language benchmarks while maintaining strong
  cross-lingual capabilities. We introduce two key technical innovations:
  Weighted Instruction Cross-Entropy Loss, which optimizes learning across
  diverse instruction types by assigning quality-based weights to training
  examples, and Adaptive Learning Rate, which dynamically adjusts based on
  context length. Comprehensive evaluation across multiple benchmarks
  demonstrates that Bielik 11B v2 outperforms many larger models, including those
  with 2-6 times more parameters, and significantly surpasses other specialized
  Polish language models on tasks ranging from linguistic understanding to
  complex reasoning. The model's parameter efficiency and extensive quantization
  options enable deployment across various hardware configurations, advancing
  Polish language AI capabilities and establishing new benchmarks for
  resource-efficient language modeling in less-represented languages.
paper_authors._id: 6819f19e5c7ea9f74284d3a3
paper_authors.name: Krzysztof Ociepa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T09:03:52.265Z
paper_authors._id: 63ecbccac8827dd0f0f59579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ecbccac8827dd0f0f59579/kz-2F9Z0QKllifgZmr8tH.jpeg
paper_authors.fullname: Chris Ociepa
paper_authors.type: user
paper_authors.user: chrisociepa
paper_authors._id: 6819f19e5c7ea9f74284d3a4
paper_authors.name: Łukasz Flis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:36:13.128Z
paper_authors._id: 6609d7a8362a1d713aa4b260
paper_authors.avatarUrl: /avatars/5ed9fbc9352e8664787aa56b89b96611.svg
paper_authors.fullname: Lukasz Flis
paper_authors.type: user
paper_authors.user: lflis
paper_authors._id: 6819f19e5c7ea9f74284d3a5
paper_authors.name: Krzysztof Wróbel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T09:03:50.340Z
paper_authors._id: 5e47d3eb178ca95365287400
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633554153914-5e47d3eb178ca95365287400.png
paper_authors.fullname: Krzysztof Wróbel
paper_authors.type: user
paper_authors.user: djstrong
paper_authors._id: 6819f19e5c7ea9f74284d3a6
paper_authors.name: Adrian Gwoździej
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T12:56:22.907Z
paper_authors._id: 636b9b006890dd5a450081a5
paper_authors.avatarUrl: /avatars/2b4ba89894d952f95ecd1e9926580608.svg
paper_authors.fullname: AG
paper_authors.type: user
paper_authors.user: adgw
paper_authors._id: 6819f19e5c7ea9f74284d3a7
paper_authors.name: Remigiusz Kinas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T06:51:13.426Z
paper_authors._id: 61786d0b038518aa2827c6b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61786d0b038518aa2827c6b7/d1UnfivoVreYebS5JM3P9.jpeg
paper_authors.fullname: Remek Kinas
paper_authors.type: user
paper_authors.user: Remek
paper_title: Bielik 11B v2 Technical Report
paper_summary: We present Bielik 11B v2, a state-of-the-art language model optimized for
  Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to
  11B parameters using depth up-scaling, this model demonstrates exceptional
  performance across Polish language benchmarks while maintaining strong
  cross-lingual capabilities. We introduce two key technical innovations:
  Weighted Instruction Cross-Entropy Loss, which optimizes learning across
  diverse instruction types by assigning quality-based weights to training
  examples, and Adaptive Learning Rate, which dynamically adjusts based on
  context length. Comprehensive evaluation across multiple benchmarks
  demonstrates that Bielik 11B v2 outperforms many larger models, including those
  with 2-6 times more parameters, and significantly surpasses other specialized
  Polish language models on tasks ranging from linguistic understanding to
  complex reasoning. The model's parameter efficiency and extensive quantization
  options enable deployment across various hardware configurations, advancing
  Polish language AI capabilities and establishing new benchmarks for
  resource-efficient language modeling in less-represented languages.
paper_ai_summary: Bielik 11B v2, a scaled language model with 11B parameters, excels on Polish benchmarks through Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate, outperforming larger models and demonstrating resource-efficient deployment.
paper_ai_keywords: Weighted Instruction Cross-Entropy Loss Adaptive Learning Rate depth up-scaling parameter efficiency quantization