title: Bielik v3 Small: Technical Report
summary: We introduce Bielik v3, a series of parameter-efficient generative text
  models (1.5B and 4.5B) optimized for Polish language processing. These models
  demonstrate that smaller, well-optimized architectures can achieve performance
  comparable to much larger counterparts while requiring substantially fewer
  computational resources. Our approach incorporates several key innovations: a
  custom Polish tokenizer (APT4) that significantly improves token efficiency,
  Weighted Instruction Cross-Entropy Loss to balance learning across instruction
  types, and Adaptive Learning Rate that dynamically adjusts based on training
  progress. Trained on a meticulously curated corpus of 292 billion tokens
  spanning 303 million documents, these models excel across multiple benchmarks,
  including the Open PL LLM Leaderboard, Complex Polish Text Understanding
  Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter
  model achieves results competitive with models 2-3 times its size, while the
  1.5B model delivers strong performance despite its extremely compact profile.
  These advances establish new benchmarks for parameter-efficient language
  modeling in less-represented languages, making high-quality Polish language AI
  more accessible for resource-constrained applications.
paper_authors._id: 6819ef0b2ff435c58da4d860
paper_authors.name: Krzysztof Ociepa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T09:03:56.213Z
paper_authors._id: 63ecbccac8827dd0f0f59579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ecbccac8827dd0f0f59579/kz-2F9Z0QKllifgZmr8tH.jpeg
paper_authors.fullname: Chris Ociepa
paper_authors.type: user
paper_authors.user: chrisociepa
paper_authors._id: 6819ef0b2ff435c58da4d861
paper_authors.name: Łukasz Flis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:36:18.501Z
paper_authors._id: 6609d7a8362a1d713aa4b260
paper_authors.avatarUrl: /avatars/5ed9fbc9352e8664787aa56b89b96611.svg
paper_authors.fullname: Lukasz Flis
paper_authors.type: user
paper_authors.user: lflis
paper_authors._id: 6819ef0b2ff435c58da4d862
paper_authors.name: Remigiusz Kinas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T06:51:15.217Z
paper_authors._id: 61786d0b038518aa2827c6b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61786d0b038518aa2827c6b7/d1UnfivoVreYebS5JM3P9.jpeg
paper_authors.fullname: Remek Kinas
paper_authors.type: user
paper_authors.user: Remek
paper_authors._id: 6819ef0b2ff435c58da4d863
paper_authors.name: Krzysztof Wróbel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T09:03:54.135Z
paper_authors._id: 5e47d3eb178ca95365287400
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633554153914-5e47d3eb178ca95365287400.png
paper_authors.fullname: Krzysztof Wróbel
paper_authors.type: user
paper_authors.user: djstrong
paper_authors._id: 6819ef0b2ff435c58da4d864
paper_authors.name: Adrian Gwoździej
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T12:56:24.966Z
paper_authors._id: 636b9b006890dd5a450081a5
paper_authors.avatarUrl: /avatars/2b4ba89894d952f95ecd1e9926580608.svg
paper_authors.fullname: AG
paper_authors.type: user
paper_authors.user: adgw
paper_title: Bielik v3 Small: Technical Report
paper_summary: We introduce Bielik v3, a series of parameter-efficient generative text
  models (1.5B and 4.5B) optimized for Polish language processing. These models
  demonstrate that smaller, well-optimized architectures can achieve performance
  comparable to much larger counterparts while requiring substantially fewer
  computational resources. Our approach incorporates several key innovations: a
  custom Polish tokenizer (APT4) that significantly improves token efficiency,
  Weighted Instruction Cross-Entropy Loss to balance learning across instruction
  types, and Adaptive Learning Rate that dynamically adjusts based on training
  progress. Trained on a meticulously curated corpus of 292 billion tokens
  spanning 303 million documents, these models excel across multiple benchmarks,
  including the Open PL LLM Leaderboard, Complex Polish Text Understanding
  Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter
  model achieves results competitive with models 2-3 times its size, while the
  1.5B model delivers strong performance despite its extremely compact profile.
  These advances establish new benchmarks for parameter-efficient language
  modeling in less-represented languages, making high-quality Polish language AI
  more accessible for resource-constrained applications.
paper_ai_summary: Bielik v3, a series of parameter-efficient generative text models, achieves high performance in Polish language processing with a custom tokenizer, Weighted Instruction Cross-Entropy Loss, and Adaptive Learning Rate.
paper_ai_keywords: parameter-efficient generative text models custom Polish tokenizer Weighted Instruction Cross-Entropy Loss Adaptive Learning Rate Open PL LLM Leaderboard Complex Polish Text Understanding Benchmark Polish EQ-Bench Polish Medical Leaderboard