title: Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and
    Evaluation
summary: We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for
  Polish language processing. Trained on curated Polish corpora, this model
  addresses key challenges in language model development through innovative
  techniques. These include Weighted Instruction Cross-Entropy Loss, which
  balances the learning of different instruction types, and Adaptive Learning
  Rate, which dynamically adjusts the learning rate based on training progress.
  To evaluate performance, we created the Open PL LLM Leaderboard and Polish
  MT-Bench, novel frameworks assessing various NLP tasks and conversational
  abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9
  percentage point increase in average score compared to Mistral-7B-v0.1 on the
  RAG Reader task. It also excels in the Polish MT-Bench, particularly in
  Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model
  represents a substantial advancement in Polish language AI, offering a powerful
  tool for diverse linguistic applications and setting new benchmarks in the
  field.
paper_authors._id: 671cd85ad5e54b67236902c8
paper_authors.name: Krzysztof Ociepa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:24:02.294Z
paper_authors._id: 63ecbccac8827dd0f0f59579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ecbccac8827dd0f0f59579/kz-2F9Z0QKllifgZmr8tH.jpeg
paper_authors.fullname: Chris Ociepa
paper_authors.type: user
paper_authors.user: chrisociepa
paper_authors._id: 671cd85ad5e54b67236902c9
paper_authors.name: Łukasz Flis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:14:45.213Z
paper_authors._id: 6609d7a8362a1d713aa4b260
paper_authors.avatarUrl: /avatars/5ed9fbc9352e8664787aa56b89b96611.svg
paper_authors.fullname: Lukasz Flis
paper_authors.type: user
paper_authors.user: lflis
paper_authors._id: 671cd85ad5e54b67236902ca
paper_authors.name: Krzysztof Wróbel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:36:20.109Z
paper_authors._id: 5e47d3eb178ca95365287400
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633554153914-5e47d3eb178ca95365287400.png
paper_authors.fullname: Krzysztof Wróbel
paper_authors.type: user
paper_authors.user: djstrong
paper_authors._id: 671cd85ad5e54b67236902cb
paper_authors.name: Adrian Gwoździej
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:24:14.679Z
paper_authors._id: 636b9b006890dd5a450081a5
paper_authors.avatarUrl: /avatars/2b4ba89894d952f95ecd1e9926580608.svg
paper_authors.fullname: AG
paper_authors.type: user
paper_authors.user: adgw
paper_authors._id: 671cd85ad5e54b67236902cc
paper_authors.name: Remigiusz Kinas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:24:08.369Z
paper_authors._id: 61786d0b038518aa2827c6b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61786d0b038518aa2827c6b7/d1UnfivoVreYebS5JM3P9.jpeg
paper_authors.fullname: Remek Kinas
paper_authors.type: user
paper_authors.user: Remek
paper_title: Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and
    Evaluation
paper_summary: We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for
  Polish language processing. Trained on curated Polish corpora, this model
  addresses key challenges in language model development through innovative
  techniques. These include Weighted Instruction Cross-Entropy Loss, which
  balances the learning of different instruction types, and Adaptive Learning
  Rate, which dynamically adjusts the learning rate based on training progress.
  To evaluate performance, we created the Open PL LLM Leaderboard and Polish
  MT-Bench, novel frameworks assessing various NLP tasks and conversational
  abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9
  percentage point increase in average score compared to Mistral-7B-v0.1 on the
  RAG Reader task. It also excels in the Polish MT-Bench, particularly in
  Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model
  represents a substantial advancement in Polish language AI, offering a powerful
  tool for diverse linguistic applications and setting new benchmarks in the
  field.
paper_ai_summary: Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish, uses Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate to enhance performance in various NLP tasks, surpassing Mistral-7B-v0.1 and setting new benchmarks.
paper_ai_keywords: Weighted Instruction Cross-Entropy Loss Adaptive Learning Rate Open PL LLM Leaderboard Polish MT-Bench generative text model NLP tasks RAG Reader Reasoning Role-playing