title: H2O-Danube3 Technical Report
summary: We present H2O-Danube3, a series of small language models consisting of
  H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T
  tokens. Our models are pre-trained on high quality Web data consisting of
  primarily English tokens in three stages with different data mixes before final
  supervised tuning for chat version. The models exhibit highly competitive
  metrics across a multitude of academic, chat, and fine-tuning benchmarks.
  Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a
  modern smartphone, enabling local inference and rapid processing capabilities
  even on mobile devices. We make all models openly available under Apache 2.0
  license further democratizing LLMs to a wider audience economically.
paper_authors._id: 669488b7ce544964b27167a2
paper_authors.name: Pascal Pfeiffer
paper_authors._id: 669488b7ce544964b27167a3
paper_authors.name: Philipp Singer
paper_authors._id: 669488b7ce544964b27167a4
paper_authors.name: Yauhen Babakhin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:31.298Z
paper_authors._id: 643f884da649e1a499446c11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643f884da649e1a499446c11/4lmRszXeD1S62sfwUHg2D.jpeg
paper_authors.fullname: Yauhen Babakhin
paper_authors.type: user
paper_authors.user: ybabakhin
paper_authors._id: 669488b7ce544964b27167a5
paper_authors.name: Gabor Fodor
paper_authors._id: 669488b7ce544964b27167a6
paper_authors.name: Nischay Dhankhar
paper_authors._id: 669488b7ce544964b27167a7
paper_authors.name: Sri Satish Ambati
paper_title: H2O-Danube3 Technical Report
paper_summary: We present H2O-Danube3, a series of small language models consisting of
  H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T
  tokens. Our models are pre-trained on high quality Web data consisting of
  primarily English tokens in three stages with different data mixes before final
  supervised tuning for chat version. The models exhibit highly competitive
  metrics across a multitude of academic, chat, and fine-tuning benchmarks.
  Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a
  modern smartphone, enabling local inference and rapid processing capabilities
  even on mobile devices. We make all models openly available under Apache 2.0
  license further democratizing LLMs to a wider audience economically.
paper_ai_keywords: language models H2O-Danube3-4B H2O-Danube3-500M Web data pre-trained supervised tuning chat version semantic scene synthesis local inference