title: H2O-Danube-1.8B Technical Report
summary: We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens
  following the core principles of LLama 2 and Mistral. We leverage and refine
  various techniques for pre-training large language models. Although our model
  is trained on significantly fewer total tokens compared to reference models of
  similar size, it exhibits highly competitive metrics across a multitude of
  benchmarks. We additionally release a chat model trained with supervised
  fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B
  openly available under Apache 2.0 license further democratizing LLMs to a wider
  audience economically.
paper_authors._id: 65b9b4cf8187af6cd33b695c
paper_authors.name: Philipp Singer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:13:20.392Z
paper_authors._id: 636d18755aaed143cd6698ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636d18755aaed143cd6698ef/AalDh13Gp8jv1BfM5IASh.png
paper_authors.fullname: Philipp Singer
paper_authors.type: user
paper_authors.user: psinger
paper_authors._id: 65b9b4cf8187af6cd33b695d
paper_authors.name: Pascal Pfeiffer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:13:40.358Z
paper_authors._id: 6316fc44c92fd6fee3161e9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676417502037-6316fc44c92fd6fee3161e9a.png
paper_authors.fullname: Pascal Pfeiffer
paper_authors.type: user
paper_authors.user: ilu000
paper_authors._id: 65b9b4cf8187af6cd33b695e
paper_authors.name: Yauhen Babakhin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:13:46.762Z
paper_authors._id: 643f884da649e1a499446c11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643f884da649e1a499446c11/4lmRszXeD1S62sfwUHg2D.jpeg
paper_authors.fullname: Yauhen Babakhin
paper_authors.type: user
paper_authors.user: ybabakhin
paper_authors._id: 65b9b4cf8187af6cd33b695f
paper_authors.name: Maximilian Jeblick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:13:53.749Z
paper_authors._id: 614c9145b44c102499783617
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674725099569-614c9145b44c102499783617.jpeg
paper_authors.fullname: Maximilian Jeblick
paper_authors.type: user
paper_authors.user: MaxJeblick
paper_authors._id: 65b9b4cf8187af6cd33b6960
paper_authors.name: Nischay Dhankhar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:14:06.030Z
paper_authors._id: 659bd735d7207fb4a2d021d9
paper_authors.avatarUrl: /avatars/f9f0283231955b5034fe36861c999601.svg
paper_authors.fullname: Nischay Dhankhar
paper_authors.type: user
paper_authors.user: Nischay27
paper_authors._id: 65b9b4cf8187af6cd33b6961
paper_authors.name: Gabor Fodor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:14:31.967Z
paper_authors._id: 6576c9e4e58db15694dfe69a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6576c9e4e58db15694dfe69a/7WYZdlpqi0TEVN24za-op.jpeg
paper_authors.fullname: Gabor Fodor
paper_authors.type: user
paper_authors.user: gaborfodorh2o
paper_authors._id: 65b9b4cf8187af6cd33b6962
paper_authors.name: Sri Satish Ambati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-31T10:14:43.015Z
paper_authors._id: 60cc616faee6f81536eee756
paper_authors.avatarUrl: /avatars/5022e020a9804157a6f82804b4942d51.svg
paper_authors.fullname: sri ambati
paper_authors.type: user
paper_authors.user: srisatish
paper_title: H2O-Danube-1.8B Technical Report
paper_summary: We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens
  following the core principles of LLama 2 and Mistral. We leverage and refine
  various techniques for pre-training large language models. Although our model
  is trained on significantly fewer total tokens compared to reference models of
  similar size, it exhibits highly competitive metrics across a multitude of
  benchmarks. We additionally release a chat model trained with supervised
  fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B
  openly available under Apache 2.0 license further democratizing LLMs to a wider
  audience economically.
paper_ai_keywords: language model pre-training LLama 2 Mistral supervised fine-tuning direct preference optimization chat model LLMs Apache 2.0 license language model Pre-Training LLama 2 Mistral supervised fine-tuning direct preference optimization chat model LLMs Apache 2.0 license language model pre-training LLama 2 Mistral supervised fine-tuning direct preference optimization chat model LLMs Apache 2.0 license language model pre-training techniques LLama 2 Mistral supervised fine-tuning direct preference optimization chat model LLMs Apache 2.0 license language model