title: Evaluating Open Language Models Across Task Types, Application Domains,
    and Reasoning Types: An In-Depth Experimental Analysis
summary: The rapid rise of Language Models (LMs) has expanded their use in several
  applications. Yet, due to constraints of model size, associated cost, or
  proprietary restrictions, utilizing state-of-the-art (SOTA) LLMs is not always
  feasible. With open, smaller LMs emerging, more applications can leverage their
  capabilities, but selecting the right LM can be challenging. This work conducts
  an in-depth experimental analysis of the semantic correctness of outputs of 10
  smaller, open LMs across three aspects: task types, application domains and
  reasoning types, using diverse prompt styles. We demonstrate that most
  effective models and prompt styles vary depending on the specific requirements.
  Our analysis provides a comparative assessment of LMs and prompt styles using a
  proposed three-tier schema of aspects for their strategic selection based on
  use-case and other constraints. We also show that if utilized appropriately,
  these LMs can compete with, and sometimes outperform, SOTA LLMs like
  DeepSeek-v2, GPT-3.5-Turbo, and GPT-4o.
paper_authors._id: 667132b5bb4aa4b6a9c4471f
paper_authors.name: Neelabh Sinha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T21:26:07.288Z
paper_authors._id: 6639a17d7256327c68a0841e
paper_authors.avatarUrl: /avatars/28fcd935149c2d89661c6f642842c310.svg
paper_authors.fullname: Neelabh Sinha
paper_authors.type: user
paper_authors.user: neelabhsinha
paper_authors._id: 667132b5bb4aa4b6a9c44720
paper_authors.name: Vinija Jain
paper_authors._id: 667132b5bb4aa4b6a9c44721
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:17:11.713Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_title: Evaluating Open Language Models Across Task Types, Application Domains,
    and Reasoning Types: An In-Depth Experimental Analysis
paper_summary: The rapid rise of Language Models (LMs) has expanded their use in several
  applications. Yet, due to constraints of model size, associated cost, or
  proprietary restrictions, utilizing state-of-the-art (SOTA) LLMs is not always
  feasible. With open, smaller LMs emerging, more applications can leverage their
  capabilities, but selecting the right LM can be challenging. This work conducts
  an in-depth experimental analysis of the semantic correctness of outputs of 10
  smaller, open LMs across three aspects: task types, application domains and
  reasoning types, using diverse prompt styles. We demonstrate that most
  effective models and prompt styles vary depending on the specific requirements.
  Our analysis provides a comparative assessment of LMs and prompt styles using a
  proposed three-tier schema of aspects for their strategic selection based on
  use-case and other constraints. We also show that if utilized appropriately,
  these LMs can compete with, and sometimes outperform, SOTA LLMs like
  DeepSeek-v2, GPT-3.5-Turbo, and GPT-4o.
paper_ai_summary: An experimental analysis of smaller open Language Models across various tasks, domains, and reasoning types reveals that effective models and prompt styles vary by use-case, enabling them to compete with state-of-the-art models.
paper_ai_keywords: Language Models semantic correctness task types application domains reasoning types prompt styles three-tier schema strategic selection