title: Exploring the Abilities of Large Language Models to Solve Proportional
    Analogies via Knowledge-Enhanced Prompting
summary: Making analogies is fundamental to cognition. Proportional analogies, which
  consist of four terms, are often used to assess linguistic and cognitive
  abilities. For instance, completing analogies like "Oxygen is to Gas as <blank>
  is to <blank>" requires identifying the semantic relationship (e.g., "type of")
  between the first pair of terms ("Oxygen" and "Gas") and finding a second pair
  that shares the same relationship (e.g., "Aluminum" and "Metal"). In this work,
  we introduce a 15K Multiple-Choice Question Answering (MCQA) dataset for
  proportional analogy completion and evaluate the performance of contemporary
  Large Language Models (LLMs) in various knowledge-enhanced prompt settings.
  Specifically, we augment prompts with three types of knowledge: exemplar,
  structured, and targeted. Our results show that despite extensive training
  data, solving proportional analogies remains challenging for current LLMs, with
  the best model achieving an accuracy of 55%. Notably, we find that providing
  targeted knowledge can better assist models in completing proportional
  analogies compared to providing exemplars or collections of structured
  knowledge.
paper_authors._id: 674f5e7fada64263fc369469
paper_authors.name: Thilini Wijesiriwardene
paper_authors._id: 674f5e7fada64263fc36946a
paper_authors.name: Ruwan Wickramarachchi
paper_authors._id: 674f5e7fada64263fc36946b
paper_authors.name: Sreeram Vennam
paper_authors._id: 674f5e7fada64263fc36946c
paper_authors.name: Vinija Jain
paper_authors._id: 674f5e7fada64263fc36946d
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:46:13.632Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 674f5e7fada64263fc36946e
paper_authors.name: Amitava Das
paper_authors._id: 674f5e7fada64263fc36946f
paper_authors.name: Ponnurangam Kumaraguru
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:51:19.013Z
paper_authors._id: 67a6a4b7f379cef464950268
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yaMsT2jYsj6mvvYU1gvO_.jpeg
paper_authors.fullname: ponnurangam kumaraguru
paper_authors.type: user
paper_authors.user: pk-profgiri
paper_authors._id: 674f5e7fada64263fc369470
paper_authors.name: Amit Sheth
paper_title: Exploring the Abilities of Large Language Models to Solve Proportional
    Analogies via Knowledge-Enhanced Prompting
paper_summary: Making analogies is fundamental to cognition. Proportional analogies, which
  consist of four terms, are often used to assess linguistic and cognitive
  abilities. For instance, completing analogies like "Oxygen is to Gas as <blank>
  is to <blank>" requires identifying the semantic relationship (e.g., "type of")
  between the first pair of terms ("Oxygen" and "Gas") and finding a second pair
  that shares the same relationship (e.g., "Aluminum" and "Metal"). In this work,
  we introduce a 15K Multiple-Choice Question Answering (MCQA) dataset for
  proportional analogy completion and evaluate the performance of contemporary
  Large Language Models (LLMs) in various knowledge-enhanced prompt settings.
  Specifically, we augment prompts with three types of knowledge: exemplar,
  structured, and targeted. Our results show that despite extensive training
  data, solving proportional analogies remains challenging for current LLMs, with
  the best model achieving an accuracy of 55%. Notably, we find that providing
  targeted knowledge can better assist models in completing proportional
  analogies compared to providing exemplars or collections of structured
  knowledge.
paper_ai_summary: LLMs struggle with completing proportional analogies, even with augmented knowledge, with targeted knowledge proving most beneficial.
paper_ai_keywords: Large Language Models MCQA dataset proportional analogies exemplar structured targeted knowledge