title: How Many Parameters Does it Take to Change a Light Bulb? Evaluating
    Performance in Self-Play of Conversational Games as a Function of Model
    Characteristics
summary: What makes a good Large Language Model (LLM)? That it performs well on the
  relevant benchmarks -- which hopefully measure, with some validity, the
  presence of capabilities that are also challenged in real application. But what
  makes the model perform well? What gives a model its abilities? We take a
  recently introduced type of benchmark that is meant to challenge capabilities
  in a goal-directed, agentive context through self-play of conversational games,
  and analyse how performance develops as a function of model characteristics
  like number of parameters, or type of training. We find that while there is a
  clear relationship between number of parameters and performance, there is still
  a wide spread of performance points within a given size bracket, which is to be
  accounted for by training parameters such as fine-tuning data quality and
  method. From a more practical angle, we also find a certain degree of
  unpredictability about performance across access methods, possible due to
  unexposed sampling parameters, and a, very welcome, performance stability
  against at least moderate weight quantisation during inference.
paper_authors._id: 667a77f4536de14d364d93a1
paper_authors.name: Nidhir Bhavsar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:36:06.679Z
paper_authors._id: 60faa5d92336d38bec48b0e4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1640016076582-60faa5d92336d38bec48b0e4.png
paper_authors.fullname: Nidhir Bhavsar
paper_authors.type: user
paper_authors.user: nid989
paper_authors._id: 667a77f4536de14d364d93a2
paper_authors.name: Jonathan Jordan
paper_authors._id: 667a77f4536de14d364d93a3
paper_authors.name: Sherzod Hakimov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T08:17:48.382Z
paper_authors._id: 647061e39ad4008a29061769
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647061e39ad4008a29061769/nHm9LquGI0Nh-MutiBvah.jpeg
paper_authors.fullname: Sherzod Hakimov
paper_authors.type: user
paper_authors.user: sherzod-hakimov
paper_authors._id: 667a77f4536de14d364d93a4
paper_authors.name: David Schlangen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:37:52.639Z
paper_authors._id: 60c117aaea206f28132e3457
paper_authors.avatarUrl: /avatars/94d1dab7d7c26f69b2f865bebd13f6ee.svg
paper_authors.fullname: David Schlangen
paper_authors.type: user
paper_authors.user: davids
paper_title: How Many Parameters Does it Take to Change a Light Bulb? Evaluating
    Performance in Self-Play of Conversational Games as a Function of Model
    Characteristics
paper_summary: What makes a good Large Language Model (LLM)? That it performs well on the
  relevant benchmarks -- which hopefully measure, with some validity, the
  presence of capabilities that are also challenged in real application. But what
  makes the model perform well? What gives a model its abilities? We take a
  recently introduced type of benchmark that is meant to challenge capabilities
  in a goal-directed, agentive context through self-play of conversational games,
  and analyse how performance develops as a function of model characteristics
  like number of parameters, or type of training. We find that while there is a
  clear relationship between number of parameters and performance, there is still
  a wide spread of performance points within a given size bracket, which is to be
  accounted for by training parameters such as fine-tuning data quality and
  method. From a more practical angle, we also find a certain degree of
  unpredictability about performance across access methods, possible due to
  unexposed sampling parameters, and a, very welcome, performance stability
  against at least moderate weight quantisation during inference.
paper_ai_summary: Performance of Large Language Models varies significantly within each size bracket and is influenced by training parameters and fine-tuning data quality, showing stability against moderate weight quantisation during inference.
paper_ai_keywords: Large Language Model LLM self-play conversational games performance number of parameters fine-tuning data quality training parameters sampling parameters weight quantisation inference