How Many Parameters Does it Take to Change a Light Bulb? Evaluating
  Performance in Self-Play of Conversational Games as a Function of Model
  Characteristics
What makes a good Large Language Model (LLM)? That it performs well on the
relevant benchmarks -- which hopefully measure, with some validity, the
presence of capabilities that are also challenged in real application. But what
makes the model perform well? What gives a model its abilities? We take a
recently introduced type of benchmark that is meant to challenge capabilities
in a goal-directed, agentive context through self-play of conversational games,
and analyse how performance develops as a function of model characteristics
like number of parameters, or type of training. We find that while there is a
clear relationship between number of parameters and performance, there is still
a wide spread of performance points within a given size bracket, which is to be
accounted for by training parameters such as fine-tuning data quality and
method. From a more practical angle, we also find a certain degree of
unpredictability about performance across access methods, possible due to
unexposed sampling parameters, and a, very welcome, performance stability
against at least moderate weight quantisation during inference.
667a77f4536de14d364d93a1
Nidhir Bhavsar
admin_assigned
2024-06-25T13:36:06.679Z
60faa5d92336d38bec48b0e4
https://cdn-avatars.huggingface.co/v1/production/uploads/1640016076582-60faa5d92336d38bec48b0e4.png
Nidhir Bhavsar
user
nid989
667a77f4536de14d364d93a2
Jonathan Jordan
667a77f4536de14d364d93a3
Sherzod Hakimov
claimed_verified
2024-06-25T08:17:48.382Z
647061e39ad4008a29061769
https://cdn-avatars.huggingface.co/v1/production/uploads/647061e39ad4008a29061769/nHm9LquGI0Nh-MutiBvah.jpeg
Sherzod Hakimov
user
sherzod-hakimov
667a77f4536de14d364d93a4
David Schlangen
admin_assigned
2024-06-25T13:37:52.639Z
60c117aaea206f28132e3457
/avatars/94d1dab7d7c26f69b2f865bebd13f6ee.svg
David Schlangen
user
davids
How Many Parameters Does it Take to Change a Light Bulb? Evaluating
  Performance in Self-Play of Conversational Games as a Function of Model
  Characteristics
What makes a good Large Language Model (LLM)? That it performs well on the
relevant benchmarks -- which hopefully measure, with some validity, the
presence of capabilities that are also challenged in real application. But what
makes the model perform well? What gives a model its abilities? We take a
recently introduced type of benchmark that is meant to challenge capabilities
in a goal-directed, agentive context through self-play of conversational games,
and analyse how performance develops as a function of model characteristics
like number of parameters, or type of training. We find that while there is a
clear relationship between number of parameters and performance, there is still
a wide spread of performance points within a given size bracket, which is to be
accounted for by training parameters such as fine-tuning data quality and
method. From a more practical angle, we also find a certain degree of
unpredictability about performance across access methods, possible due to
unexposed sampling parameters, and a, very welcome, performance stability
against at least moderate weight quantisation during inference.
Performance of Large Language Models varies significantly within each size bracket and is influenced by training parameters and fine-tuning data quality, showing stability against moderate weight quantisation during inference.
Large Language Model LLM self-play conversational games performance number of parameters fine-tuning data quality training parameters sampling parameters weight quantisation inference