In recent years, Large Language Models (LLMs) [1, 2] have achieved signiﬁcant
advancements, redeﬁning the frontier of artiﬁcial intelligence. These models, such as
OpenAI’s GPT-3 with its impressive 175 billion parameters, represent a quantum
leap in complexity and capability [3]. The trend in LLM development is toward ever-
increasing model sizes, with some recent entrants boasting upwards of hundreds of
billions of parameters [4–6]. This scale ampliﬁes their utility across a spectrum of
applications, from intelligent chatbots to intricate data analyses, and even facilitating
research in diverse domains. However, the exponential growth in model sizes presents
a huge demand for various resources (e.g., computation, energy, memory) [7–9]. The
immense resource requirements to train or deploy such extensive models can be cost-
prohibitive, particularly in resource-constrained environments like academic labs or
Beyond Efficiency