title: EfficientLLM: Efficiency in Large Language Models
summary: Large Language Models (LLMs) have driven significant progress, yet their
  growing parameter counts and context windows incur prohibitive compute, energy,
  and monetary costs. We introduce EfficientLLM, a novel benchmark and the first
  comprehensive empirical study evaluating efficiency techniques for LLMs at
  scale. Conducted on a production-class cluster (48xGH200, 8xH200 GPUs), our
  study systematically explores three key axes: (1) architecture pretraining
  (efficient attention variants: MQA, GQA, MLA, NSA; sparse Mixture-of-Experts
  (MoE)), (2) fine-tuning (parameter-efficient methods: LoRA, RSLoRA, DoRA), and
  (3) inference (quantization methods: int4, float16). We define six fine-grained
  metrics (Memory Utilization, Compute Utilization, Latency, Throughput, Energy
  Consumption, Compression Rate) to capture hardware saturation,
  latency-throughput balance, and carbon cost. Evaluating over 100
  model-technique pairs (0.5B-72B parameters), we derive three core insights: (i)
  Efficiency involves quantifiable trade-offs: no single method is universally
  optimal; e.g., MoE reduces FLOPs and improves accuracy but increases VRAM by
  40%, while int4 quantization cuts memory/energy by up to 3.9x at a 3-5%
  accuracy drop. (ii) Optima are task- and scale-dependent: MQA offers optimal
  memory-latency trade-offs for constrained devices, MLA achieves lowest
  perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency
  only beyond 14B parameters. (iii) Techniques generalize across modalities: we
  extend evaluations to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and
  Vision-Language Models (Qwen2.5-VL), confirming effective transferability. By
  open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM
  provides essential guidance for researchers and engineers navigating the
  efficiency-performance landscape of next-generation foundation models.
paper_authors._id: 682d217607e45a72c3caa952
paper_authors.name: Zhengqing Yuan
paper_authors._id: 682d217607e45a72c3caa953
paper_authors.name: Weixiang Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:41:33.729Z
paper_authors._id: 6481a16f70ac5e1968a7bb97
paper_authors.avatarUrl: /avatars/39fd34fd2bfab8057961642d0ed80f09.svg
paper_authors.fullname: Weixiang Sun
paper_authors.type: user
paper_authors.user: Sweson
paper_authors._id: 682d217607e45a72c3caa954
paper_authors.name: Yixin Liu
paper_authors._id: 682d217607e45a72c3caa955
paper_authors.name: Huichi Zhou
paper_authors._id: 682d217607e45a72c3caa956
paper_authors.name: Rong Zhou
paper_authors._id: 682d217607e45a72c3caa957
paper_authors.name: Yiyang Li
paper_authors._id: 682d217607e45a72c3caa958
paper_authors.name: Zheyuan Zhang
paper_authors._id: 682d217607e45a72c3caa959
paper_authors.name: Wei Song
paper_authors._id: 682d217607e45a72c3caa95a
paper_authors.name: Yue Huang
paper_authors._id: 682d217607e45a72c3caa95b
paper_authors.name: Haolong Jia
paper_authors._id: 682d217607e45a72c3caa95c
paper_authors.name: Keerthiram Murugesan
paper_authors._id: 682d217607e45a72c3caa95d
paper_authors.name: Yu Wang
paper_authors._id: 682d217607e45a72c3caa95e
paper_authors.name: Lifang He
paper_authors._id: 682d217607e45a72c3caa95f
paper_authors.name: Jianfeng Gao
paper_authors._id: 682d217607e45a72c3caa960
paper_authors.name: Lichao Sun
paper_authors._id: 682d217607e45a72c3caa961
paper_authors.name: Yanfang Ye
paper_title: EfficientLLM: Efficiency in Large Language Models
paper_summary: Large Language Models (LLMs) have driven significant progress, yet their
  growing parameter counts and context windows incur prohibitive compute, energy,
  and monetary costs. We introduce EfficientLLM, a novel benchmark and the first
  comprehensive empirical study evaluating efficiency techniques for LLMs at
  scale. Conducted on a production-class cluster (48xGH200, 8xH200 GPUs), our
  study systematically explores three key axes: (1) architecture pretraining
  (efficient attention variants: MQA, GQA, MLA, NSA; sparse Mixture-of-Experts
  (MoE)), (2) fine-tuning (parameter-efficient methods: LoRA, RSLoRA, DoRA), and
  (3) inference (quantization methods: int4, float16). We define six fine-grained
  metrics (Memory Utilization, Compute Utilization, Latency, Throughput, Energy
  Consumption, Compression Rate) to capture hardware saturation,
  latency-throughput balance, and carbon cost. Evaluating over 100
  model-technique pairs (0.5B-72B parameters), we derive three core insights: (i)
  Efficiency involves quantifiable trade-offs: no single method is universally
  optimal; e.g., MoE reduces FLOPs and improves accuracy but increases VRAM by
  40%, while int4 quantization cuts memory/energy by up to 3.9x at a 3-5%
  accuracy drop. (ii) Optima are task- and scale-dependent: MQA offers optimal
  memory-latency trade-offs for constrained devices, MLA achieves lowest
  perplexity for quality-critical tasks, and RSLoRA surpasses LoRA efficiency
  only beyond 14B parameters. (iii) Techniques generalize across modalities: we
  extend evaluations to Large Vision Models (Stable Diffusion 3.5, Wan 2.1) and
  Vision-Language Models (Qwen2.5-VL), confirming effective transferability. By
  open-sourcing datasets, evaluation pipelines, and leaderboards, EfficientLLM
  provides essential guidance for researchers and engineers navigating the
  efficiency-performance landscape of next-generation foundation models.
paper_ai_summary: EfficientLLM evaluates efficiency techniques for LLMs across architecture pretraining, fine-tuning, and inference, demonstrating task-dependent optima and cross-modal generalization.
paper_ai_keywords: LLMs EfficientLLM MQA GQA MLA NSA sparse Mixture-of-Experts (MoE) parameter-efficient methods LoRA RSLoRA DoRA quantization methods int4 float16 Memory Utilization Compute Utilization Latency Throughput Energy Consumption Compression Rate Large Vision Models Vision-Language Models