title: Void in Language Models
summary: Despite advances in transformer-based language models (LMs), a fundamental
  question remains largely unanswered: Are all layers activated during inference?
  We investigate this question by detecting unactivated layers (which we refer to
  as Voids) using a non-trainable and parameter-free adaptive computation method
  called L2 Adaptive Computation (LAC). We adapt LAC from its original
  efficiency-focused application to trace activated layers during inference. This
  method monitors changes in the L2-norm of activations to identify voids. We
  analyze layer activation in instruction-tuned LMs across two phases: Prompt
  Processing (PP), where we trace activated layers for each token in the input
  prompts, and Response Generation (RG), where we trace activated layers for each
  generated token. We further demonstrate that distinct layers are activated
  during these two phases. To show the effectiveness of our method, we evaluated
  three distinct instruction-tuned LMs from the Llama, Mistral, and Qwen families
  on three benchmarks: MMLU, GPQA Diamond, and BoolQ. For example, on MMLU with a
  zero-shot setting, skipping voids in Qwen2.5-7B-Instruct resulted in an
  improvement from 69.24 to 71.29 while the model uses only 30% of the layers.
  Similarly, Mistral-7B-Instruct-v0.3 on GPQA Diamond improved from 13.88 to
  18.36 when using 70% of the layers during both the PP and RG phases. These
  results show that not all layers contribute equally during inference, and that
  selectively skipping most of them can improve the performance of models on
  certain tasks.
paper_authors._id: 682e32a6a8253020d05e242d
paper_authors.name: Mani Shemiranifar
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-21T20:12:25.545Z
paper_authors._id: 6109845486580d4580767223
paper_authors.avatarUrl: /avatars/9192fda36782a3f208daaff701884544.svg
paper_authors.fullname: Mani shemranifar
paper_authors.type: user
paper_authors.user: Manishemi
paper_title: Void in Language Models
paper_summary: Despite advances in transformer-based language models (LMs), a fundamental
  question remains largely unanswered: Are all layers activated during inference?
  We investigate this question by detecting unactivated layers (which we refer to
  as Voids) using a non-trainable and parameter-free adaptive computation method
  called L2 Adaptive Computation (LAC). We adapt LAC from its original
  efficiency-focused application to trace activated layers during inference. This
  method monitors changes in the L2-norm of activations to identify voids. We
  analyze layer activation in instruction-tuned LMs across two phases: Prompt
  Processing (PP), where we trace activated layers for each token in the input
  prompts, and Response Generation (RG), where we trace activated layers for each
  generated token. We further demonstrate that distinct layers are activated
  during these two phases. To show the effectiveness of our method, we evaluated
  three distinct instruction-tuned LMs from the Llama, Mistral, and Qwen families
  on three benchmarks: MMLU, GPQA Diamond, and BoolQ. For example, on MMLU with a
  zero-shot setting, skipping voids in Qwen2.5-7B-Instruct resulted in an
  improvement from 69.24 to 71.29 while the model uses only 30% of the layers.
  Similarly, Mistral-7B-Instruct-v0.3 on GPQA Diamond improved from 13.88 to
  18.36 when using 70% of the layers during both the PP and RG phases. These
  results show that not all layers contribute equally during inference, and that
  selectively skipping most of them can improve the performance of models on
  certain tasks.
paper_ai_summary: Analyzing layer activation using L2 Adaptive Computation reveals that not all layers in transformer-based language models are used during inference, and selectively skipping unactivated layers can improve performance.
paper_ai_keywords: transformer-based language models L2 Adaptive Computation Voids Prompt Processing Response Generation MMLU GPQA Diamond BoolQ zero-shot setting