DOTResize: Reducing LLM Width via Discrete Optimal Transport-based
  Neuron Merging
Model compression offers a promising path to reducing the cost and
inaccessibility of large pre-trained models, without significantly compromising
their impressive performance. Large Transformer models, including large
language models (LLMs), often contain computational redundancy, which can serve
as a target for new model compression methods. In this work, we specifically
target neuron-level redundancies in model layers by combining groups of similar
neurons into fewer neurons. We frame this width reduction as a Discrete Optimal
Transport problem, and propose DOTResize, a novel Transformer compression
method that uses optimal transport theory to transform and compress model
weights. To ensure applicability within the Transformer architecture, we
motivate and incorporate entropic regularization and matrix factorization into
the transportation maps produced by our method. Unlike pruning-based approaches
which discard neurons based on importance measures, DOTResize re-projects the
entire neuron width, allowing the retention and redistribution of useful signal
across the reduced layer. Empirical results show that compared to simple or
state-of-the-art neuron width-pruning techniques, DOTResize can outperform
these methods across multiple LLM families and sizes, while achieving
measurable reductions in real-world computational cost.
687052f5c8391850d60977cf
Neha Verma
claimed_verified
2025-07-11T08:02:35.591Z
6138e0f027fc7c37930c7a12
/avatars/28f19853c7c365b5222caf8cf5297958.svg
Neha Verma
user
nverma
687052f5c8391850d60977d0
Kenton Murray
687052f5c8391850d60977d1
Kevin Duh
DOTResize: Reducing LLM Width via Discrete Optimal Transport-based
  Neuron Merging
Model compression offers a promising path to reducing the cost and
inaccessibility of large pre-trained models, without significantly compromising
their impressive performance. Large Transformer models, including large
language models (LLMs), often contain computational redundancy, which can serve
as a target for new model compression methods. In this work, we specifically
target neuron-level redundancies in model layers by combining groups of similar
neurons into fewer neurons. We frame this width reduction as a Discrete Optimal
Transport problem, and propose DOTResize, a novel Transformer compression
method that uses optimal transport theory to transform and compress model
weights. To ensure applicability within the Transformer architecture, we
motivate and incorporate entropic regularization and matrix factorization into
the transportation maps produced by our method. Unlike pruning-based approaches
which discard neurons based on importance measures, DOTResize re-projects the
entire neuron width, allowing the retention and redistribution of useful signal
across the reduced layer. Empirical results show that compared to simple or
state-of-the-art neuron width-pruning techniques, DOTResize can outperform
these methods across multiple LLM families and sizes, while achieving
measurable reductions in real-world computational cost.
DOTResize, a novel Transformer compression method using optimal transport theory, reduces neuron-level redundancies and outperforms pruning techniques in computational efficiency across various large language models.
Discrete Optimal Transport DOTResize optimal transport theory entropic regularization matrix factorization neuron-level redundancies neuron width-pruning large language models computational efficiency