comparisons in linear layer outputs, facilitating the exclusion of less signiﬁcant weights
in LLMs. [119] proposes a novel dynamic inference scheme, DynaTran, which prunes
activations at runtime with low overhead, which improves the throughput of trans-
former inference. The author designed an application-speciﬁc integrated circuit (ASIC)
based architecture called AccelTran to implement the DynaTran. Speciﬁcally, several
hardware-aware designs are proposed including matrix tiling and various dataﬂow to
improve data reuse.
Structured pruning. Structured pruning involves the selective removal of an entire
group of weights. The deﬁnition of ‘group’, which makes those amenable to hardware
speedup, could refer to weight blocks, neurons, ﬁlters/channels, attention heads, or
other dedicated ﬁne-grained sparse patterns. [120] introduce LLM-Pruner, a pioneering
framework tailored for structured pruning of LLMs oﬀering task-agnostic compression
Beyond Efficiency