3https://www.tensorﬂow.org/datasets/catalog/c4

6

Published as a conference paper at ICLR 2023

Hardware There are numerous efforts towards hardware and software support for sparsity. An
example of hardware acceleration for inference is 2:4 ﬁne-grained structured sparsity that was in-
troduced by the Nvidia A100 GPU (Pool et al., 2021; Pool & Yu, 2021). When each contiguous
block of four elements contains two zeros (viz. 50% sparsity), a low-overhead compression be-
comes possible which stores the non-zero values together with 2-bit indices. The hardware supports
this compressed format by only operating on the nonzero values during the computation.
JAXPRUNER