SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are:<p>- Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.3B model<p>- The core issue to scale is time to calculate the Hessian during prune analysis (effectively a matrix of second-order derivatives, famously computationally intense to calculate)<p>- They follow the existing literature and use a local approach to each layer. By doing this (and doing it well), it can preserve the input/output contract for surrounding layers, which makes the whole thing paralellizable across machines<p>- Their solution approximates reconstruction loss by approximating a quadratic loss and then running a OBS update (with a few other optimizations on ordering and iteration on the side)<p>I'm particularly excited for these smaller models, mostly for inference efficiency gains in realtime applications. The general con of weight pruning is they still require incredibly large training clusters / investment in training resources upfront to get the original parameter weight. But if the lottery ticket hypothesis holds true, this might be the best way we have at the moment to get models with same performance and lower longterm operational costs.