Pruning is a good way to reduce model size, but it won’t automatically give you performance improvements.
The reason is that when pruning a model you’re introducing zeros to Tensors (i.e. they become sparse). These tensors can be compressed (with some extra overhead) without much problem. Hence the reduced model size.
However, the underlying kernels (i.e. code that operates on the Tensors holding the values, like a matrix multiply) are by default written assuming dense Tensors, because hardware provides instructions to make those dense operations fast (e.g via vectorized on adjacent values in memory).
Unfortunately there’s not a lot of hardware, including CPUs, that can benefit directly from sparse Tensors.
There’s some work arounds to still speeding up sparse operations in some cases. In CPU, for example, doing structured pruning (i.e. removing blocks of values rather than one by one) in a way that matches the size of the registers used in those vectorized operations such that you can skip entire blocks of continuous zeros. For example, pruning blocks of 1x16 or 4x4 and quantizing afterwards, matches the register sizes of vectorized instructions in CPUs (128bit).
However, even if you use structured pruning, you will still depend on having underlying kernels for the operations used in your model being implemented to take advantage of the structured nature of your tensors (i.e. to know that they can skip entire sequences of 16 values when doing things like a matrix multiply).
Why is it slower?
In theory a pruned model shouldn’t be slower since the dense operations could ignore tensors coming in that happen to have a bunch of zeros. However, if a special representation is used to represent the sparse Tensor (i.e. to avoid having to represent the zeros), as it’s likely the case here, then that will cause inefficiencies when operating with those special sparse Tensors (e.g. some copying of values may be needed to restore the original structure of the Tensor to go through the dense kernels).
Hope this helps.