Hi @mriganktiwari, Francois is referring to his brand-new nn_pruning 7 library that extends the work done by Victor et al on movement pruning and provides an inference speed-up without quantization.
If you’re having trouble quantizing GPT-2, one idea could be to convert it to ONNX format and then optimize the graph with ONNX runtime as done here: onnxruntime/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb at master · microsoft/onnxruntime · GitHub
I’ve generally found the ONNX runtime supports more operators for quantization and the notebook I linked to shows you how to do it