Just an idea: what if you prune the model first with nn_pruning, convert to ONNX and then quantize / optimize with ORT?
I’m not sure whether the optimized models produced by nn_pruning (i.e. after the heads/rows/columns with zeroes are removed) can be exported to ONNX format, but this might be a way to get the best of both worlds