Hello, I used torch.nn.utils.prune to apply weight pruning on BERT. It generates sparse matrices as expected. However, the model’s size doesn’t decrease and the inference speed doesn’t increase? How can I speed up after pruning and actually compress the model’s size? Thank you!
BERT_QA = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
for child in BERT_QA.modules():
    if isinstance(child, nn.Linear):
        prune.l1_unstructured(child, name='weight', amount=0.2)
        prune.l1_unstructured(child, name='bias', amount=0.2)
        prune.remove(child, 'weight')
        prune.remove(child,'bias')