Yes, it makes sense, but I don’t understand the assumption of why the matrix would be sparse most of the cases. For old-fashioned convolutions/models/datasets, sparsity seems not so common if not “model induced”.
So I guess that’s the reason why I couldn’t understand your advice  .
But maybe you are telling me that most of the feature maps are indeed sparse, and under the hood pytorch does lots of checks in this regard. So apart the first layer, everything else is sparsified.