There is a huge amount of optimization to dense weight conv, especially if you use cudnn. So even if sparse weights work, they are likely slower than dense weights. Considering that the size of conv kernels is usually not very large, I don’t think this is a very worthy thing to do.