@ani0075 is right, you’d need to rely on a specific sparse matrix algebra library for speedups.
You can find my old answer to a similar question here: https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076 20