I’ve checked the dense version before and it is indeed so much faster.
The dense representation doesn’t fit into my memory though 
I just wrote my own index_select functions for sparse Tensors instead, which works for my needs.
Thanks for your help though