Hi,

This is not worrying and could be cause by a few things. The first cuda op is always going to be slow as we lazily initialize cuda only when need it. Also since you don’t sync entering the forward, there might be some ops from before your main loop that are not completed (large data transfers or preprocessing).
The thing is that the GPUs are really (really) good at number crunching and really bad at logic. Working with sparse tensors makes a lot of things much more logic that just number crunching. Unless your matrix is very very sparse (<1% values), there is no chance that the sparse version is going to be faster.
Note that this is true if you have large enough workloads. For small ones, everything is simply going to be hidden by just the cost of launching the task on the GPU which is non negligible.