We recently fixed a performance issue with SGD on sparse gradients! If you checkout the master code, it should be vastly faster (and use much less memory!. 
The fix is at https://github.com/pytorch/pytorch/pull/3139 93 if you are interested.