From what I have read so far, it seems the option sparse=True is necessary when tuning the embedding matrix during training, since otherwise the backward step will take a long time. (This was my experience: an average of ~7secs for backward with non-sparse; 0.38 with sparse).
However I have encountered an issue when trying to apply optimization (with Adam):
torch.cuda.sparse.FloatTensor has no attribute addcmul_
I have tried to build a minimalistic model to reproduce the error, it is below:
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.embedding = nn.Embedding(10000, 300, sparse=True).cuda()
        self.linear1 = nn.Linear(300, 1).cuda()
        self.optimizer = optim.Adam(self.parameters(), 0.0001)

    def forward(self):
        ixs = Variable(torch.LongTensor([1, 2, 3, 4])).cuda()
        vecs = self.embedding(ixs)
        logits = F.sigmoid(self.linear1(vecs))
        return logits

    def optimize(self, logits):
        labels = Variable(torch.LongTensor([0, 1, 0, 1])).cuda()
        criterion = nn.CrossEntropyLoss().cuda()
        loss = criterion(logits, labels)
        loss.backward()
        self.optimizer.step()

model = MyModel()
logits = model.forward()
model.optimize(logits)

Please let me know if I am doing something wrong! Thanks very much for your kind help.