Adding retain_grad() fixes it though!  Kinda odd…
    def testGradCheckEmbeddingBasic(self):
        seqs = ['ghatmasala', 'nicela', 'c-pakodas']
        e = nn.Embedding(10, 3, sparse=False).double()
        indices = Variable(torch.LongTensor([[1], [4]]))
        embed = e(indices)
        print(embed)
        embed.retain_grad()
        input = (embed, )
        model = nn.Linear(3, 3).double()
        test = gradcheck(model, input, eps=1e-6, atol=1e-4)

Variable containing:
(0 ,.,.) =
-1.5149  0.3036 -0.8191
(1 ,.,.) =
-0.6803  0.7728  2.4776
[torch.DoubleTensor of size 2x1x3]
Ran 1 test in 0.246s
OK