It’s because you called .cuda() before assignment to tmap. Casting or copying the Varaible to another device returns you a different Variable, so tmap is not a leaf and won’t have the gradient accumulated. Either do this:
tmap_leaf = Variable(...)
tmap = tmap_leaf.cuda()

or better
tmap = Variable(torch.from_numpy(...).float().cuda(), requires_grad=True)