Using a Variable was my first choice, but then Module.cuda() does not propagate to it.
With the use of Variable you suggest, blah.cuda() will convert blah.t.data to torch.cuda.FloatTensor as expected, but will leave blah.s.data unchanged.
Wouldn’t it be more consistent that Module.zero_grad() deal with requires_grad=False?
Edit: That would be in modules.py 6
def zero_grad(self):
    """Sets gradients of all model parameters to zero."""
    for p in self.parameters():
        if hasattr(p, 'grad'): p.grad.data.zero_()