Hi @ptrblck!



 ptrblck:

Take this “debugging” with a grain of salt, but that would be my best guess:
param_need_l1_penalty_case_1 was defined as an nn.Parameter and just wrapped in a list.
Iterating this list will yield these parameters, which were properly pushed to the device by calling model.to('cuda'), since they were also properly registered inside the module.
However, an operation on param_need_l1_penalty_case_2 was executed before storing them in the list (the slicing op). The result of this operation will be a tensor with a grad_fn (SelectBackward in this case).
Since this operation was performed inside the __init__ function of the module, and thus before pushing the parameters to the device, the slice will stay on CPU and you would need to push it manually to the device afterwards.


That was my thinking as well.  Anyway, explicitly moving the list
elements did work (for me).
A couple of odd things:  According to Fly’s original post, he apparently
didn’t see this error.  For me, this error occurred before calling
loss.backward(), and was, understandably, of the form
expected type torch.cuda.FloatTensor (instead of the
other way around).
Also I think (I don’t remember everything I tried) that it involved
the list in an essential way.  I think I tried getting rid of the list:
self.param_need_l1_penalty_case_2 = self.subModule.my_param[:,0]

and the (pre-loss.backward()) error went away, even though I was
slicing before moving the model (and hence my_param  to the gpu.
(Maybe model.cuda() knows to move Tensor variables of the
model, but not lists.  And since Fly didn’t see it, maybe it’s an 0.3.0
thing.)


I doubt this will yield the desired behavior, since you are detaching the tensor with the .data call.
I would consider the usage of .data dangerous, as you might get rid of error messages, but in fact create unwanted behavior.


I’m sure you’re right about this – I should have realized.  The forward
calculation works, but (presumably) the backward calculation will fail
to include the penalty-loss gradient in its overall gradient (defeating
the purpose).


Is the code working without the .data call and by manually using cuda() on the slice?
If so, I would stick to this solution. 


No, two different errors:
Explicitly moving the slice gets rid of the reasonably-understandable
expected type torch.cuda.FloatTensor error.
The error I tried to address with my incorrect modification was the
expected type torch.FloatTensor that Fly originally reported,
and that I was able to reproduce.
To me it’s baffling.  I probed all of the tensors in sight to make sure
that they were torch.cuda.FloatTensors, so I couldn’t figure out
where (in the bowels of loss.backward()) somebody was expecting
a torch.FloatTensor.  And it seems like this isn’t just 0.3.0 weirdness
because Fly sees it with whatever his version is, as well.
Remaining baffled,
K. Frank