KFrank:

First, for reasons I don’t understand, I had to explicitly move the
parameters in param_need_l1_penalty_case_2 to the gpu
(but not those in param_need_l1_penalty_case_1 ).


Take this “debugging” with a grain of salt, but that would be my best guess:
param_need_l1_penalty_case_1 was defined as an nn.Parameter and just wrapped in a list.
Iterating this list will yield these parameters, which were properly pushed to the device by calling model.to('cuda'), since they were also properly registered inside the module.
However, an operation on param_need_l1_penalty_case_2 was executed before storing them in the list (the slicing op). The result of this operation will be a tensor with a grad_fn (SelectBackward in this case).
Since this operation was performed inside the __init__ function of the module, and thus before pushing the parameters to the device, the slice will stay on CPU and you would need to push it manually to the device afterwards.



 KFrank:

Second, when running on the gpu, I had to convert the “penalty loss”
to a python scalar before adding it to loss in order to get rid of your
specific error


I doubt this will yield the desired behavior, since you are detaching the tensor with the .data call.
I would consider the usage of .data dangerous, as you might get rid of error messages, but in fact create unwanted behavior.
Is the code working without the .data call and by manually using cuda() on the slice?
If so, I would stick to this solution.