Solved by ptrblck in post #2 
              
              
                nn.ReLU doesn’t contain any parameters, so nothing has to be moved to the device. 
Internally the functional API will be used on the input (which is already on the GPU) via F.relu. 
You can check for parameters in a modules via: print(dict(module.named_parameters())). 
Also, while this code works fi…