nn.ReLU doesn’t contain any parameters, so nothing has to be moved to the device.
Internally the functional API will be used on the input (which is already on the GPU) via F.relu.
You can check for parameters in a modules via: print(dict(module.named_parameters())).
Also, while this code works fine (and I guess it’s just used as an example for this topic), I would generally recommend to remove the to operations from the model internals, and move the complete model to the device via model.to('cuda').
If you want to use model sharding using different devices, your workflow would be correct.