By default, torch creates objects on CPU and you have to transfer whatever you need to other devices.
You can define a global variable like dtype=torch.cuda but you still need to specify that when making new objects.