Thank you for the reply. Actually, the situation for network distributed in multiple devices servers as a motivation for device-agnostic modules.
My objective is more like implementing custom Conv2D. And Here is a toy example:
class Toy(torch.nn.Module):
	def __init__(self):
		super(Toy,self).__init__()
		self.var = torch.ones([],requires_grad=True)
	def forward(self,x):
		print(self.var.device)
		return self.var*x

toy = Toy()
print(toy.var.device) #cpu 
a = torch.ones([],device='gpu')
print(toy(a).device) #cpu 
toy.to('cuda')
print(toy(a).device) #cpu 
toy.cuda()
print(toy(a).device) #cpu 

On the other hand, for example, the Linear module will be pushed to cuda after running .to('cuda')
l = Linear(3,4)
print(l.weight.device) #cpu 
l.to('cuda')
print(l.weight.device) #cuda

Should I overwrite the to function to implement it?