Yes, overwriting the to function solves my problem. For the previous example, the codes to overwrite are as follows:
class Toy(torch.nn.Module):
	def __init__(self):
		super(Toy,self).__init__()
		self.var = torch.ones([],requires_grad=True)
	def forward(self,x):
		print(self.var.device)
		return self.var*x
	def to(self,device):
		super(Toy,self).to(device)
		self.var = self.var.to(device)