you are probably better off using:
class A(nn.Module): 
    ...:     def __init__(self): 
    ...:         super(A, self).__init__() 
    ...:         self.linear = nn.Linear(10, 5) 
    ...:         tensor = torch.randn(10) 
    ...:         self.extra_param = nn.Parameter(tensor.to(self.linear.weight.device))

This way it will work for both GPU and CPU