By default only Parameters will be recognized and pushed to the according device.
Try to wrap your random tensor into torch.nn.Parameter(torch.randn(1), requires_grad=False) and it should work.
Alternatively you could also store it in a nn.ModuleList.