Solved by ptrblck in post #2 
              
              
                By default only Parameters will be recognized and pushed to the according device. 
Try to wrap your random tensor into torch.nn.Parameter(torch.randn(1), requires_grad=False) and it should work. 
Alternatively you could also store it in a nn.ModuleList.