Solved by ptrblck in post #2 
              
              
                Yes, you can move around activations, as Autograd will track the to()/cpu()/cuda() operations. 
E.g.: 
# setup
x_cuda = torch.randn(1, 1).to('cuda')
lin_cuda = nn.Linear(1, 1).to('cuda')
lin_cpu = nn.Linear(1, 1)

# workload on the GPU
out = lin_cuda(x_cuda)
# transfer to CPU
out_cpu = out.to('cpu')…