I am able to compute the loss and gradients on the GPU.  I am simply unable to store the resultant gradients in a single GPU while also using the same GPU to compute future losses and gradients.  So I simply want to gather and store the gradients elsewhere, like another GPU or CPU memory.
I tried computing loss on CPU, however the gradients still end up on the primary GPU.