Oh sorry, let me clarify.  My problem is that I cannot fit a single mini-batch, gradients, and optimizer on a single GPU (in this case the primary GPU).  I originally thought, wrongly, the extra memory usage on the primary GPU was some sort of bug.
I now want to try to get around this by having a separate GPU dedicated to storing the gradients and optimizer which is not used in the forward or backward pass.  So if I have 5 GPUs, I would use 1-4 to calculate the loss and gradients, then send the resulting gradients to the 5th GPU in order to update the parameters.
Thanks for the help so far though!