Then is there a way to store accumulated gradients and/or the optimizer on a separate GPU or perform the forward pass without using the primary GPU?  As it stands, 3/4 of my GPUs are not even half utilized.