Using a card solely for the to accumulate gradients doesn’t sound so good. Whats your model look like?
In many models the activations use more memory than the weights and gradients, so try decreasing your batch size if you’re running out of memory.