Upon further profiling, it seems almost all the remaining space on primary GPU is being consumed by gradient accumulation.  When calling step() on the optimizer, the remaining space is used resulting in an out of memory error.  Is this intended behavior to store gradients and optimizers on GPU 0?  If so this seems highly inconvenient…