I see, that is interesting about the batch and computation history in memory. But like I said nvidia-smi shows the GPU memory is not used up.
I need to do whatever I can to make the backprop faster as I’ve determined that to be by far the bottleneck (30 seconds to compute backwards() on 128,000 samples, each sample a vector of length 50). The only thing I can think of right now is to scale with multi GPU