Hi @ptrblck, the code works for some iterations and throws the OOM error after a while.


The repo code is kind of different from mine. It is fixed to four width(0.25x, 0.5x, 0.75x, 1.0x), while I am randomly sampling four widths in each iteration. One thing weird is that, if I set it to four fixed widths, I won’t have the OOM error. And the memory cost will be constant 4000MB after the first iteration. However, with randomly sampled width, the memory cost jumps around from 1000MB to 10000MB, and get OOM after some iterations.


Yes, I am accumulating the gradients for each different width and update the weights at the end of the iteration. I tried not to accumulate the gradients(by adding optimizer.step() and optimizer.zero_grad() after each loss.backward(); I don’t know if this is what you mean), I still have the OOM error.