Hi Sebastian,
I think it’s a tradeoff between time and GPU memory.
For the memory: With the setup in the code, you can produce and process the autograd graphs for real and fake separately, so in a very lame sense, you can “double the batch size”.
In terms of performance: In the end, yes, without memory constraints, you would probably go for doing both at the same time (but also for the forward!). However, the advantage of “processing more data in one go” levels off at some point, so it might less general when you manage to efficiently use the GPU with the current setup, you have little to gain by doing real and fake in one go. (Basically just from the gradient accumulation itself, not so much the intermediates in the backward pass.)
Now, if you run on multi-GPU, the scaling probably works in favor of the backward once approach you suggest, at least if you also parallelize the forward passes.
In the end, the situation seems complicated enough that with the right batch size, I’d expect the sequential setup to be competitive in terms of speed. But it’s not optimization until you measure it, so it’d be interesting to hear about your experience.
Best regards
Thomas