What if we use pipeline mechanism to load micro-batches instead of mini-batches?
And do the final weight vector update at the end of the mini-batch.
Then that sequential relationship can still be maintained (as it cannot be broken because of weight vector update requirement). But when backward takes more time, more GPUs’ can do backward task and reduce backward timing. Isn’t this a practical use-case where we could improve performance?