Hi all,
I also have a network which performs a backward in the forward pass, although it doesn’t use an optimizer step.
My issue is a bit unrelated, but it’s the closes example I have found online. Glad to open a new discussion if you think it would be best.
My issue is the following : I seem to have noticed an inefficiency in my forward pass when on multi-GPU. The GPUs are suddenly running with a low util, which doesn’t happen on a single GPU and doesn’t happen for models which don’t use a backward in the forward pass.
I would like to know if some syncing could be going on when using backward therefore making multi-GPU less efficient.
I am at the moment trying to find a reproducible example but it’s not that easy.