I was stuck thinking about the same thing for a while now. Just to confirm, using the same module at different positions in the graph will have separate gradients and forward buffers? And pytorch will handle averaging these gradients when backward is called?
Sorry if this is very obvious.