Regarding 1 & 2, yes the shared parameters will be updated. With the call to model.share_memory parameter tensors are put into SHM, so each process is actually operating on the same model.

The wording is probably not the best, but that part means the grad tensor is also shared as long as grad is not none. If you’ve called model.shared_memory as example does, model params will always be shared.
There aren’t any application-level threads here, loss is computed per-process so it can be different across each process. So it is not total loss.
If the model has conditional branches, autograd automatically handles this by constructing the backward graph on the fly during forward pass.
Backward is run locally on each process, there is no inter-process coordination. The .grad field is updated in a shared way across each process, this can lead to some processes stepping on each other’s updates, but this is a result of hogwild training.