If you don’t care for doing hogwild, the second example you list is not applicable to you. Everything is declared in the primary process because the model weights are to be shared (physically shared through shared memory) between the worker processes.
If you want to simply use a single process per GPU and don’t care for physical weight sharing, then you should declare everything in the subprocesses. Think of it as if you were running this on different machines instead of multiple processes on a single machine. Then you’d also have to declare everything in the processes that you launch on those machine instead of a single launcher processes.