Well, then its very simple, critic V(x)_i of agent i will collect the current observed state of agent i and the so called “partial state” of other agents j, then predict a value. So the code structure of the training part would be parallelly sample an action from all agent policies P(x)_m, m \in {0, n} where n is the number of agents, then you can exchange the state information between all agents, then parallelly use critics to evaluate values, then parallely update critics, and finally parallely update actors using values produced by critics.
The simplest (with medium performance) way to implement this is using a threadpool, eg: the one from joblib or multiprocessing module, if your model is small. A more complex way would be using torch.jit.fork_, which is even better than using the threadpool because it completely avoids the GIL problem (about 50% faster than threadpool). ProcessPool is only recommended if your model is on aa GPU and very large (like using a resnet or something of similar complexity and parameter size, there are many caveats in using a ProcessPool and I don’t recommend it.