Kimonili:

So in general, there will be N=number_of_agents critic and actor networks. The centralized learning appears in the fact that each critic network recieves state information from the other agents and then updates its weights?


Yes



 Kimonili:

So here all the agents make an action simultaneously.


Yes



 Kimonili:

Given the action each agent made in the previous step, they transision to a new state. The state of each agent will be shared to all the agents. So basically, every agent will be aware of the position of every agent.


Yes



 Kimonili:

Here the critics are evaluating its agent’s actions but being aware of the new dynamics of the environments? (after getting informed of the relative position of every agent)


Well, PPO critic do not evaluate actions of agents, just state of the current agent, and partial states from other agents, since the critic represents value function V(s) and not Q value function Q(s, a).



 Kimonili:

Here the critics are evaluating its agent’s actions but being aware of the new dynamics of the environments? (after getting informed of the relative position of every agent)


Yes
A reference to my MADDPG implementation, includes three variants of parallel mechanism mentioned above, if you would like to take a look:

  
      github.com
  
  
    iffiX/machin/blob/master/machin/frame/algorithms/maddpg.py 11
import copy
import inspect
import itertools
from random import choice, randint
from .utils import determine_device
from machin.utils.visualize import make_dot
from machin.utils.logging import default_logger
from machin.model.nets.base import static_module_wrapper
from machin.parallel.pool import P2PPool, ThreadPool
# pylint: disable=wildcard-import, unused-wildcard-import
from .ddpg import *


class SHMBuffer(Buffer):
    @staticmethod
    def make_tensor_from_batch(batch, device, concatenate):
        # this function is used in post processing, and we will
        # move all cpu tensors to shared memory.
        if concatenate and len(batch) != 0:
            item = batch[0]


  This file has been truncated. show original