We propose a deep reinforcement learning algorithm for semi-cooperative multiagent tasks, where agents are equipped with their separate reward functions, yet with willingness to cooperate. Under these semi-cooperative scenarios, popular methods of centralized training with decentralized execution for inducing cooperation and removing the non-stationarity problem do not work well due to lack of a common shared reward as well as inscalability in centralized training. Our algorithm, called Peer Evaluation based Dual DQN (PED-DQN), proposes to give peer evaluation signals to observed agents, which quantifies how they feel about a certain transition. This exchange of peer evaluation over time turns out to render agents to gradually reshape their reward functions so that their action choices from the myopic best-response tend to result in the good joint action with high cooperation. This evaluation-based method also allows flexible and scalable training by not assuming knowledge of the number of other agents and their observation and action spaces. We provide the performance evaluation of PED-DQN for the scenarios ranging from a simple two-person prisoner's dilemma to more complex semi-cooperative multi-agent tasks. In special cases where agents share a common reward function as in the centralized training methods, we show that inter-agent evaluation leads to better performance.
This work is well-written, but the quality of some sections can be improved significantly as suggested in the comments. I have a few main concerns that I explain in detailed comments. Among those, the paper argues that the algorithms converge without discussing why. Also, the amount of overestimation of the Q-values are one of my big concerns and not intuitive for me in the game of Prisoner's Dilemma that needs to be justified. For these reasons, I am voting for a weak reject now and conditional on the authors' rebuttal, I might increase my score later.
1) I have a series of questions about Prisoner's Dilemma example. I am curious to see what are the Q-values for t=100000 in PD. Is table 1h shows the converged values? What I am expecting to see is that the Q-values should converge to some values slightly larger than 3, but the values are  ~ 30. It is important to quantify how much bias you add to the optimal solution by reward shaping, and why this difference in the Q-values are observed.
2) One thing that is totally missing is the discussion of convergence of the proposed method. In section 3.4, you say that the Q-values converge, but it is not discussed why we expect convergence. The only place in the paper which I can make a conjecture about the convergence is in figure 4c which implicitly implies the convergence of the Mission DQN, but for the other one, I don't see such an observation. Is it possible to formalize the proposed method in the tabular case and discuss whether the Q-values should converge or not? Also, I would like to see the comparison of the Q-values plots in the experiments for both networks.
3) The intuition behind (2) should be better clarified. An example will be informative. I don't understand what |Z_a| is doing in this formula.
4) One of the main contributions of the paper is proposing the reward shaping mechanism. When I read section 3.3, I was expecting to see some result for policy gradient algorithms as well, but this paper does not analyze these algorithms. That would be very nice to see its performance in PG algorithms though. In such case that you are not going to implement these algorithms, I would suggest moving this section to the end of the paper and add it to a section named discussion and conclusion.
5) Is it possible to change the order of parts where you define $\hat{r}$ with the next part where you define $z_a$? I think that the clarity of this section should be improved. This is just a suggestion to explore. I was confused at the first reading when I saw $z_a$, "evaluation of transition" and then (2) without knowing how you define evaluation and why.
6) Is there any reason that ablation testing is only done for trio case? or you choose it randomly. Does the same behavior hold for other cases too?
7) Why in figure 4a, random is always around zero?
8) What will happen if you pass the location of the agent in addition to its observation? In this way, it is possible to have one  Dual-Q-network shared for all agents. This experiment might be added to the baselines in future revisions.
Minor: 
* afore-mentioned -> aforementioned
section 4.2: I-DQN is used before definition
* Is it R_a in (4)?
* I assume that the table 1f-1h are not for the case of using independent Q-learning. Introducing these tables for the first time right after saying "This is observed when we use independent Q-learning" means that these values are coming from independent Q-learning, while they are not as far as I understand. Please make sure that this is correct.
section 4.1: * who's -> whose
* This work is also trying to answer a similar question to yours and should be referenced: "Learning Policy Representations in Multiagent Systems, by Grover et al. 2018"
* Visual illustrations of the game would be helpful in understanding the details of the experiment. Preparing a video of the learned policies also would informative.
-----------------------------------------------
After rebuttal: after reading the answers, I got answers to most of my questions. Some parts of the paper are vague that I see that other reviewers had the same questions. Given the amount of change required to address these modifications, I am not sure about the quality of the final work, so I keep my score the same.