Adversarial learning has shown its advances in generating natural and diverse descriptions in image captioning. However, the learned reward of existing adversarial methods is vague and ill-defined due to the reward ambiguity problem. In this paper, we propose a refined Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in a sentence, as well as achieve stable adversarial training by refining the loss function to shift the stationary point towards Nash equilibrium. In addition, we introduce a conditional term in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions. Our experiments on MS COCO show that our method can learn compact reward for image captioning.
The authors propose using a recent method for adversarial inverse reinforcement learning (AIRL) for the task for generating high-quality image captions. Leveraging the GAN framework, a discriminator is trained to distinguish real captions from those produced by the generator, while the generator is optimized with policy-gradients (REINFORCE) to maximize the pseudo-reward from the discriminator. The main difference from prior work seems to be that the discriminator acts on a word-level, rather than sentence-level (as done, for instance in Dai et. al. 2017). Correspondingly, the generator policy is updated with the objective of 1-step reward maximization (more like contextual bandits), rather than with a long-term sequential decision-making objective (as done in Dai et. al. 2017). The evaluation is done using 2 data splits – standard and robust, with various metrics such as SPICE, CIDEr, BLEU, CHAIR. Diversity analysis and ablations are also performed to dissect the performance of the proposed approach. 
My 2 main issues with the paper are confusing motivation (in section 1) and various imprecise parts (in section 3 and 4).  
1.	The authors argue that current GAN-based captioning models provide ill-defined rewards due to the “reward ambiguity problem”. This problem is not explained or motivated well in the paper, but instead the readers are referred to the AIRL paper. “Reward ambiguity” in inverse-RL arises because there could be many reward functions that yield the same optimal policy. The AIRL algorithm recovers one of these possible reward functions, and since such a recovered reward could be shaped by the environment dynamics, AIRL attempts to disentangle the reward from the dynamics. The motivation there is to use the recovered reward on a new system with different transition dynamics. In the context of this paper though, I would like to understand the angle of reward ambiguity. The authors disentangle the sentence reward into word-wise rewards; however, I’m not sure if there’s any relation between this and the disentanglement done in AIRL for solving reward ambiguity.
2.	One of the objectives is learning compact rewards. It is claimed that addition of a constant term to the reward provided to the generator policy results in this, but what’s the intuition behind this? As for evaluation, it needs to be shown that words with similar semantics have similar discriminator score. How do we conclude this from Figure 2.? Also, please include Up-Down method results in Figure 2.
3.	Section 3 questions
     a.	How is state s_t defined? It is very hard to follow sections 3 and 4 without a clear definition and example for this.
     b.	“Finn et. al. 2016 proved that IRL is mathematically equivalent …” --- this is imprecise. Maximum-Entropy-IRL is equivalent to the GAN formulation, not general IRL. 
     c.	p_theta(a,s) is referred to as “reward distribution”. I don’t think it’s a distribution.
     d.	Equation 3. AIRL defines g only as a function of state (s_t) for the disentanglement, and not like what the authors have written.
     e.	Equation 4. How is s_t sampled?
4.	Section 4 questions
     a.	4.1 says discriminator “maximizes the divergence”. This doesn’t seem correct.
     b.	f is referred to as state-value. This doesn’t seem correct.
     c.	Shouldn’t the -1 term in Equation 8 disappear under expectation?
     d.	Don’t understand how second line of Equation 11 is arrived at.
There are quite a few other sources of mathematically imprecise writing that I noticed. I would recommend the authors to be more robust in their presentation.