Recent advances in Generative Adversarial Networks facilitated by improvements to the framework and successful application to various problems has resulted in extensions to multiple domains. IRGAN attempts to leverage the framework for Information-Retrieval (IR), a task that can be described as modeling the correct conditional probability distribution p(d|q) over the documents (d), given the query (q). The work that proposes IRGAN claims that optimizing their minimax loss function will result in a generator which can learn the distribution, but their setup and baseline term steer the model away from an exact adversarial formulation, and this work attempts to point out certain inaccuracies in their formulation. Analyzing their loss curves gives insight into possible mistakes in the loss functions and better performance can be obtained by using the co-training like setup we propose, where two models are trained in a co-operative rather than an adversarial fashion.
This paper tries to argue that the formulation of IRGAN (a method from 2017 that aimed to use GANs for the standard IR task of estimation query-document relevance) is now well-founded and has inherent weaknesses. Specifically the paper claims that (unlike regular GANs and what was likely intended by the authors or IRGAN) the discriminator and generator are working against each other. The paper then aims to show a couple of more well-founded different (generator-free) setups that perform about as well (if not better) as the original IRGAN work.
Overall I found the work to be quite interesting and the findings to be illuminating. That said I think the paper notably lacked rigor and depth which definitely hurt the quality of the paper.
Below are my thoughts on the different facets as well as more detailed strengths / weaknesses breakdown:
Quality: Above average
As mentioned I think some of the findings are illuminating and thus overage the paper scores well on this aspect.
Clarity: Slightly above average
While the paper is largely easy to follow, there are certain key sections that are not well explained / have fundamental errors.
Originality: Strong
Significance: Little below average
My (main) concern here with this work is that the missing rigor and depth of the work is what is needed for readers to have a deeper understanding of the fundamental issue so as to be able to rectify it in future works.
---
Strengths / Things I liked about the work:
+ The topic / theme of the work: I believe as a community we should encourage more such works that take a critical deep dive into recently proposed methods that may have some inherent weaknesses. As the authors noted the IRGAN work has become quite popular despite some of these (previously unknown) issues.
+ The experimental results in general do a fair job illustrating the likely issue (though I would have liked to see more rigor and depth here as well as detailed below)
Weaknesses / Things that concerned me:
- (W1) Lacking rigor / depth: One of my big concerns with this work is that the analysis to demonstrate the inherent flaws of IRGAN if fairly shallow and not detailed enough. For example, Section 5 (which should have been the key section of the work) is quite poorly written and not rigorous enough. Claiming that log(1-z) can be replaced with - log(z) is incorrect -- how can this substitution be made as is?
Overall my sense after reading the work is that I understand that the IRGAN formulation is not completely well-formed in terms of discriminator/generator synergy (the pairwise formulation has the additional issue of separating real pairs rather than higher rank-lower rank pairs). However I do not buy that the generator and discriminator directly oppose each other as is claimed in the work (I believe this arises only due to the incorrect claim that log(1-z) can be replaced with -log(z)).
Thus at the end of the day I feel the reader is willing to buy there is a issue with the formulation, but they do not fully understand it not do they understand enough to understand how to rectify the underlying issues. To me that was unfortunate as the paper would have been an excellent work if it had done so.
- (W2) Missing some experimental results / deeper insights : There were some notable empirical results that were missing or not provided, that raised some concerns in my mind. For instance I don't see the co-training approach listed for the MovieLens dataset .. Why so? The authors make a secondary claim that they are able to improve upon IRGANs via their proposed approach but then they do not substantiate these on all the datasets which seems like a notable oversight.
 - (W3) Missing details: To add to the above I think the authors can clearly be more detailed in describing for instance the models for D, G, p_\psi etc .. Right now I can speculate what they are but I don't think a reader should be expected to speculate in such cases. Likewise empirical details about the datasets and their sizes could easily have been added.
Also the paper presents the IRGAN pairwise approach and mentions pairs in a couple of places but I don't see an approach that can learn from pairs among the ones proposed.
Another example is the two proposed models Fig 2a (Only discriminator) vs Fig 2b (cotraining). I don't see an explanation or intuition for why 2b is expected to be better than 2a. Given the claims of the work I would have wanted to understand this better.
- (W4) Significance testing: This is an important experimental process to understand the validity of some of the claims. While I understand it is not the main claim of the paper, understand the significance of these differences helps put things in perspective. I would strongly urge the authors to add this for all of their experimental results not just the ones the proposed models are outperformed by IRGANs.
- Lastly I would urge the authors to be rigid and clear in their notations. For example in the equation in section 4.5, "o'" occurs out of nowhere.