Saliency methods attempt to explain a deep net's decision by assigning a score to each feature/pixel in the input, often doing this credit-assignment via the gradient of the output with respect to input. Recently Adebayo et al. (2018) questioned the validity of many of these methods since they do not pass simple sanity checks, which test whether the scores shift/vanish when layers of the trained net are randomized, or when the net is retrained using random labels for inputs. We propose a simple fix to existing saliency methods that helps them pass sanity checks, which we call competition for pixels. This involves computing saliency maps for all possible labels in the classification task, and using a simple competition among them to identify and remove less relevant pixels from the map. Section 3 provides a theoretical justification to support our method and indicates when it can be applied, and its performance is empirically demonstrated on several popular methods.
I Summary
The paper directly answers two sanity checks for saliency maps proposed by Adebayo et al (2018): 1. randomizing the weights of a model to prove that the input's resulting saliency map is different from a trained model' saliency map.  2. randomizing the inputs' labels to make the same proof. The authors propose a "competitive version of saliency method" which uses the saliency scores of every pixel for each labels and zero out: positive scored pixels that would not be maximal for the predicted class and negative scored pixels that are not minimal for the worst predicted label.
Overall the method solves the aforementioned sanity checks, the authors claim it also generates more refined saliency maps.
II Comments
1. Content
The paper can be hard to read, due to multiple writing mistakes, abrupt phrasing, not well-articulated sentences. However, the idea is easy to understand and interesting but the contribution does not seem strong enough in its actual state. 
My main concern is that the method seems to be designed only to answer the sanity checks: the resulting saliency maps can hardly be seen as more informative as other existing methods (eg figure 1). Quantitative measures (ROAR & KAR, Hooker et al. 2018) or surveys to show that the newly obtained saliency maps are more refined or help to best localize regions of interest would be a big bonus.
2. Writing
The paper comports numerous typos, those do not impact the score of the review except if the sentence is not understandable. Please see the following points as support to improve the clarity of the paper.
- Abstract last sentence: "Some theoretical justification is provided" -> "Some" is vague and makes your claim less credible -> "theoretical justifications are given in the last paragraph to support our method..."
- Intro 
   paragraph 2 first sentence lack some words, l 2 product -> a product
   "See paper XX et al" -> "As in XX et al, we can see that" or "As stated in XX et al", "See" is too familiar, formalizing the phrasing gives more credibility to your work
- Related work
  "To give an example, does the map change a lot if we blank out or modify a portion of the image that humans find insignificant Kim et al. (2019)? " This is not very well articulated, "a lot" is vague and a little familiar, "significantly" could be used here. Moreover, the citation is a little abrupt "as we can see in XX" would work better 
Little typo on etc..
"fare best" -> far better? The wording is still vague, it would help to add a quantitative measure
that's -> that is
- Section 3
"This figure highlights" -> which figure? (I think you just missed citing the fig here)
- Section 4
First sentence: Why is it a good idea? The claim is a little abrupt and could be detailed a little more
"destroy the saliency map" -> destroy is a very strong word
"These random variables are complicated." -> This statement seems a little out of place and abrupt
"some constants" -> "constants" (too vague otherwise as before)
- Subsection 4.1
"randomly sampling a few such methods" I believe there is a typo?
"See figure 3" is abrupt as a sentence itself "as you can see in figure 3 bla bla"
Figure 4 The image is small and hard to see on printed paper (same for the images in the appendix, they could be stacked over multiple lines instead of just one horizontal row)
Definition 2 punctuation at the end
- Section 5
"The available code for these maps is slow, and computing even gradient for all 1000 ImageNet labels can be rather slow." What is the aim of this sentence?
- subsection 5.3 
lables -> labels
III Conclusion
The method itself is interesting, it would be interesting to see more qualitative results on the obtained saliency map itself: Does it produce more information? Is it more meaningful etc. Because as of now, it only seems to answer the two aforementioned sanity checks.  As for the writing, it is not always clear and can impede the understanding of the paper. I would be glad to change my review if those points are addressed.