This paper focuses on valuating training data for supervised learning tasks and studies the Shapley value, a data value notion originated in cooperative game theory. The Shapley value defines a unique value distribution scheme that satisfies a set of appealing properties desired by a data value notion. However, the Shapley value requires exponential complexity to calculate exactly. Existing approximation algorithms, although achieving great improvement over the exact algorithm, relies on retraining models for multiple times, thus remaining limited when applied to larger-scale learning tasks and real-world datasets. In this work, we develop a simple and efficient heuristic for data valuation based on the Shapley value with complexity independent with the model size. The key idea is to approximate the model via a K-nearest neighbor (KNN) classifier, which has a locality structure that can lead to efficient Shapley value calculation. We evaluate the utility of the values produced by the KNN proxies in various settings, including label noise correction, watermark detection, data summarization, active data acquisition, and domain adaption. Extensive experiments demonstrate that our algorithm achieves at least comparable utility to the values produced by existing algorithms while significant efficiency improvement. Moreover, we theoretically analyze the Shapley value and justify its advantage over the leave-one-out error as a data value measure.
In short, the paper reports improved results on a few applications of Shapley Value of data points using an introduced approximation method that is orders of magnitude faster compared to existing methods. 
I vote for rejection of this paper mainly because of two reasons. First, the contributions are not enough for this venue. The paper uses an already existing method (Jia et al. (2019a)) with the only difference being that they use it on top of learned features and therefore the main contribution seems to be the discussions in Sec4. Secondly, the paper makes technically "false" claims (as will be dicussed below). 
The positive aspects of the work are as follows: First, the elephant in the room for data valuation methods, which is assessing how good or bad a data point is, is against privacy and this work addresses this question for the first time in the (very small) literature. Secondly, the experimental results are very comprehensive and make a very good case for usefulness of the introduced algorithm. Thirdly, new useful terms for the emerging community of data valuation are introduced through the clear and well-written definitions of the paper. 
The paper mentions that the previously introduced KNN-Shapley method applied to the learned features of a deep neural network could be used as an approximation of data points' Shapley values. This is false. All data points contribute value to the feature learning part and the approximation simply ignores this crucial fact. The whole point of using Shapley values a measure of data value is its properties which are not satisfied for the "collaborative game of ML model training" by this approximation; the approximation can be heavily biased due to the fact that ignores the contributions to the feature learning (let's not forget what made deep network's desirable in the first place is their feature extracting power). One cannot use the training data to learn the feature extractor and then ignore the contributions by definition of the Shapley value being the average contribution to a random subset of data points (which means the rest of the data points are removed from the game). Or, one can do such a thing but the method cannot be called an approximation for the true Shapley Value of data points. The G-Shapley heuristic mentioned in the paper from previous work also seems to suffer from the same drawback as it is not playing the same collaborative game of training the ML model (unless one assumes that simply taking one step of the gradient for every data points would be a good approximation for a complete training!)
The experimental results are comprehensive and convincing.  The main issue is that the work discusses these experiments as if the goal of computing data value is performing such tasks (for each of which there exist simpler methods not related to data valuation). " The valuation methods often serve as a preprocessing step to filter out low-quality data,
such as mislabeled or noisy data, in a given dataset" is not a correct statement. The valuation methods serve as valuation methods which the introduced method, although providing " a valuation method", is not providing an unbiased estimate of the equitable Shapley value valuation method. For many of the provided tasks in the experiments section, previous works (Ghorbani & Zou, Jia et al 2019 b) report the same experiments as further inspections into the Shapley Value for data and not as goals of computing these computationally expensive values. 
All in all, although the paper's experimental and theoretical results are useful and interesting as the use case of "a valuation method", but is technically incorrect as calling it an approximation for Data Shapley values makes it not publishable. My score is subject to drastic change if a major rework is done to make this point clear.
A few questions and suggestions:
* One of the most striking results from the Data-Shapley works were removing points from most valuable to least valuable and looking at the accuracy drop speed. It would be very necessary and also very convincing if the introduced method is good at detecting very positive points as well as very negative points.
* An interesting empirical experiment would be to look at the Rank Correlation between the introduced approximation and other unbiased Shapley Value approximations. If the correlation is high, it means that empirically the data points contribute equally to feature learning and their value can actually be approximated just by looking at the accuracy on extracted features.
* Is any unbiased estimator of true Shapley Values is "order-preserving" by definition?
* In Sec 2 it would be more useful for the general audience to include the third Shapley property too.
* For almost all of the cases of comparison where previous methods are present for comparison, there seems to be no meaningful advantage. This makes interpreting Figures like 4b, 4c, 5b and most importantly Fig 6b. It would be necessary to add other benchmarks that are not Shapley based. For instance, for data summarization, there has been a line of work the methods of which could be used as a measure of comparison.