With the increasing adoption of Deep Learning (DL) models in various applications, testing DL models is vitally important. However, testing DL models is costly and expensive, especially when developers explore alternative designs of DL models and tune the hyperparameters. To reduce testing cost, we propose to use only a selected subset of testing data, which is small but representative enough for quick estimation of the performance of DL models. Our approach, called DeepReduce, adopts a two-phase strategy. At first, our approach selects testing data for the purpose of satisfying testing adequacy. Then, it selects more testing data in order to approximate the distribution between the whole testing data and the selected data leveraging relative entropy minimization. Experiments with various DL models and datasets show that our approach can reduce the whole testing data to 4.6% on average, and can reliably estimate the performance of DL models. Our approach significantly outperforms the random approach, and is more stable and reliable than the state-of-the-art approach.
The paper presents a new approach to create subsets of the testing examples that are representative of the entire test set so that the model can be tested quickly during the training and leaving the check on the full test set only at the end to validate its validity.
The key idea is to create the smaller possible subset with the same or similar coverage (in the paper the neurons coverage is considered) and output distribution, maintaining the difference below a (small) threshold.
In particular, the output distribution is approximated by dividing the output range of each neuron into K intervals. In this way, an estimate of the output distribution is considered during the extraction of the representative subset of the testing data.
The whole process is divided into two phases, the first is to create a first subset using HGS, the second refines this subset in order to achieve the desired precision in the output distribution.
The paper also presents a good experimental campaign that shows good performances.
The paper is really well written and enjoyable, clear in its description and in the objectives it aims to achieve. To improve the readability a bit further, I would suggest trying to move equation 1 after or close to its reference, or at least to describe before it what KL is.
Moreover, a similar problem is present for TI, which is introduced in Algorithm 1 and described later. I would add, for example in the input section of the algorithm, a sentence stating that TI is used in getCandidate and described later.
The first sentence on page 5 seems to be incomplete as it is written. I would suggest rephrasing the sentence.
In Algorithm 2, about the two "for" cycles for i in 1,m and foreach k in 2,K, I suggest unifying them and use the same cycle (only for reasons of readability). Moreover, I think that using braces instead of parentheses would be more correct in these cycles.
It is interesting to note that in the first line for VGG19, last column of Table 3, the accuracy is lower with KL < 0.001 than with KL < 0.005. I would expect monotonicity here. Do the authors have any idea of the reasons for this?
Finally, it would be interesting to know the runtime to obtain the subsets of the test data of Table 2 required by the considered systems.
Typos:
On page 6, penultimate paragraph, the word "the" is repeated twice in the parentheses.
On page 8, at the beginning of the first sentence after Table 3, there is a comma that seems to be useless after the word "that".
On page 8, "When the termination crite gets stricter", crite should be corrected in criterion.
There is a typo in the README of the github project linked in the paper: "coveraeg data" instead of "coverage data".