Knowledge extraction techniques are used to convert neural networks into symbolic descriptions with the objective of producing more comprehensible learning models. The central challenge is to find an explanation which is more comprehensible than the original model while still representing that model faithfully. The distributed nature of deep networks has led many to believe that the hidden features of a neural network cannot be explained by logical descriptions simple enough to be understood by humans, and that decompositional knowledge extraction should be abandoned in favour of other methods. In this paper we examine this question systematically by proposing a knowledge extraction method using M-of-N rules which allows us to map the complexity/accuracy landscape of rules describing hidden features in a Convolutional Neural Network (CNN). Experiments reported in this paper show that the shape of this landscape reveals an optimal trade off between comprehensibility and accuracy, showing that each latent variable has an optimal M-of-N rule to describe its behaviour. We find that the rules with optimal tradeoff in the first and final layer have a high degree of explainability whereas the rules with the optimal tradeoff in the second and third layer are less explainable. The results shed light on the feasibility of rule extraction from deep networks, and point to the value of decompositional knowledge extraction as a method of explainability.
The paper proposes to rewrite each neuron of a neural network as a M-of-N decision rule. An measure of rule complexity (which takes into the account the number of terms in the rule) is proposed, and an approximate rule induction algorithm which binarizes the neurons using an information gain criterion is provided.
The paper gives no evaluation of the accuracy of extracted rules on a test set. Instead, the fidelity to network is evaluated on the train set for individual neurons (see below).
Since even for moderately-sized networks the method would result in lots of rules, the authors propose instead to use the proposed algorithm and complexity criterion as a tool for understanding the complexity of concepts detected by a layer. The results, gathered in Figure 1, suggest that for some layers complex rules do approximate the behavior of neurons, while for other layers a neuron can't be replaced with a single but complex M-of-N rule.
The ideas presented in this paper seem rudimentary and require further exploration before being publishable. First of all, the main result of complexity-vs-layer doesn't differentiate between failures of the approximate rule induction algorithm (the terms in rules are considered for inclusion in a single order, reducing the search space) and the genuine complexity of the rules - this can be verified by evaluating a more exhaustive algorithm on at leas a few neurons (not necessarily on all of them).
Second, if the rules are extracted in a layerwise fashion,  their errors accumulate for deeper layers. However, Algorithm 1 suggests that each neuron is replaced by a rule independently from others, and moreover that it requires the true value of the neurons in the layer below, not of their rule counterparts. This means that the rules can't be combined and explains why the paper doesn't provide any measure for aggregate rule accuracy.
Similarly, the robustness of rules to adversarial examples is meaningless - it seems that applying the rules results in a system which is less accurate (only 8/10 of rules mimick their neurons with no rule complexity penalty) overall, but also makes fewer adversarial examples.
Minor remarks.
Please don't ever produce Figures such as Figure 1: no legend (description in text), color selection is not black and white friendly, font is so small that the axis labels are hard to read... In fact, the poor quality of the Figure by itself should be sufficient  to reject the paper for not abiding to the author guidelines (sec. 4.3: All artwork must be neat, clean, and legible.)