Relevant-features based Auxiliary Cells for Robust and Energy Efficient Deep Learning
Deep neural networks are complex non-linear models used as predictive analytics tool and have demonstrated state-of-the-art performance on many classification tasks. However, they have no inherent capability to recognize when their predictions might go wrong. There have been several efforts in the recent past to detect natural errors i.e. misclassified inputs but these mechanisms pose additional energy requirements. To address this issue, we present a novel post-hoc framework to detect natural errors in an energy efficient way. We achieve this by appending relevant features based linear classifiers per class referred as Relevant features based Auxiliary Cells (RACs). The proposed technique makes use of the consensus between RACs appended at few selected hidden layers to distinguish the correctly classified inputs from misclassified inputs. The combined confidence of RACs is utilized to determine if classification should terminate at an early stage. We demonstrate the effectiveness of our technique on various image classification datasets such as CIFAR10, CIFAR100 and Tiny-ImageNet. Our results show that for CIFAR100 dataset trained on VGG16 network, RACs can detect 46% of the misclassified examples along with 12% reduction in energy compared to the baseline network while 69% of the examples are correctly classified.
The authors introduce a mechanism for exiting the inference process early, without doing any more processing, if the network determines that there should be a classification early. The proposed mechanism is: learn separate linear classifiers on the outputs of layers in the convnet and use a decision rule based on agreement of these classifiers (and their confidences) in sequence, sometimes combined with the final output (described in Algorithm 3). The paper is poorly written (there are many simple grammar mistakes, the methods are not well motivated, and the methods are described in a confusing way), and more seriously the evaluation of the method is not complete. Details: The evaluation is not complete. The threat model that the defense is evaluated under is not realistic: it assumes that the attacker does not have knowledge of the model: *we assume that the attacker has no knowledge about the detection mechanism but has complete knowledge about the baseline DNN (white-box attack with respect to baseline DNN)* (page 9). There are a number of mechanisms that are known that work well in this assumed threat model already, and other works have indicated why this is not a realistic threat model, see: [0]. As such the evaluation of adversarial robustness does not stand since there are no best effort, dynamic attacks on the defense presented. [0] https://arxiv.org/abs/1902.06705
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Relevant-features based Auxiliary Cells for Robust and Energy Efficient Deep Learning
Deep neural networks are complex non-linear models used as predictive analytics tool and have demonstrated state-of-the-art performance on many classification tasks. However, they have no inherent capability to recognize when their predictions might go wrong. There have been several efforts in the recent past to detect natural errors i.e. misclassified inputs but these mechanisms pose additional energy requirements. To address this issue, we present a novel post-hoc framework to detect natural errors in an energy efficient way. We achieve this by appending relevant features based linear classifiers per class referred as Relevant features based Auxiliary Cells (RACs). The proposed technique makes use of the consensus between RACs appended at few selected hidden layers to distinguish the correctly classified inputs from misclassified inputs. The combined confidence of RACs is utilized to determine if classification should terminate at an early stage. We demonstrate the effectiveness of our technique on various image classification datasets such as CIFAR10, CIFAR100 and Tiny-ImageNet. Our results show that for CIFAR100 dataset trained on VGG16 network, RACs can detect 46% of the misclassified examples along with 12% reduction in energy compared to the baseline network while 69% of the examples are correctly classified.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors introduce a mechanism for exiting the inference process early, without doing any more processing, if the network determines that there should be a classification early. The proposed mechanism is: learn separate linear classifiers on the outputs of layers in the convnet and use a decision rule based on agreement of these classifiers (and their confidences) in sequence, sometimes combined with the final output (described in Algorithm 3). The paper is poorly written (there are many simple grammar mistakes, the methods are not well motivated, and the methods are described in a confusing way), and more seriously the evaluation of the method is not complete. Details: The evaluation is not complete. The threat model that the defense is evaluated under is not realistic: it assumes that the attacker does not have knowledge of the model: *we assume that the attacker has no knowledge about the detection mechanism but has complete knowledge about the baseline DNN (white-box attack with respect to baseline DNN)* (page 9). There are a number of mechanisms that are known that work well in this assumed threat model already, and other works have indicated why this is not a realistic threat model, see: [0]. As such the evaluation of adversarial robustness does not stand since there are no best effort, dynamic attacks on the defense presented. [0] https://arxiv.org/abs/1902.06705
assistant
The authors introduce a mechanism for exiting the inference process early, without doing any more processing, if the network determines that there should be a classification early. The proposed mechanism is: learn separate linear classifiers on the outputs of layers in the convnet and use a decision rule based on agreement of these classifiers (and their confidences) in sequence, sometimes combined with the final output (described in Algorithm 3). The paper is poorly written (there are many simple grammar mistakes, the methods are not well motivated, and the methods are described in a confusing way), and more seriously the evaluation of the method is not complete. Details: The evaluation is not complete. The threat model that the defense is evaluated under is not realistic: it assumes that the attacker does not have knowledge of the model: *we assume that the attacker has no knowledge about the detection mechanism but has complete knowledge about the baseline DNN (white-box attack with respect to baseline DNN)* (page 9). There are a number of mechanisms that are known that work well in this assumed threat model already, and other works have indicated why this is not a realistic threat model, see: [0]. As such the evaluation of adversarial robustness does not stand since there are no best effort, dynamic attacks on the defense presented. [0] https://arxiv.org/abs/1902.06705