Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) (Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently higher sample complexity ) and/or model capacity (Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view of that, give a classification task, growing the model capacity appears to help draw a win-win between accuracy and robustness, yet at the expense of model size and latency, therefore posing challenges for resource-constrained applications. Is it possible to co-design model accuracy, robustness and efficiency to achieve their triple wins? This paper studies multi-exit networks associated with input-adaptive efficient inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks (RDI-Nets), allows for each input (either clean or adversarial) to adaptively choose one of the multiple output layers (early branches or the final one) to output its prediction. That multi-loss adaptivity adds new variations and flexibility to adversarial attacks and defenses, on which we present a systematical investigation. We show experimentally that by equipping existing backbones with such robust adaptive inference, the resulting RDI-Nets can achieve better accuracy and robustness, yet with over 30% computational savings, compared to the defended original models.
The paper exploited input-adaptive multiple early-exits, an idea drawn from the efficient CNN inference, to the field of adversarial attack and defense. It is well-motivated by the dilemma between the large model capacity required by accurate and robust classification, and the resulting model complexity as well as inference latency. 
Overall, this paper presents an interesting perspective, with strong results. The usage of input-adaptive inference reduces the average inference complexity, without conflicting the "larger capacity" assumption for co-winning robustness and accuracy. 
Since no literature has discussed the attacks for a multi-exit network, the authors constructed three attack forms, and then utilized adversarial training to defend correspondingly. The design of Max-Average Attack is particularly smart - to balance between "benefiting all" and "maximally boosting one" (its result is also convincingly good).
The authors presented three groups of experiments, from relatively heavy networks (ResNet38), to very compact ones (MobileNet-V2). It is especially meaningful to see their strategy work on MobileNet too (though the computational saving is a bit less, no surprise). The authors also did due diligence in ablation study and comparing with recent alternatives.
Several points that could be addressed to potentially improve the paper:
- The authors want to make it clearer that: their "triple win" is not about constructing a light-weight model that is both accurate and robust. It's instead about given an accurate + robust, yet heavy-weight model, how to reduce its AVERAGE computational load per sample inference, by routing "easier" examples to earlier exits. 
- Can the authors think of and construct more diverse and stronger attacks for RDI-Nets? For example, it would be interesting to attacking RDI-Nets (e.g., defended by Max-Average) with randomized weighted combinations of single attacks? 
Note that, at inference time, the same "randomized combination" cannot be also adopted as defense, because an input always wants to exit the earliest possible for efficiency gains.
- The advantage over ATMC is not obvious: slightly lower TA, slightly higher ATA, and slightly more parameters. Could the authors try to align their parameters more closely (to the extent possible)?
- A missing related work: "Shallow-Deep Networks: Understanding and Mitigating Network Overthinking", ICML 2019. It also discussed how to append early exits to pre-trained backbones.