Recent work has introduced attacks that extract the architecture information of deep neural networks (DNN), as this knowledge enhances an adversary's capability to conduct attacks on black-box networks. This paper presents the first in-depth security analysis of DNN fingerprinting attacks that exploit cache side-channels. First, we define the threat model for these attacks: our adversary does not need the ability to query the victim model; instead, she runs a co-located process on the host machine where the victim's deep learning (DL) system is running and passively monitors the accesses of the target functions in the shared framework. Second, we introduce DeepRecon, an attack that reconstructs the architecture of the victim network using the internal information extracted via Flush+Reload, a cache side-channel technique. Once the attacker observes function invocations that map directly to architecture attributes of the victim network, the attacker can reconstruct the victim's entire network architecture. In our evaluation, we demonstrate that an attacker can accurately reconstruct two complex networks (VGG19 and ResNet50) having observed only one forward propagation. Based on the extracted architecture attributes, we also demonstrate that an attacker can build a meta-model that accurately fingerprints the architecture and family of the pretrained model in a transfer learning setting. From this meta-model, we evaluate the importance of the observed attributes in the fingerprinting process. Third, we propose and evaluate new framework-level defense techniques that obfuscate our attacker's observations. Our empirical security analysis represents a step toward understanding DNNs' vulnerability to cache side-channel attacks.
This paper considers the problem of fingerprinting neural network architectures using cache side channels. In the considered threat model, the attacker runs a process co-located with the victim's, and uses standard FLUSH+RELOAD attacks to infer high-level architectural information such as the number and types of layers of the victim's ML model. The paper concludes with the discussion of some "security-through-obscurity" defenses.
I don't quite understand the threat model considered in this paper. The main motivating factor given by the authors for uncovering model architecture details is for facilitating black-box attacks against ML models (e.g., for adversarial examples or membership inference). 
Yet, in the case of adversarial examples for instance, knowledge of the architecture is often considered a given as keeping it secret has very little influence on attacks. There are black-box attacks that require no knowledge of the architecture and only a few queries (e.g., Black-box Adversarial Attacks with Limited Queries and Information, Ilyas et al., ICML'18). 
So overall, learning such coarse-grained features about a model just doesn't seem particularly useful, especially since architecture-level details are often not considered private or secret to begin with.
After architectural details have been extracted, the end-goal attacks on ML models considered by the authors (e.g., model stealing, adversarial examples, etc.) require query access anyways. Thus, additionally assuming co-location between the adversary and the victim's model seems to unnecessarily strengthen the attacker model.
Maybe the most interesting scenario to consider for cache side-channels in ML is when ML models are run on trusted hardware (e.g., Oblivious Multi-Party Machine Learning on Trusted Processors, Ohrimenko et al.; or this work also submitted to ICLR: https://openreview.net/forum?id=rJVorjCcKQ).
Cache side channels are much more relevant to that threat model (i.e., ML code running in a trusted hardware enclave hosted by a malicious party). And indeed, there have been many cache side-channel attack papers against trusted hardware such as Intel's SGX (e.g., Software Grand Exposure: SGX Cache Attacks Are Practical, Brasser et al.)
But given what we know about the strength of these cache side channel attacks, one would expect to be able to extract much more interesting information about a target model, such as its weights, inputs or outputs. In the above trusted hardware scenario, solely extracting architecture-level information would also not be considered a very strong attack, especially since coarse-grained information (e.g., a rough bound on the number of layers), can be trivially obtained via timing side channels.
Minor comments:
- In the introduction, you say that white-box attacks for adversarial examples are rendered ineffective by gradient masking. This isn't true in general. Only "weak" white-box attacks can be rendered ineffective this way. So far, there are no examples of models that resist white-box attacks yet are vulnerable to black-box attacks.
- What exactly causes the cache-level differences you observe? Can you give some  code examples in the paper that showcase what happens? Are the TensorFlow code lines listed in Table 1 from a specific commit or release?
- The defenses discussed in Section 5 are all forms of "security through obscurity" that seem easily defeated by a determined attacker that adapts its attack (and maybe uses a few additional observations).
--REVISION--
I thank the authors for their rebuttal and clarifications on the threat model and end goals of their attacks. I remain somewhat unconvinced by the usefulness of extracting architectural information. For most of the listed attacks (e.g., building substitute models for adversarial examples, or simply for model extraction) it is not clear from prior work that knowledge of the architecture is really necessary, although it is of course always helpful to have this knowledge. As I mentioned in my review, with current (undefended) ML libraries, it should be possible to extract much more information (e.g., layer weights) using cache side channels.