How to 0wn NAS in Your Spare Time
New data processing pipelines and novel network architectures increasingly drive the success of deep learning. In consequence, the industry considers topperforming architectures as intellectual property and devotes considerable computational resources to discovering such architectures through neural architecture search (NAS). This provides an incentive for adversaries to steal these novel architectures; when used in the cloud, to provide Machine Learning as a Service (MLaaS), the adversaries also have an opportunity to reconstruct the architectures by exploiting a range of hardware side channels. However, it is challenging to reconstruct novel architectures and pipelines without knowing the computational graph (e.g., the layers, branches or skip connections), the architectural parameters (e.g., the number of filters in a convolutional layer) or the specific pre-processing steps (e.g. embeddings). In this paper, we design an algorithm that reconstructs the key components of a novel deep learning system by exploiting a small amount of information leakage from a cache side-channel attack, Flush+Reload. We use Flush+Reload to infer the trace of computations and the timing for each computation. Our algorithm then generates candidate computational graphs from the trace and eliminates incompatible candidates through a parameter estimation process. We implement our algorithm on PyTorch and Tensorflow. We demonstrate experimentally that we can reconstruct MalConv, a novel data pre-processing pipeline for malware detection, and ProxylessNAS-CPU, a novel network architecture for the ImageNet classification optimized to run on CPUs, without knowing the architecture family. In both cases, we achieve 0% error. These results suggest hardware side channels are a practical attack vector against MLaaS, and more efforts should be devoted to understanding their impact on the security of deep learning systems.
Summary
---
This paper proposes to use a computer security method, "Flush+Reload" to infer the DNN architecture of a victim in the setting where both the attacker and the victim share the same machine in a cloud computing scenario. This does not require any physical access to the machine, however it does require that a CPU is shared, and the inference of the architectural details is based on the time it takes to reload computations from cache. 
The paper is overall clear and well written.
Motivations of the paper
---
However, concerning the motivations of the paper, I'd like some clarifications. As far as I know, in the deep learning community, the most effective architectures are published and public (VGG, Inception, ResNet, Transformer...).
I am a bit confused by the sentence "As a result, in the industry such novel DL systems are kept as trade secrets or
intellectual property as they give their owners a competitive edge (Christian & Vanhoucke, 2017)." which justifies that architectures are kept secret and thus may be prone being stolen.
This US patent is public and explains the method. As far as I know, it has never been enforced. Furthermore, this patent is associated with the paper "Going deep with convolutions", Szegedy et al. which introduced the Inception architecture, is public, very well-known, and thus I do not believe anyone would have any commercial interest in stealing it. 
Furthermore, I do have the impression that the edge many companies have over their competitors is the private datasets they own much more than the architectural details.
Method and applicability
---
While the method Flush+Reload itself is not novel, its application to the DNNs case and the way to reconstruct the architecture (generating the candidates, pruning) is.
However, I do have some practical concerns about the applicability of the method.
As far as I understand, it can only work on one CPU. Most DNNs, even for inference, are run on (one or multiple) GPUs. Can the method be extended to work on GPUs?
Also, while the assumption that both the attacker and the victim use the same framework is realistic to me, I believe, they should also both use the same version of the said library, no? Otherwise some operations might be faster in some versions and slower in others, this is thus an additional and much stronger assumption to make.
At last, this would require the victim to use a public cloud service. However, as far as I know, many of the companies who could potentially design new architectures have their own private cloud. I am not certain that someone disposing of a new, private, and powerful architecture would use it on a public cloud service. 
Experiments
---
The experimental section seems very limited to me. The authors show that they are able to reconstruct perfectly 2 architectures. While this is encouraging, I would like to see the limits of the proposed method.
Why not generate N random (or not so random) architectures and try to reconstruct them? Where does the method fail, where does it succeed?
What if the victim used a custom layer that the method could not recover? Does it still recover a similar architecture?
Conclusion
---
While the paper, proposed to use Flush+Reload for recovering DNNs architectures and succeeds for at least 2 non trivial architectures, I do not recommend acceptance.
First I am concerned by the problem this paper is tackling. Can this realistically happen in a real-life scenario?
Second, I am worried that the method suffers from very strong limitations in practice (eg the usage of a CPU for both victim and attacker).
Finally, and importantly, while the experiments show some interesting first results, they are limited, I am not able to judge the strengths and weaknesses of the method, and thus I cannot assess the usefulness of the proposed method.
Note: I have to say that this paper is definitely out of my area of expertise, even though I am confident in my understanding of the paper, it may be that some of my concerns are unfounded. If this is the case I will adjust my score accordingly.