We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction-membership classification and API watermarking-which while successful against some adversaries can also be circumvented by more clever ones.
This paper studies the effectiveness of model extraction techniques on large pretrained language models like BERT. The core hypothesis of the paper is that using pretrained language models, and pretrained contextualized embeddings, has made it easier to reconstruct models using model stealing/extraction methods. Furthermore, this paper demonstrates that an attacker needn't have access to queries from the training set, and that using random sequences of words as a query to the "victim" model is an effective strategy. They authors also show that their model stealing strategies are very cost effective (based on Google Cloud compute cost). 
The basic set up of their experiments has a fine-tuned BERT model as the victim model, and a pre-trained BERT model as a the attacker model. The attacker model is assumed to not have access to the training set distribution and the queries are randomly generated. There are 2 strategies for query generation (with additional task specific heuristics): 1) randomly selecting words from WikiText-103, and 2) randomly selecting sentences or paragraphs from WikiText-103. The victim model is passed a generated query and the attacker model is fine-tuned using the output from the victim model. Overall, this paper find that this simple strategy for query generation is effective on 4 different datasets: SST2, MNLI, SQuAD 1.1 and BooolQ. The method is also cost-effective, a few hundred dollars depending on the dataset and the number of queries used to train the attacker model. 
The paper also present some analysis. They find that queries with higher agreement across victim models (5 BERTs with different random seeds) also leda to better results for the attacker model. The authors also run some experiments with humans to test the interpretability of the queries they generated. They collect annotations on SQuAD using questions that were generated with the WIKI and RANDOM strategies (they also compare highest agreement and lowest agreement queries), and also collect a control with the original SQuAD questions. While this is an interesting analysis to present, showing that most of the generated queries are nonsensical to humans and there is low inter-annotator agreement, I have an issue with the experimental procedure here: the victim model is fine-tuned on the original data, therefore it has picked up some of the data heuristics used to generate the queries, the annotators are not trained on, or shown any of the original examples (there is a control run, but these are presumably a separate set of annotators).  Through interviews, the authors learn that the annotators were using word overlap heuristics, but perhaps training the annotators on a small set of the original data would draw a closer example to the victim model. Either way, while this is an interesting result, it seems a bit misplaced in this paper. I'm not sure this human annotation experiment is contributing in any real way to the core thesis of the paper.
The authors also test the results of having a mismatch between the victim and attacker model. They consider the mismatch of BERT-base and BERT-large models. They conclude that "attackers can maximize their accuracy by fine-tuning more powerful language models, and that matching architectures is a secondary concern." This conclusion feels like a bit of a stretch. I would suggest that the authors add another few rows of experiments comparing less similar model architectures.
The paper's finding that  a model trained from scratch, QANet on SQuAD, suffers significantly without access to the training set inputs is strong supportive evidence for their hypothesis that using pretrained language models has made model extraction easier. 
The authors also present a few defense strategies, membership inference, implicit membership classification, and watermarking. They also discuss the limitations of these strategies and do not claim to have solved the problem at hand.
Overall, I think this paper makes a useful  contribution to the field and I would accept this paper. While I have a couple of issues with some of the experiments (human evaluation and architecture mismatch), I think this paper is thorough and the experiments are well presented. This is the first paper, to the best of my knowledge, showing the efficacy of model extraction of large pretrained language models using rubbish/nonsensical inputs.