We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.
This paper discusses the phenomena of “neural brainwashing”, which refers to that the performance of one model is affected via another model sharing model parameters. To solve the issue, the authors derived a new loss out from maximizing the posterior of the parameters. With the new loss, the neural brainwashing is largely diminished. 
The derived new loss looks meaningful to me and I think this is a valuable work for handling the weights coadaptation between two neural models, which with no doubt will bring great interests within the community of neural architecture search.
Here are some comments on the aspects that this paper can be improved: 
1)	A very important related work [1] is missed in this paper. [1] discussed the properties of “one-shot model”, which means that several different architectures are unified into the same model by sharing model weights. Furthermore, [1] discussed “neural brainwashing” (although not with the same name) and how to handle it in a very simple way (by randomly dropping path). This definitely should be a baseline to compare with. In addition, a very recent work [2] also leverages model sharing to conduct neural architecture search.
2)	Although I understand that to improve accuracy of NAS is not the main goal of this paper, the baseline number to be improved over is too weak. For example, 4.87 of CIFAR10 in ENAS. Per my own hands on experience, it does not need too many hyperparameter tuning of ENAS to obtain < 4% error rate. Please provide more convincing baseline numbers and supporting evidences of the better performance of WPL in NAS.
[1] Bender, Gabriel, et al. "Understanding and simplifying one-shot architecture search." International Conference on Machine Learning. 2018.
[2] Luo, Renqian, et al. "Neural architecture optimization." NIPS (2018).