Deep neural networks achieve remarkable performance in many computer vision tasks. For most networks in semantic segmentation (seg) and object detection (det) tasks, the backbone directly reuses the network manually designed for classification tasks. Utilizing a network pre-trained on ImageNet as the backbone has been a popular practice for seg/det tasks. However, because of the gaps between different tasks, adapting the network to the target task brings performance promotion. Some recent neural architecture search (NAS) methods search for the backbone of seg/det networks. ImageNet pre-training of the search space representation or the searched network bears huge computational cost. In this paper, we propose a fast neural network adaptation (FNA) method, which can adapt both the architecture and parameters of a manually designed seed network to the new seg/det tasks efficiently. A parameter remapping mechanism is designed to accelerate the whole adaptation process which takes full advantage of the knowledge from the seed network. The NAS method is utilized for architecture adaptation. In experiments, we conduct network adaptation on the MobileNetV2 network to both seg and det tasks. FNA demonstrates clear performance gains compared with both manually and NAS designed networks. The total computational cost of FNA is significantly less than many SOTA seg/det NAS methods: 1737× less than DPC, 6.8× less than Auto-DeepLab and 7.4× less than DetNAS.
In this paper, the authors take a MobileNet v2 trained for ImageNet classification, and adapt it either (i) semantic segmentation on Cityscapes, or (ii) object detection on COCO. They do this by first expanding the network into a "supernet" and copy weights  in an ad-hoc manner, then, they perform DARTS-style architecture search before fine-tuning for the task at hand.
There is no TLDR for this paper, and I must admit, on reading the abstract and introduction I wasn't entirely sure what this paper was doing at first. Perhaps I was being slow.
From a narrative perspective, one of the main selling points is not needing to perform any expensive ImageNet pre-training; however, a pre-trained MobileNetv2 is being utilised. While this was off-the-shelf, it still incurred an initial training cost, so it isn't really fair in e.g. Table 4 to put pre-training cost as zero. On a related note, the authors write that this network is used for its "generality". I'd argue that MobileNetv2 is a highly engineered network specialised for mobile computation; a standard ResNet-50 would be more general really.
I would like to see a comparison to a random search, as there are several papers (https://arxiv.org/abs/1902.07638, https://arxiv.org/abs/1902.08142) indicating that this is a very strong baseline. 
As mentioned earlier, the choices for remapping weights seem very ad-hoc. I can't really tell what's going on in Table 5 (why is PR in the NE and PA row?) so the ablation study of how effective this weight mapping is lost on me.  The stuff in Table 6 is pretty interesting however, if convoluted.
I find the odd choices of hyperparameters (tau as 45, gamma as 10, lambda as 9e-3) rather alarming. How important are these? Would this technique work under any other circumstances?
Error bars would be a welcome inclusion, particularly in Table 3 where you have 0.1% separating FNA and MNasnet-92. I appreciate that this can be expensive however.
Pros:
- Some promising results
- Good figures
Cons:
- Ad-hoc design choices
- Not a fair comparison regarding pre-training. 
- Very specific to one network choice
- Lack of error bars or comparison to random search.
I am giving this paper a weak reject, as there is insufficient experimental evidence that the technique works, or generalises beyond Mobilenetv2. I am also concerned about the ad-hoc hyperparmaters or weight-mapping. A comprehensive ablation study, along with error bars, and another choice of seed network would do much to strengthen this paper.