Convolutional neural networks (CNN) have become the most successful and popular approach in many vision-related domains. While CNNs are particularly wellsuited for capturing a proper hierarchy of concepts from real-world images, they are limited to domains where data is abundant. Recent attempts have looked into mitigating this data scarcity problem by casting their original single-task problem into a new multi-task learning (MTL) problem. The main goal of this inductive transfer mechanism is to leverage domain-specific information from related tasks, in order to improve generalization on the main task. While recent results in the deep learning (DL) community have shown the promising potential of training task-specific CNNs in a soft parameter sharing framework, integrating the recent DL advances for improving knowledge sharing is still an open problem. In this paper, we propose the Deep Collaboration Network (DCNet), a novel approach for connecting task-specific CNNs in a MTL framework. We define connectivity in terms of two distinct non-linear transformation blocks. One aggregates taskspecific features into global features, while the other merges back the global features with each task-specific network. Based on the observation that task relevance depends on depth, our transformation blocks use skip connections as suggested by residual network approaches, to more easily deactivate unrelated task-dependent features. To validate our approach, we employed facial landmark detection (FLD) datasets as they are readily amenable to MTL, given the number of tasks they include. Experimental results show that we can achieve up to 24.31% relative improvement in landmark failure rate over other state-of-the-art MTL approaches. We finally perform an ablation study showing that our approach effectively allows knowledge sharing, by leveraging domain-specific features at particular depths from tasks that we know are related.
This paper proposes a multi-pathway neural network for facial landmark detection with multitask learning. In particular, each pathway corresponds to one task, and the intermediate features are fused at multiple layers. The fused features are added to the task-specific pathway using a residual connection (the input of the residual connection are the concatenation of the task-specific features and the fuse features). The residual connection allows each pathway to selectively use the information from other pathways and focus on its own task.
This paper is well written. The proposed neural network architectures are reasonable. 
The residual connection can help each pathway to focus on its own task (suggested by Figure 8). This phenomenon is not guaranteed by the training objective but happens automatically due to the architecture, which is interesting. 
The proposed model outperforms several baseline models. On MTFL, when using the AlexNet, the improvement is significant; when using the ResNet18, the improvement is encouraging but not so significant. On AFLW (trained on MTFL), the improvements are significant in both cases. 
What is missing is the comparison with other methods (besides the baseline). For examples, it will be helpful to compare with existing non-multitask learning methods, like TCDCN (Zhang et al., 2014) (it seems to achieve 25% failure rate on AFLW, which is lower than the numbers in Figure 5), and  multi-task learning method, like MTCNN (Zhang et al., 2016). It is important to show that proposed multitask learning method is useful in practice. 
In addition, many papers take the average error as the performance metric. Providing results in the average error can make the experiments more comprehensive.
The proposed architecture is a bit huge. It scales linearly with the number of tasks, which is not quite preferable. It is also not straightforward to add new tasks to finetune a trained model. 
In Figure 5 (left), it is a bit weird that the pretrained model underperforms the nonpretrained one. 
I am likely to change the rating based on the comparison with other methods.