As a concise and classic framework for object detection and instance segmentation, Mask R-CNN achieves promising performance in both two tasks. However, considering stronger feature representation for Mask R-CNN fashion framework, there is room for improvement from two aspects. On the one hand, performing multi-task prediction needs more credible feature extraction and multi-scale features integration to handle objects with varied scales. In this paper, we address this problem by using a novel neck module called SA-FPN (Scale Aware Feature Pyramid Networks). With the enhanced feature representations, our model can accurately detect and segment the objects of multiple scales. On the other hand, in Mask R-CNN framework, isolation between parallel detection branch and instance segmentation branch exists, causing the gap between training and testing processes. To narrow this gap, we propose a unified head module named EJ-Head (Effective Joint Head) to combine two branches into one head, not only realizing the interaction between two tasks, but also enhancing the effectiveness of multitask learning. Comprehensive experiments show that our proposed methods bring noticeable gains for object detection and instance segmentation. In particular, our model outperforms the original Mask R-CNN by 1˜2 percent AP in both object detection and instance segmentation task on MS-COCO benchmark.
The paper proposes several modifications to the Mask R-CNN model:
1. a variation of Feature Pyramid Networks (FPN): SA-FPN which merges features top-down and bottom-up.
2. the "effective joint head" (EJ-Head) which consists of
    a) moving the segmentation head behind the detection one,
    b) Doubling resolution of RoI crops, calling this "enriched feature".
    c) adding Boundary Refinement (really just a residual block).
Experiments show slightly improved scores on MS-COCO
I vote to reject this paper.
First of all, it is badly written. Not only are grandiose formulations of "smart framework", "innovative way", "we slickly mix" not professional, but the whole write-up of the method is very confusing. After reading it multiple times, I am still not 100% sure I fully understood the SA-FPN.
Second, it is not clear to me that SA-FPN is really novel, the original FPN paper already compared top-down and bottom-up performances (Tables 1-3).
Then, two modifications which simply increase capacity and could explain all improved scores are not ablated: increased resolution of RoI crops, and added "boundary refinement", which is really just a residual block.
Furthermore, moving the segmentation *after* the bounding-box prediction is *not* joint prediction. If predicting the box is called p(b) and predicting the mask is called p(m), the Original Mask R-CNN does p(b|features)p(m|features), the proposed model in this paper does p(b|features)p(m|b,features), and actual joint prediction would be p(b,m|features).
Finally, I think the paper is much better suited for a conference like ICCV/ECCV or CVPR and will get better reviewers than me there.