Understanding object motion is one of the core problems in computer vision. It requires segmenting and tracking objects over time. Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time. We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation. Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth embeddings over time. Further, our model also estimates monocular depth, with a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring a time-consistent embedding. Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.
This paper propose a video instance embedding loss for jointly tackling the instance tracking and depth estimation from self-supervised learning. 
Pros:
1: I think the method is moving towards the right direction that 3d geometry and 2d instance representation should be considered jointly under the scenario of video learning.
2: The video instance embedding loss is also making sense as an extension of image instance embedding.
Cons: 
1: I think the major argument I have is this method is lack of technical novelty, since it is straight forward to adopt the loss of  Brabandere et.al 2017 to video cases for including pixels in the same group under ground truth tracking, and the self-supervised loss is exactly the same as previous methods. The fusion between depth and segments are relatively weak since it just ask the embedding to also decode depth, is there any further analysis of visual effect of explaining where the depth helps segments? 
2: In the experiments, the baseline for comparison over MOTS is fairly old, and I think it makes sense to include the number of MOTS paper, which is currently hard to align with that shown in the paper.  In Tab.2, the author only highlight the improved motion metric, while in per-frame AP the results are actually lower than the baselines. It also needs to be well explained. 
3: The paper claims `"it generates temporal consistent segmentation " (which is not guaranteed, maybe just statistically better but not exact). 
Overall, in my opinion I suggest it to be a workshop paper, but the contribution is somehow not significant for a major publication.