See and Think: Disentangling Semantic Scene Completion
Semantic scene completion predicts volumetric occupancy and object category of a 3D scene, which helps intelligent agents to understand and interact with the surroundings. In this work, we propose a disentangled framework, sequentially carrying out 2D semantic segmentation, 2D-3D reprojection and 3D semantic scene completion. This three-stage framework has three advantages: (1) explicit semantic segmentation significantly boosts performance; (2) flexible fusion ways of sensor data bring good extensibility; (3) progress in any subtask will promote the holistic performance. Experimental results show that regardless of inputing a single depth or RGB-D, our framework can generate high-quality semantic scene completion, and outperforms state-of-the-art approaches on both synthetic and real datasets.
I want to like this paper. I think its well motivated and using a staged approach to this problem seems quite sensible. Additionally, they show quite good quantitative results on a challenging dataset. That being said, the model itself is not overly unique (not the worst thing) and the quantitative / qualitative results don't really cement the main purpose of the paper. Because the authors changed the network itself AND the approach, its not clear whether any superiority is due to multiple inputs (they use RGBD while other papers use RGB or D), a different network (perhaps this paper has a truly advantageous network structure) or the main idea: that predicting 2D first is better, is what really leads to good performance.  === Main Ideas === The main idea of the work is to produce 3D semantic segmentation voxel grids, but do so via a staged approach: (1) 2D semantic segmentations are first estimated. (2) these are projected into 3D space (3) the unseen (occluded) voxels are inferred.  The authors attempt three variants of this approach:  1. RGB or Depth --> Incomplete 3D --> Complete 3D 2. RGB and Depth pathways --> Incomplete 3D --> Complete 3D 3. RGB and Depth pathways --> Incomplete 3D from RGB, Incomplete 3D from depth --> Complete 3D   === Quality === I like the idea overall: by breaking up the network into components, its much easier to debug and analyze than in pure end-to-end models. That being said, I really wish the authors had more clearly validated the efficacy of each model component. NYUDV2 is, in reality, a 2D dataset with depth. So if SATNet is good at predicting filled in voxel grids, it might do one of two things well: (1) estimate 2D semantic segmentations well or (2) somehow correct mistakes made in (1) but in 3D space. Filling in occluded pixels for this problem, while providing nice qualitative results, cannot help performance, because NYUDV2 has no labels for occluded pixels.   Indeed, the clearest way to demonstrate that the proposed 3-stage approach is a good one would be to (a) show that better 2D segmentations result in better 3D results (surely this is true) or (b) that their 3D network is very good at fixing mistakes. If their model happens to do (a), then they need to clearly demonstrate this superiority in terms of 2D results and explain how they were able to obtain superior 2D results. If their model happens to do (b) this would make the paper ever more compelling.  Overall, I'm left a bit wanting in terms of the quality. I want to like the paper and I think the authors approach is a potentially very good one, but I feel that much can be done to more clearly cement the main motivation of the work (staged approach is better than end-to-end) and demonstrate why indeed this would be the case.  === Clarity ===  Overall, the paper is easy to read. Minor grammatical issues throughout but these do not impact the quality of the paper: - ln 77: "with the assist" => "with the assistance" - ln 81: "the semantic for semantic completion" => "the voxels for semantic completion" (I assume?) - ln 114: "it takes few time" => "it takes little time"  There are a number of literary choices made by the authors that are extraneous and actually make the paper less clear overall: - ln 91: This is the first time I've seen the acronym DUC. Aside from paper [38], this is quite uncommon and actually makes the paper less clear. I would recommend just referring to the Decoder Network.  - Naming / Branding the entire approach SATNet is fine. Giving names to the individual sub-components (SeeNet, ThinkNet, SeeNetFuse, ThinkNetFuse) seems extraneous and very confusing. Its far clearer to simply describe the components of the network, Figure 5 does a very good job of showing this off without the need to sub-brand parts of the network.  - ln 85: "Images have higher resolution than volumes.". This is plainly untrue. Either one can be constructed to have an arbitrarily high resolution. It happens that in practice, researchers tend to use voxel grids that have a smaller spatial resolution, mostly due to computational constraints. This might be what the authors meant?  Minor Nit: the following reference is missing: A contour completion model for augmenting surface reconstructions which does volume / voxel reconstruction, partially semantically.   === Originality === The paper is mildly original. Many works solve problems in a staged approach. To my knowledge, this is the first to do so on this particular task.  === Significance === The problem of 3D semantic scene completion is an increasingly popular one. The authors demonstrate some very promising quantitative results which would be generally interesting to the community.