Grouping-Based Low-Rank Trajectory Completion and 3D Reconstruction
Extracting 3D shape of deforming objects in monocular videos, a task known as non-rigid structure-from-motion (NRSfM), has so far been studied only on synthetic datasets and controlled environments. Typically, the objects to reconstruct are pre-segmented, they exhibit limited rotations and occlusions, or full-length trajectories are assumed. In order to integrate NRSfM into current video analysis pipelines, one needs to consider as input realistic -thus incomplete-tracking, and perform spatio-temporal grouping to segment the objects from their surroundings. Furthermore, NRSfM needs to be robust to noise in both segmentation and tracking, e.g., drifting, segmentation "leaking", optical flow "bleeding" etc. In this paper, we make a first attempt towards this goal, and propose a method that combines dense optical flow tracking, motion trajectory clustering and NRSfM for 3D reconstruction of objects in videos. For each trajectory cluster, we compute multiple reconstructions by minimizing the reprojection error and the rank of the 3D shape under different rank bounds of the trajectory matrix. We show that dense 3D shape is extracted and trajectories are completed across occlusions and low textured regions, even under mild relative motion between the object and the camera. We achieve competitive results on a public NRSfM benchmark while using fixed parameters across all sequences and handling incomplete trajectories, in contrast to existing approaches. We further test our approach on popular video segmentation datasets. To the best of our knowledge, our method is the first to extract dense object models from realistic videos, such as those found in Youtube or Hollywood movies, without object-specific priors.
The paper introduces a new method for non-rigid structure from motion (NRSfM). Existing work on NRSfM typically make assumptions about their inputs, and usually require complete point trajectories, perfect segmentations, or strong object-specific shape priors. The proposed method overcomes these limitations. The method achieves competitive results on an existing synthetic NRSfM metric, and is applied to real-world videos.Quality: Overall the quality is acceptable, but there are some concerns about the experiments that should be addressed: - The only quantitative comparison of the proposed method with existing methods is on the synthetic dataset introduced in [11], where the proposed method is provided with full-length trajectories. In this experiment the method from [11] outperforms the proposed method on all video sequences; this is somewhat concerning.- Even so, the proposed method is not directly comparable to the methods of [6], [28], and [11] since the proposed method can be used even in the presence of incomplete trajectories and imperfect segmentations. The authors claim that these benefits make their method useful for real-world video sequences, but there is no quantitative evaluation of these claims. The authors provide reconstruction results of their method on example videos from the VSB100 and Moseg datasets, and the reader is expected to qualitatively assess these results. A quantitative metric for evaluating these results would be much more convincing.Clarity: Overall the paper does a good job explaining the method and comparing it to existing work. The description of the method is mostly clear up to line 206; after that the explanation becomes more dense and harder to follow. Some specific comments:- Why are Equations 5 and 6 solved via successive subproblems? Is this standard in the literature? Would it be possible to optimize jointly over all variables using some convex relaxation of Equations 5 and 6?- Typo on line 232: W = UDV^T = UDV^T = …- The Euclidean upgrade step is confusing. This could be a result of my unfamiliarity with the literature, but this section is difficult for me to follow. Why is the rank of W truncated? Why is orthogonality of \hat{R}_{2t-1}G and \hat{R}_{2t}G desirable?- You claim that your work shows that a rank 3 decomposition of W is better than the standard rank 3K decomposition used in prior work. Do you have any explanation for why this might be the case? Do higher values of K_r lead to overfitting? If so, could this effect be mitigated somehow?- What is the intuition behind minimizing \|S^V\|_\ast? My (possibly incorrect) interpretation is that when S^V is low rank, the object can be described by a small number of planar regions which do not deform too much between frames. Is really a type of shape or deformation prior? If so how does it differ from the shape priors used in other NRSfM methods?Originality: I am not very familiar with existing SfM methods or literature, so it is hard for me to accurately assess the originality of the proposed method.Significance: Nonrigid structure from motion from real-world monocular video sequences is an an important problem. The method proposed by this paper claims to make sizeable advances toward solving this problem. If these claims are valid then this paper is quite significant.