## Deep Non-Rigid Structure from Motion

## Chen Kong Carnegie Mellon University

## Simon Lucey Carnegie Mellon University

chenk@cs.cmu.edu

## Abstract

Current non-rigid structure from motion (NRSfM) algorithms are mainly limited with respect to: (i) the number of images, and (ii) the type of shape variability they can handle. This has hampered the practical utility of NRSfM for many applications within vision. In this paper we propose a novel deep neural network to recover camera poses and 3D points solely from an ensemble of 2D image coordinates. The proposed neural network is mathematically interpretable as a multi-layer block sparse dictionary learning problem, and can handle problems of unprecedented scale and shape complexity. Extensive experiments demonstrate the impressive performance of our approach where we exhibit superior precision and robustness against all available state-of-the-art works by an order of magnitude. We further propose a quality measure (based on the network weights) which circumvents the need for 3D ground-truth to ascertain the confidence we have in the reconstruction.

## 1. Introduction

Building an AI capable of inferring the 3D structure and pose of an object from a single image is a problem of immense importance. Training such a system using supervised learning requires a large number of labeled images - how to obtain these labels is currently an open problem for the vision community. Rendering [27] is problematic as the synthetic images seldom match the appearance and geometry of the objects we encounter in the real-world. Hand annotation is preferable, but current strategies rely on associating the natural images with an external 3D dataset ( e.g . ShapeNet [8], ModelNet [32]), which we refer to as 3D supervision . If the 3D shape dataset does not capture the variation we see in the imagery, it is inherently ill-posed.

Non-Rigid Structure from Motion (NRS M) offers comf puter vision a way out of this quandary - by recovering the pose and 3D structure of an object category solely from hand annotated 2D landmarks with no need for 3D supervision. Classically [6], the problem of NRS M has been f applied to objects that move non-rigidly over time such as slucey@cs.cmu.edu

Figure 1: In this paper, we want to reconstruct 3D shapes solely from a sequence of annotated images-shown on top-with no need of 3D ground truth. Our proposed hierarchical sparse coding model and corresponding deep solution outperforms state-of-the-art by an order of magnitude.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-8a1a3f22-3cd3-41c0-9a3a-7021ee2d09d7.jpg)

the human body and face. But NRS M is not restricted f to non-rigid objects; it can equally be applied to rigid objects whose object categories deform non-rigidly [19, 2, 30]. Consider, for example, the five objects in Figure 1 (top), instances from the visual object category 'chair'. Each object in isolation represents a rigid chair, but the set of all 3D shapes describing 'chair' is non-rigid. In other words, each object instance can be modeled as a deformation from its category's general shape.

NRS M is well-noted in literature as an ill-posed probf lem due to the non-rigidity. This has been mainly addressed by imposing additional shape priors, e.g . low rank [6, 10], union-of-subspaces [36, 2], and block-sparsity [18, 19]. However, low rank is only applicable to simple non-rigid objects with limited deformations and union-of-subspaces relies heavily on frame clustering which is still an open problem. Block-sparsity, where each shape can be represented by at most K bases out of L , is considered as one of the most promising assumptions in terms of modeling broad shape variations. This is because sparsity can be thought of as a union of ( L K ) subspaces constraint and an over-complete dictionary could be utilized. However, it was noted by Kong et al . [18], that searching for the best sub-

space out of ( L K ) is extremely costly and not robust. Based on this observation we propose a novel shape prior that employs hierarchical sparse coding. The introduction of additional layers, in comparison to classical single-layer sparse coding, provides a mechanism for controlling the number of active subspaces. The ability of hierarchical sparse coding to provide a model that is both expressible and robust for general NRS M forms the central thesis of our paper. f

## Contributions:

- · We propose a novel shape prior based on hierarchical sparse coding and demonstrate that the 2D projections under orthogonal cameras can be represented by the hierarchical dictionaries in a block sparse way.
- · We designed a deep neural network to approximately solve the proposed hierarchical block sparse model and show how the network architecture is derived from a classical sparse coding algorithm.
- · Finally, extensive experiments are conducted using various datasets. Both quantitative and qualitative results demonstrate our superior performance, outperforming all state-of-the-arts in the order of magnitude.

## 2. Related Work

Low-rank NRS M: f In rigid structure from motion, the rank of 3D structure is fixed as three [29] since 3D shapes remain the same between frames. Based on this insight, Bregler et al . [6] advocated that non-rigid 3D structure could be represented by a linear subspace of low rank. Dai et al . [10] developed this prior by proving that low-rank assumption itself is sufficient to address the ill-posedness of NRS M with no need of additional priors. f The lowrank assumption has also been applied temporally [3, 15] - 3D point trajectories can be represented by pre-defined ( e.g . DCT) or learned bases. Although exhibiting impressive performance, the low-rank assumption has a major drawback. The rank is strictly limited by the number of points and frames (whichever is smaller [10]). This makes low-rank NRS M infeasible if we want to solve large-scale f problems with complex shape variations when the number of points is substantially smaller than the number of frames.

Union-of-subspaces NRS M: f Inspired by an intuition that complex non-rigid deformations could be clustered into a sequence of simple motions, Zhu et al . [36] proposed to model non-rigid 3D structure by a union of local subspaces. This was later extended to spatial-temporal domain [1] and applied to rigid object category reconstruction [2]. The major difficulty of union-of-subspaces is how to effectively cluster shape deformations purely from 2D observations and how to estimate affinity matrix when the number of frame is large e.g . more than tens of thousand frames.

Sparse NRS M: f Sparse prior [18, 35, 19] is more generic than union-of-subspaces since it is equivalent to the union of all possible local subspaces. One obvious advantages of this is the large number of subspaces enables the effective modeling of a broader set of 3D structures. However, the sheer number of subspaces that can be entertained by the sparsity prior is its fundamental drawback. Since there are so many possible subspaces to choose from, the approach is sensitive to noise, dramatically limiting its applicability to 'real-world' NRS M problems. In this paper we want to f leverage the elegance and expressibility of the sparsity prior without suffering from its inherent sensitivity to noise.

## 3. Background

Sparse dictionary learning can be considered as an unsupervised learning task and divided into two sub-problems: (i) dictionary learning, and (ii) sparse code recovery. Let us consider sparse code recovery problem, where we estimate a sparse representation z for a measurement vector x given the dictionary D i.e .

$$\min _ { z } \| x - \text{Dz} \| _ { 2 } ^ { 2 } \text{ s.t.} \, \| z \| _ { 0 } < \lambda, \quad \text{ (1)}$$

where λ related to the trust region controls the sparsity of recovered code. One classical algorithm to recover the sparse representation is Iterative Shrinkage and Thresholding Algorithm (ISTA) [11, 26, 5]. ISTA iteratively executes the following two steps with z [0] = 0 :

$$\mathbf v = \mathbf z ^ { [ i ] } - \alpha \mathbf D ^ { T } ( \mathbf D \mathbf z ^ { [ i ] } - \mathbf x ), \text{ \quad \ \ } ( 2 )$$

$$z ^ { [ i + 1 ] } = \underset { u } { \arg \min } \frac { 1 } { 2 } \| u - \mathbf v \| _ { 2 } ^ { 2 } + \tau \| u \| _ { 1 }, \quad ( 3 )$$

which first uses the gradient of ‖ x -Dz ‖ 2 2 to update z [ i ] in step size α and then finds the closest sparse solution using an glyph[lscript] 1 convex relaxation. It is well known in literature that the second step has a closed-form solution using the soft thresholding operator. Therefore, ISTA can be summarized as the following recursive equation:

$$z ^ { [ i + 1 ] } = h _ { \tau } ( \mathbf z ^ { [ i ] } - \alpha \mathbf D ^ { T } ( \mathbf D \mathbf z ^ { [ i ] } - \mathbf x ) ), \quad ( 4 )$$

where h τ is a soft thresholding operator and τ is related to λ for controlling sparsity.

Recently, Papyan [25] proposed to use ISTA and sparse coding to reinterpret feed-forward neural networks. They argue that feed-forward passing a single-layer neural network z = ReLU( D x T -b ) can be considered as one iteration of ISTA when z ≥ 0 , α = 1 and τ = b . Based on this insight, the authors extend this interpretation to feedforward neural network with n layers

$$\text{cuda} \text{crowd} \text{ with } n \text{ layers} \\ \text{z} _ { 1 } & = \text{ReLU} ( \text{D} _ { 1 } ^ { T } x - b _ { 1 } ) \\ \text{z} _ { 2 } & = \text{ReLU} ( \text{D} _ { 2 } ^ { T } z _ { 1 } - b _ { 2 } ) \\ & \quad \vdots \\ \text{z} _ { n } & = \text{ReLU} ( \text{D} _ { n } ^ { T } z _ { n - 1 } - b _ { n } )$$

as executing a sequence of single-iteration ISTA, serving as an approximate solution to the multi-layer sparse coding problem: find { z i } n i =1 , such that

$$x & = \text{D} _ { 1 } z _ { 1 }, \quad \| z _ { 1 } \| _ { 0 } < \lambda _ { 1 }, z _ { 1 } \geq 0, \quad \underset { \substack { \text{$\text{$\text{$\text{$\prime$}$}$} } } { \text{$\text{$\text{$\prime$}$}$} } } \text{$\text{$\text{$\text{$\text{$\prime$}$}$} } \text{$\text{$\text{$\text{$\prime$}$} } } \text{$\text{$\text{$\text{$\prime$}$} } } \text{$\text{$\text{$\text{$\prime$}$} } } \text{$\text{$\text{$\text{$\text{$\prime$}$} } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{ $\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{ $\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text$$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{ $\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{ $\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } } \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\prime$} } } } \text{$\text{$\text{$\text{$\text{$$$

where the bias terms { b i } n i =1 (in a similar manner to τ ) are related to { λ i } n i =1 , adjusting the sparsity of recovered code. Furthermore, they reinterpret back-propagating through the deep neural network as learning the dictionaries { D i } n i =1 . This connection offers a novel breakthrough for understanding DNNs. In this paper, we extend this to the block sparse scenario and apply it to solving our NRS f Mproblem.

## 4. Deep Non-Rigid Structure from Motion

Under orthogonal projection, NRS M deals with the f problem of factorizing a 2D projection matrix W ∈ R p × 2 as the product of a 3D shape matrix S ∈ R p × 3 and camera matrix M ∈ R 3 × 2 . Formally,

$$W = \text{SM}, \quad \ \ \ \ ( 7 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

$$W = \begin{bmatrix} u _ { 1 } & v _ { 1 } \\ u _ { 2 } & v _ { 2 } \\ \vdots & \vdots \\ u _ { p } & v _ { p } \end{bmatrix}, \, S = \begin{bmatrix} x _ { 1 } & y _ { 1 } & z _ { 1 } \\ x _ { 2 } & y _ { 2 } & z _ { 2 } \\ \vdots & \vdots & \vdots \\ x _ { p } & y _ { p } & z _ { p } \end{bmatrix}, \, M ^ { T } M = \mathbf I _ { 2 }, \quad \text{(8)} \quad \text{$from} \\ \text{where $(u_{i},v_{i}),(x_{i},y_{i},z_{i})$ are the image and world coordi$-} \quad \text{and} \quad \text{$d$}.$$

where ( u , v i i ) , ( x , y , z i i i ) are the image and world coordinates of the i -th point. The goal of NRS M is to recover f simultaneously the shape S and the camera M for each projection W in a given set W of 2D landmarks. In a general NRS M including S C, this set f f W could contain deformations of a non-rigid object or various instances from an object category.

## 4.1. Modeling via multi-layer sparse coding

To alleviate the ill-posedness of NRS f M and also guarantee sufficient freedom on shape variation, we propose a novel prior assumption on 3D shapes via multi-layer sparse coding: The vectorization of S satisfies

$$\psi _ { i }$$

$$s & = \text{D} _ { 1 } \psi _ { 1 }, \quad \| \psi _ { 1 } \| _ { 0 } < \lambda _ { 1 }, \psi _ { 1 } \geq 0, \\ \psi _ { 1 } & = \text{D} _ { 2 } \psi _ { 2 }, \quad \| \psi _ { 2 } \| _ { 0 } < \lambda _ { 2 }, \psi _ { 2 } \geq 0, \\ & \quad \vdots \quad, \quad \vdots \\ \psi _ { n - 1 } & = \text{D} _ { n } \psi _ { n }, \quad \| \psi _ { n } \| _ { 0 } < \lambda _ { n }, \psi _ { n } \geq 0, \\ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot$$

where D 1 ∈ R 3 p × k 1 , D 2 ∈ R k 1 × k 2 , . . . , D n ∈ R k n -1 × k n are hierarchical dictionaries. In this prior, each non-rigid shape is represented by a sequence of hierarchical dictionaries and corresponding sparse codes. Each sparse code is determined by its lower-level neighbor and affects the nextlevel. Clearly this hierarchy adds more parameters, and thus more freedom into the system. We now show that it paradoxically results in a more constrained global dictionary and sparse code recovery.

More constrained code recovery: In a classical single dictionary system, the constraint on the representation is element-wise sparsity. Further, the quality of its recovery entirely depends on the quality of the dictionary. In our multi-layer sparse coding model, the optimal code not only minimizes the difference between measurements s and D 1 ψ 1 along with sparsity regularization ‖ ψ 1 ‖ 0 , but also satisfies constraints from its subsequent representations. This additional joint inference imposes more constraints on code recovery, helps to control the uniqueness and therefore alleviates its heavy dependency on the dictionary quality.

More constrained dictionary: When all equality constraints are satisfied, the multi-layer sparse coding model degenerates to a single dictionary system. From Equation 9, by denoting D ( ) l = ∏ l i =1 D i , it is implied that s = D D 1 2 . . . D n ψ n = D ( n ) ψ n . However, this differs from other single dictionary models [36, 37, 18, 19, 34] in terms that a unique structure is imposed on D ( n ) [28]. The dictionary D ( n ) is composed by simpler atoms hierarchically. For example, each column of D (2) = D D 1 2 is a linear combination of atoms in D 1 , each column of D (3) = D (2) D 3 is a linear combination of atoms in D (2) and so on. Such a structure results in a more constrained global dictionary and potentially leads to higher quality with lower mutual coherence [14].

## 4.2. Multi-layer block sparse coding

Given the proposed multi-layer sparse coding model, we now build a conduit from the proposed shape code { ψ i } k i =1 to the 2D projected points. From Equation 9, we reshape vector s to a matrix S ∈ R p × 3 such that S = D glyph[sharp] 1 ( ψ 1 ⊗ I 3 ) , where ⊗ is Kronecker product and D glyph[sharp] 1 ∈ R p × 3 k 1 is a reshape of D 1 [10]. From linear algebra, it is well known that AB ⊗ I = ( A ⊗ I )( B ⊗ I ) given three matrices A B , , and identity matrix I . Based on this lemma, we can derive that

$$\text{int} & S = D _ { 1 } ^ { \# } ( \psi _ { 1 } \otimes I _ { 3 } ), \quad \| \psi _ { 1 } \| _ { 0 } < \lambda _ { 1 }, \psi _ { 1 } \geq 0, \\ \psi _ { 1 } \otimes I _ { 3 } = ( D _ { 2 } \otimes I _ { 3 } ) ( \psi _ { 2 } \otimes I _ { 3 } ), \quad \| \psi _ { 2 } \| _ { 0 } < \lambda _ { 2 }, \psi _ { 2 } \geq 0, \\ & \vdots \quad, \quad \vdots \\ \psi _ { n - 1 } \otimes I _ { 3 } = ( D _ { n } \otimes I _ { 3 } ) ( \psi _ { n } \otimes I _ { 3 } ), \quad \| \psi _ { n } \| _ { 0 } < \lambda _ { n }, \psi _ { n } \geq 0. \\ \times k _ { n } & \text{Further, from Equation 7, by right multiplying the cam-}$$

Further, from Equation 7, by right multiplying the camera matrix M ∈ R 3 × 2 to the both sides of Equation 10 and

Figure 2: Deep NRS M architecture. f The network can be divided into two parts: encoder and decoder that are symmetric and share convolution kernels ( i.e . dictionaries). The symbol a × b, c → d refers to the operator using kernel size a × b with c input channels and d output channels.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-b0937ba1-a273-46f9-a0d3-76b514dd9b76.jpg)

denote Ψ i = ψ i ⊗ M , we obtain that

$$\Psi _ { 1 }$$

$$\text{dword} \, \ast \, \iota \, \sigma \, \xi \, \xi \, \sigma \, \xi \, \sigma \, \xi \, \sigma \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \sigma \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \, \xi \,                                                                                                                                                                                                        
                                                                                                                                                                                                        }                                                                                                                                                                                                        \,                                                                                                                                                                                                     
  \, \quad W = D _ { 1 } ^ { \dagger } \Psi _ { 1 }, \quad \| \Psi _ { 1 } \| _ { 0 } ^ { ( 3 \times 2 ) } < \lambda _ { 1 }, \quad \| \Psi _ { 2 } \| _ { 0 } ^ { ( 3 \times 2 ) } < \lambda _ { 2 }, \quad \| \Psi _ { 3 } \| _ { 0 } ^ { ( 3 \times 2 ) } < \lambda _ { 3 }, \quad \| \Psi _ { 4 } \| _ { 0 } ^ { ( 3 \times 2 ) } < \lambda _ { 4 }, \quad \| \Psi _ { 5 } \| _ { 6 } \| _ { 7 } \| _ { 8 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 6 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 6 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 8 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Psi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Phi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Phi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Phi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Phi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Phi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \quad \| \Phi _ { 7 } \| _ { 9 } ^ { ( 3 \times 2 ) } < \lambda _ { 9 }, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \, \cdots \$$

where ‖ · ‖ (3 × 2) 0 divides the argument matrix into blocks with size 3 × 2 and counts the number of active blocks. Since ψ i has active elements less than λ i , Ψ i has active blocks less than λ i , that is Ψ i is block sparse. This derivation demonstrates that if the shape vector s satisfies the multi-layer sparse coding prior described by Equation 9, then its 2D projection W must be in the format of multilayer block sparse coding described by Equation 11. We hereby interpret NRS M as a hierarchical f block sparse dictionary learning problem i.e . factorizing W as products of hierarchical dictionaries { D i } n i =1 and block sparse coefficients { Ψ i } n i =1 .

## 4.3. Block ISTA and DNNs solution

Before solving the multi-layer block sparse coding problem in Equation 11, we first consider the single-layer problem:

$$\min _ { Z } \| X - \text{DZ} \| _ { F } ^ { 2 } \ \text{s.t.} \ \| \text{Z} \| _ { 0 } ^ { ( 3 \times 2 ) } < \lambda. \quad ( 1 2 )$$

Inspired by ISTA, we propose to solve this problem by iteratively executing the following two steps:

$$V = \mathbf Z ^ { [ i ] } - \alpha \mathbf D ^ { T } ( \mathbf D \mathbf Z ^ { [ i ] } - \mathbf X ), \quad \ ( 1 3 ) \quad \text{NI}$$

$$\mathbf Z ^ { [ i + 1 ] } = \underset { U } { \arg \min } \frac { 1 } { 2 } \| \mathbf U - \mathbf V \| _ { F } ^ { 2 } + \tau \| \mathbf U \| _ { F 1 } ^ { ( 3 \times 2 ) }, \quad ( 1 4 ) \quad \underset { \text{$com$} } { ( e. g. }$$

where ‖ · ‖ (3 × 2) F 1 is defined as the summation of Frobenius norm of each 3 × 2 block, serving as a convex relaxation of block sparsity constraint. It is derived in [13] that the second step has a closed-form solution computing each block separately by Z [ i +1] j = ( h τ ( ‖ V j ‖ F ) / ‖ V j ‖ F ) V j , where the subscript j represents the j -th block and h τ is a soft thresholding operator. However, soft thresholding the Frobenius norms for every block brings unnecessary computational complexity. We show in the supplementary material that an efficient approximation is Z [ i +1] j = h b j ( V j ) , where b j is the threshold for the j -th block, controlling its sparsity. Based on this approximation, a single-iteration block ISTA with step size α = 1 can be represented by :

$$\cdot \quad \mathbf Z = h _ { \mathbf b } ( \mathbf D ^ { T } \mathbf X ) = \text{ReLU} ( \mathbf D ^ { T } \mathbf X - \mathbf b \otimes \mathbf 1 _ { 3 \times 2 } ), \quad ( 1 5 )$$

where h b is a soft thresholding operator using the j -th element b j as threshold of the j -th block and the second equality holds if Z is non-negative.

Encoder: Recall from Section 3 that the feed-forward pass through a deep neural network can be considered as a sequence of single ISTA iterations and thus provides an approximate recovery of multi-layer sparse codes. We follow the same scheme: we first assume the multi-layer block sparse coding to be non-negative and then sequentially use single-iteration block ISTA to solve it i.e .

$$\Psi _ { 1 } & = \text{ReLU} ( ( \text{D} _ { 1 } ^ { \ddagger } ) ^ { T } \text{W} - \text{b} _ { 1 } \otimes \text{1} _ { 3 \times 2 } ), \\ \Psi _ { 2 } & = \text{ReLU} ( ( \text{D} _ { 2 } \otimes \text{I} _ { 3 } ) ^ { T } \Psi _ { 1 } - \text{b} _ { 2 } \otimes \text{1} _ { 3 \times 2 } ), \\ & \quad \vdots$$

$$\mathfrak { x } _ { 2 } - \mathfrak { w } _ { 1 } \vee \mathfrak { w } _ { 2 } \vee \mathfrak { x } _ { 3 } / \mathfrak { x } _ { 1 } - \mathfrak { w } _ { 2 } \vee \mathfrak { x } _ { 3 } \vee 2 / \mathfrak { z } _ { 2 } \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } _ { 3 } \vee 2 / \mathfrak { z } ^ { ( 1 6 ) } \\ \Psi _ { n } = \text{ReLU} ( ( \text{D} _ { n } \otimes \text{I} _ { 3 } ) ^ { T } \Psi _ { n - 1 } - \text{b} _ { n } \otimes \text{1} _ { 3 \times 2 } ),$$

where thresholds b 1 , ..., b n are learned, controlling the block sparsity. This learning is crucial because in previous NRS M algorithms utilizing low-rank [10], subspaces [36] f or compressible [18] priors, the weight given to this prior ( e.g . rank or sparsity) is hand-selected through a cumbersome cross validation process. In our approach, this weighting is learned simultaneously with all other parameters removing the need for any irksome cross validation process. This formula composes the encoder of our proposed DNN.

Decoder: Let us for now assume that we can extract camera M and regular sparse hidden code ψ n from Ψ n by some functions i.e . M = F ( Ψ n ) and ψ n = G ( Ψ n ) , which will be discussed in the next section. Then we can compute the 3D shape vector s by:

$$\Psi _ { n - 1 } & = \text{ReLU} ( \text{D} _ { n } \psi _ { n } - \text{b} _ { n } ^ { \prime } ), \quad & \quad \\ & \quad \vdots \quad & ( \text{T} ) \quad & \quad \vdots \\ \psi _ { 1 } & = \text{ReLU} ( \text{D} _ { 2 } \psi _ { 2 } - \text{b} _ { 2 } ^ { \prime } ), \quad & \quad \vdots \\ s & = \text{D} _ { 1 } ^ { \ddagger } \psi _ { 1 }, \quad & \quad \text{Ta} \\ \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdus \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vdots \quad \vd.$$

Note we preserve the ReLU and bias term during decoding to further enforce sparsity and improve robustness. These portion forms the decoder of our DNN.

Variation of implementation: The Kronecker product of identity matrix I 3 dramatically increases the time and space complexity of our approach. To eliminate it and make parameter sharing easier in modern deep learning environments ( e.g . TensorFlow, PyTorch), we reshape the filters and features and show that the matrix multiplication in each step of the encoder and decoder can be equivalently computed via multi-channel 1 × 1 convolution ( ∗ ) and transposed convolution ( ∗ T ) i.e .

$$( \mathbf D _ { 1 } ^ { \ddagger } ) ^ { T } \mathbf W = d _ { 1 } ^ { \ddagger } * ^ { T } w, \quad \quad ( 1 8 )$$

where d glyph[sharp] 1 ∈ R 3 × × 1 k 1 × p , w ∈ R 1 × × 2 p 1 .

$$( \mathbf D _ { i + 1 } \otimes \mathbf I _ { 3 } ) ^ { T } \Psi _ { i } = \mathbf d _ { i + 1 } * ^ { T } \Psi _ { i }, \quad \ \ ( 1 9 )$$

where d i +1 ∈ R 1 × × 1 k i +1 × k i , Ψ i ∈ R 3 × × 2 k i .

$$\mathbf D _ { i } \psi _ { i } = \mathbf d _ { i } * \psi _ { i }, \quad \quad \ \ ( 2 0 ) \quad \mathbf f$$

where d i ∈ R 1 × × 1 k i × k i -1 , ψ i ∈ R 1 × × 1 k i .

Code and camera recovery: Estimating ψ n and M from Ψ n is discussed in [18] and solved by a closed-form formula. Due to its differentiability, we could insert the solution directly within our pipeline. An alternative solution is using an approximation i.e . a fully connected layer connecting Ψ n and ψ n and a linear combination among each blocks of Ψ n to estimate M , where the fully connected layer parameters and combination coefficients are learned from data. In our experiments, we use the approximate solution and represent them via convolutions, as shown in Figure 2, for conciseness and maintaining proper dimensions. Since the approximation has no way to force the orthonormal constraint on the camera, we seek help from the loss function.

Loss function: The loss function must measure the reprojection error between input 2D points W and reprojected 2D points SM while simultaneously encouraging orthonormality of the estimated camera M . One solution is to use

1 The filter dimension is height × width × # of input channel × # of output channel. The feature dimension is height × width × # of channel.

Table 1: Quantitative comparison against state-of-the-art algorithms using IKEA dataset in normalized 3D error.

| Furnitures   |   Bed |   Chair |   Sofa |   Table |   Mean |   Relative |
|--------------|-------|---------|--------|---------|--------|------------|
| KSTA [16]    | 0.069 |   0.158 |  0.066 |   0.217 |  0.128 |      12.19 |
| BMM[10]      | 0.059 |   0.33  |  0.245 |   0.211 |  0.211 |      20.12 |
| CNR [20]     | 0.227 |   0.163 |  0.835 |   0.186 |  0.352 |      33.55 |
| NLO [12]     | 0.245 |   0.339 |  0.158 |   0.275 |  0.243 |      23.18 |
| RIKS [17]    | 0.202 |   0.135 |  0.048 |   0.218 |  0.117 |      11.13 |
| SPS [18]     | 0.971 |   0.946 |  0.955 |   0.28  |  0.788 |      74.96 |
| SFC [19]     | 0.247 |   0.195 |  0.233 |   0.193 |  0.217 |      20.67 |
| OURS         | 0.004 |   0.019 |  0.005 |   0.012 |  0.01  |       1    |

spectral norm regularization of M because spectral norm minimization is the tightest convex relaxation of the orthonormal constraint [34]. An alternative solution is to hard code the singular values of M to be exact ones with the help of Singular Value Decomposition (SVD). Even though SVD is generally non-differentiable, the numeric computation of SVD is differentiable and most deep learning packages implement its gradients ( e.g . PyTorch, TensorFlow). In our implementation and experiments, we use SVD to ensure the success of the orthonormal constraint and a simple Frobenius norm to measure reprojection error,

$$L o s s = \| { \mathbf W } - { \mathbf S } \tilde { \mathbf M } \| _ { F }, \ \tilde { \mathbf M } = { \mathbf U } { \mathbf V } ^ { T }, \quad ( 2 1 )$$

where UΣV T = M is the SVD of the camera matrix.

## 5. Experiments

We conduct extensive experiments to evaluate the performance of our deep solution for solving NRS f Mand S C f problems. For quantitative evaluation, we follow the metric i.e . normalized mean 3D error, reported in [4, 10, 16, 2]. A detailed description of our architectures is in the supplementary material. Our implementation and processed data will be publicly accessible for future comparison.

## 5.1. S f C on IKEA furniture

We first apply our method to a furniture dataset, IKEA dataset [23, 31]. The IKEA dataset contains four object categories: bed, chair, sofa, and table. For each object category, we employ all annotated 2D point clouds and augment them with 2K ones projected from the 3D ground-truth using randomly generated orthonormal cameras 2 . The error evaluated on real images are reported and summarized into Table 1. One can observe that our method outperforms baselines in the order of magnitude, clearly showing the superiority of our model. For qualitative evaluation, we randomly select a frame from each object category and show them in Figure 6 against ground truth and baselines. It shows that our reconstructed landmarks effectively depict

2 Augmentation is utilized due to limited valid frames, because the ground-truth cameras are partially missing.

| Methods     | KSTA [16]   |   KSTA [16] | BMM[10]   |   BMM[10] | CNS [20]   |   CNS [20] | MUS [2]   | MUS [2]   | NLO [12]   |   NLO [12] | RIKS [17]   |   RIKS [17] | SPS [18]   |   SPS [18] | SFC [19]   |   SFC [19] | OURS   |   OURS |
|-------------|-------------|-------------|-----------|-----------|------------|------------|-----------|-----------|------------|------------|-------------|-------------|------------|------------|------------|------------|--------|--------|
| Aeroplane   | 0.145       |       0.175 | 0.843     |     1.459 | 0.263      |      0.416 | 0.261     | -         | -          |      0.876 | -           |       0.132 | -          |      0.93  | -          |      0.504 | -      |  0.024 |
| Bicycle     | 0.442       |       0.245 | 0.308     |     1.376 | -          |      0.356 | 0.178     | -         | -          |      0.269 | -           |       0.136 | -          |      1.322 | -          |      0.372 | -      |  0.003 |
| Bus         | 0.214       |       0.199 | 0.300     |     1.023 | -          |      0.25  | 0.113     | -         | -          |      0.14  | -           |       0.16  | -          |      0.604 | -          |      0.251 | -      |  0.004 |
| Car         | 0.159       |       0.152 | 0.266     |     1.278 | 0.099      |      0.258 | 0.078     | -         | -          |      0.104 | -           |       0.097 | -          |      0.872 | -          |      0.282 | -      |  0.009 |
| Chair       | 0.399       |       0.186 | 0.357     |     1.297 | -          |      0.17  | 0.210     | -         | -          |      0.146 | -           |       0.192 | -          |      1.046 | -          |      0.226 | -      |  0.007 |
| Diningtable | 0.372       |       0.267 | 0.422     |     1     | -          |      0.17  | 0.264     | -         | -          |      0.109 | -           |       0.207 | -          |      1.05  | -          |      0.221 | -      |  0.06  |
| Motorbike   | 0.270       |       0.255 | 0.336     |     0.857 | -          |      0.457 | 0.222     | -         | -          |      0.432 | -           |       0.118 | -          |      0.986 | -          |      0.361 | -      |  0.002 |
| Sofa        | 0.298       |       0.307 | 0.279     |     1.126 | 0.214      |      0.25  | 0.167     | -         | -          |      0.149 | -           |       0.228 | -          |      1.328 | -          |      0.302 | -      |  0.004 |
| Average     | 0.287       |       0.223 | 0.388     |     1.178 | 0.192      |      0.291 | 0.186     | -         | -          |      0.278 | -           |       0.159 | -          |      1.017 | -          |      0.315 | -      |  0.014 |
| Relative    | -           |      15.33  | -         |    80.76  | -          |     19.95  | -         | -         | -          |     19.09  | -           |      10.92  | -          |     69.74  | -          |     21.61  | -      |  1     |
| Aeroplane   | 0.183       |       0.207 | 0.566     |     1.465 | 0.294      |      0.46  | 0.271     | -         | -          |      0.758 | -           |       0.146 | -          |      0.888 | -          |      0.521 | -      |  0.032 |
| Bicycle     | 0.457       |       0.232 | 0.307     |     1.404 | -          |      0.359 | 0.188     | -         | -          |      0.275 | -           |       0.139 | -          |      0.851 | -          |      0.379 | -      |  0.007 |
| Bus         | 0.218       |       0.197 | 0.255     |     0.764 | -          |      0.264 | 0.122     | -         | -          |      0.141 | -           |       0.159 | -          |      1.11  | -          |      0.264 | -      |  0.021 |
| Car         | 0.164       |       0.139 | 0.161     |     1.744 | 0.122      |      0.265 | 0.093     | -         | -          |      0.105 | -           |       0.102 | -          |      0.804 | -          |      0.281 | -      |  0.01  |
| Chair       | 0.396       |       0.203 | 0.258     |     1.197 | -          |      0.171 | 0.220     | -         | -          |      0.145 | -           |       0.193 | -          |      1.016 | -          |      0.223 | -      |  0.017 |
| Diningtable | 0.383       |       0.249 | 0.358     |     1.105 | -          |      0.172 | 0.267     | -         | -          |      0.114 | -           |       0.227 | -          |      1.213 | -          |      0.222 | -      |  0.034 |
| Motorbike   | 0.290       |       0.227 | 0.299     |     1.117 | -          |      0.459 | 0.233     | -         | -          |      0.254 | -           |       0.125 | -          |      0.915 | -          |      0.351 | -      |  0.011 |
| Sofa        | 0.294       |       0.436 | 0.240     |     1.143 | 0.228      |      0.255 | 0.174     | -         | -          |      0.152 | -           |       0.239 | -          |      1.164 | -          |      0.306 | -      |  0.008 |
| Average     | 0.298       |       0.236 | 0.305     |     1.232 | 0.215      |      0.3   | 0.196     | -         | -          |      0.243 | -           |       0.166 | -          |      0.995 | -          |      0.318 | -      |  0.017 |
| Relative    | -           |      16.9   | -         |    88.78  | -          |     21.49  | -         | -         | -          |     17.39  | -           |      11.9   | -          |     71.11  | -          |     22.78  | -      |  1.27  |

Table 2: Quantitative evaluation on PASCAL3D+ dataset. We conduct experiments on both original and noisy 2D annotations, listed at the upper and lower half of table respectively. The symbol '-' indicates either algorithm implementation or data is missing. The shaded columns are erros using our processed data and others are copied from Table 2 in [2]. Relative errors are computed with respect to our method, the most accurate solution, without noise perturbation. Our data and implementation will be publicly accessible for future comparison.

the 3D geometry of objects and our method is able to cover subtle geometric details.

## 5.2. S f C on PASCAL3D+

We then apply our method to PASCAL3D+ dataset [33] which contains twelve object categories and each category is labeled by approximately eight 3D CADs. To compare against more baselines, we follow the experiment setting reported in [2] and use the same normalized 3D error metric. We report our errors in Table 2 emphasized by shading and concatenate the numbers copied from the Table 2 in [2] for comparison. Note that the errors are not exactly reproduced even though using the same dataset and algorithm implementation, because the data preparation details are missing. However, one can clearly see that our proposed method achieves extremely accurate reconstructions with more than ten times of smaller 3D error. This large margin makes the slight difference caused by data preparation even less noticeable. It clearly demonstrates the high precision of our proposed deep neural network and also the superior robustness in noisy situations.

we concatenate 80% of motions to form large image collections and remain the left 20% as unseen motions for testing generalization. Note that in this experiment, each subject contains more than ten thousands of frames. We compare our method against state-of-the-art methods, summarized in Table 3. Due to huge volume of frames, KSTA [16], BMM [10], MUS [2], RIKS [17] all fail and thus are omitted in the table. We also report the normalized 3D error on unseen motions, labeled as UNSEEN. One can see that our method obtains impressive reconstruction performance and outperforms others again in every sequences. Moreover, our network also show a well generalization to unseen data which improve the effectiveness in real world applications. For qualitative evaluation, we randomly select a frame from

## 5.3. Large-scale NRS M on CMU MoCap f

We finally apply our method to solving the problem of NRS M using the CMU motion capture dataset 3 . f We randomly select 10 subjects out of 144 and for each subject

3 http://mocap.cs.cmu.edu/

Figure 3: NRS M with noise perturbation. f The red solid line is ours while the green dashed line is CNS [20], the best performance of state-of-the-arts with no noise perturbation.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-53de6362-ab0e-42fb-84d5-fafe089004d0.jpg)

Table 3: Quantitative comparison on solving large-scale NRS f Mproblem using CMU MoCap dataset. Each subject contains more than ten thousand of frames. Due to huge volume of frames, KSTA [16], BMM [10], MUS [2], RIKS [17] all fail and thus are omitted in the table. UNSEEN refers to the errors of the motions that are not accessible during training. This is used to demonstrate the well generalization of our proposed network, which is especially important in real world applications.

| Subjects   |    01 |    05 |    18 |    23 |    64 |    70 |   102 |   106 |   123 |   127 |   Average |   Relative |
|------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-----------|------------|
| CNS [20]   | 0.613 | 0.657 | 0.541 | 0.603 | 0.543 | 0.472 | 0.581 | 0.636 | 0.479 | 0.644 |     0.577 |       5.66 |
| NLO [12]   | 1.218 | 1.16  | 0.917 | 0.998 | 1.218 | 0.836 | 1.144 | 1.016 | 1.009 | 1.05  |     1.057 |      10.37 |
| SPS [18]   | 1.282 | 1.122 | 0.953 | 0.88  | 1.119 | 1.009 | 1.078 | 0.957 | 0.828 | 1.021 |     1.025 |      10.06 |
| OURS       | 0.175 | 0.22  | 0.081 | 0.053 | 0.082 | 0.039 | 0.115 | 0.113 | 0.04  | 0.095 |     0.101 |       1    |
| UNSEEN     | 0.362 | 0.331 | 0.437 | 0.387 | 0.174 | 0.09  | 0.413 | 0.194 | 0.091 | 0.388 |     0.287 |       2.81 |

each subject and render the reconstructed human skeleton in Figure 5. This visually verifies the impressive performance of our deep solution.

reconstruction from sparse observations [9, 24, 21, 22, 7]. These two solutions make our central pipeline of DNN more easily to adapt to handling missing data.

Robustness analysis: To analyze the robustness of our method, we re-train the neural network for Subject 70 using projected points with Gaussian noise perturbation. The results are summarized in Figure 3. The noise ratio is defined as ‖ noise ‖ F / ‖ W ‖ F . One can see that the error increases slowly with adding higher magnitude of noise and when adding up to 20% noise to image coordinates, our method in red still achieves better reconstruction compared to the best baseline with no noise perturbation (in green). This experiment clearly demonstrates the robustness of our model and its high accuracy against state-of-the-art works.

Missing data: Landmarks are not always visible from the camera owing to the occlusion by other objects or itself. In the present paper, we focus on a complete measurement situation not accounting for invisible landmarks. However, thanks to recent progress in matrix completion, our method can be easily extended to missing data. Moreover, in our experiments, we observe that deep neural network shows a well tolerance of missing data. Simply setting missing 2D coordinates as zeros provides satisfactory results. Such technique is widely used in deep-learning-based depth map

Figure 4: A scatter plot of the shape error ratio in percentage against the final dictionary coherence. A line is fitted based on the data. The left comes from subject 05, the middle from subject 18, the right from subject 64.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-f9d67226-a692-4f0c-9dd3-4ca8c2aa8485.jpg)

## 5.4. Coherence as guide

As explained in Section 4.1, every sparse code ψ i is constrained by its subsequent representation and thus the quality of code recovery depends less on the quality of the corresponding dictionary. However, this is not applicable to the final code ψ n , making it least constrained with the most dependency on the final dictionary D n . From this perspective, the quality of the final dictionary measured by mutual coherence [14] could serve as a lower bound of the entire system. To verify this, we compute the error and coherence in a fixed interval during training in NRS f Mexperiments. We consistently observe strong correlations between 3D reconstruction error and the mutual coherence of the final dictionary. We plot this relationship in Figure 4. We thus propose to use the coherence of the final dictionary as a measure of model quality for guiding training to efficiently avoid overfitting especially when 3D evaluation is not available. This improves the utility of our deep NRS f M in future applications without 3D ground-truth.

## 6. Conclusion

In this paper, we proposed multi-layer sparse coding as a novel prior assumption for representing 3D non-rigid shapes and designed an innovative encoder-decoder neural network to solve the problem of NRS M using f no 3D supervision. The proposed network was derived by generalizing the classical sparse coding algorithm ISTA to a block sparse scenario. The proposed network architecture is mathematically interpretable as solving a NRS f Mmultilayer sparse dictionary learning problem. Extensive experiments demonstrated our superior performance against the state-of-the-art methods and our generalization to unseen data. Finally, we proposed to use the coherence of the final dictionary as a model quality measure, offering a practical way to avoid over-fitting and select the best checkpoint during training without relying on 3D ground-truth.

Figure 5: Qualitative evaluation on CMU Mocap dataset. Top to bottom: ground-truth, ours, CNS [20], SPS [18], NLO [12]. Each column corresponds to reconstructions of a certain frame, randomly selected from each subject. Spheres are reconstructed landmarks while bars are for visualization. 3D shapes are already aligned to the ground truth by orthonormal matrix.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-3fb6c4b4-6010-41fe-9394-6b832c48a7d5.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-b5ce090a-b28d-45ca-a3b6-c170eb9df9b1.jpg)

Figure 6: Qualitative evaluation on IKEA dataset. Landmarks projected by annotated cameras are omitted from images. In each rendering, red cubes are reconstructed points while the planes and bars are manually added for descent visualization. Left to right: annotated image, ground-truth, ours, RIKS [17], KSTA [16], NLO [12], SFC [19], CNS [20], BMM [10].

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-a4f31e2d-52aa-43b1-b945-b17c8d50900f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-08d38274-dab3-430f-a6ff-bdff2ee7b43b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-780530f5-a875-4012-ab7f-f950364ebe92.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-2efe633e-9560-47da-8191-261bc1aff804.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-60c166f3-3af9-4536-8027-146da1429a01.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-4e0bb3b0-bf49-4db9-b28e-04f3027f4440.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-cbc02715-1e3a-4e74-b410-83774be893ce.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-5e533337-5621-4b1c-a7c8-6d26c5184f86.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-bbc279ed-7764-4e4e-b96d-2b769c58eda1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-ecd3a5ca-737e-46fe-83b5-d3f7a073e233.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-2c6269bd-28cb-488f-bd64-5c8c6e455389.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-39597eb1-3783-47ee-8a8e-55ebe4b05da7.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-2cef7fe0-dc4a-4442-9831-065bd993a121.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-a4b3c077-3891-409b-a556-a9907885304c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-4b073d4b-8e6d-4a7c-8732-d26b5fe8266c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-c6816ffd-4487-4625-a2f5-8b4f24e0d6bf.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-eba54c15-0f92-43e2-8003-ebb5ef6012e1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-0f9655af-dbe2-478c-9e23-8b0288d4aef3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-fbda4165-5259-4c04-abd2-cee6a832e43b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-4bf4da7b-c35b-41a2-a354-19076e6cbdce.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-1458617b-4dbf-47fa-9e3a-7adf4307a5da.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-66c433f4-24b5-4880-880a-c147256fe631.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-13acdcaf-cd48-4b21-ab4b-7f37c04386b7.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-7c1cf56d-c110-471f-b9a7-1f9270fd307b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-c0a874c4-7da8-426c-b534-794253f30cf3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-48562492-7fd9-467a-811a-869d517b2c48.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-c9c71d3e-654e-4891-8f38-4eee55fc49c2.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-03712646-933b-437e-8514-404a1cb74ebc.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-e6cf94d9-a363-40c5-896e-d8fb6d21efd3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-00131b11-3ccb-4fc5-833f-993b690acd7d.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-187a3d75-7b9c-4866-96bb-835ab9e6ee14.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-de9b6e3a-3a34-4421-8412-e6bfeadfb308.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-a97342df-aec9-43a4-b9ec-9bd1e9a4e361.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00052v2-figure-79babdff-adc9-40d5-aef3-553df70a7c87.jpg)

## References

- [1] Antonio Agudo and Francesc Moreno-Noguer. Dust: Dual union of spatio-temporal subspaces for monocular multiple object 3d reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6262-6270, 2017. 2
- [2] Antonio Agudo, Melcior Pijoan, and Francesc MorenoNoguer. Image collection pop-up: 3d reconstruction and clustering of rigid and non-rigid categories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2607-2615, 2018. 1, 2, 5, 6, 7
- [3] Ijaz Akhter, Yaser Sheikh, and Sohaib Khan. In defense of orthonormality constraints for nonrigid structure from motion. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on , pages 1534-1541. IEEE, 2009. 2
- [4] Ijaz Akhter, Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Nonrigid structure from motion in trajectory space. In Advances in neural information processing systems , pages 4148, 2009. 5
- [5] Amir Beck and Marc Teboulle. A fast iterative shrinkagethresholding algorithm with application to wavelet-based image deblurring. In Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on , pages 693-696. IEEE, 2009. 2
- [6] Christoph Bregler, Aaron Hertzmann, and Henning Biermann. Recovering non-rigid 3d shape from image streams. In Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on , volume 2, pages 690-696. IEEE, 2000. 1, 2
- [7] Cesar Cadena, Anthony R Dick, and Ian D Reid. Multimodal auto-encoders as joint estimators for robotics scene understanding. In Robotics: Science and Systems , 2016. 7
- [8] Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. CoRR , abs/1512.03012, 2015. 1
- [9] Zhao Chen, Vijay Badrinarayanan, Gilad Drozdov, and Andrew Rabinovich. Estimating depth from rgb and sparse sensing. European Conference on Computer Vision (ECCV) , 2018. 7
- [10] Yuchao Dai, Hongdong Li, and Mingyi He. A simple priorfree method for non-rigid structure-from-motion factorization. International Journal of Computer Vision , 107(2):101122, 2014. 1, 2, 3, 4, 5, 6, 7, 8
- [11] Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 57(11):14131457, 2004. 2
- [12] Alessio Del Bue, Fabrizio Smeraldi, and Lourdes Agapito. Non-rigid structure from motion using ranklet-based tracking and non-linear optimization. Image and Vision Computing , 25(3):297-310, 2007. 5, 6, 7, 8
- [13] Wei Deng, Wotao Yin, and Yin Zhang. Group sparse optimization by alternating direction method. In SPIE Optical Engineering+ Applications , pages 88580R-88580R. International Society for Optics and Photonics, 2013. 4
- [14] David L Donoho, Michael Elad, and Vladimir N Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Transactions on information theory , 52(1):6-18, 2006. 3, 7
- [15] Katerina Fragkiadaki, Marta Salas, Pablo Arbelaez, and Jitendra Malik. Grouping-based low-rank trajectory completion and 3d reconstruction. In Advances in Neural Information Processing Systems , pages 55-63, 2014. 2
- [16] Paulo FU Gotardo and Aleix M Martinez. Kernel non-rigid structure from motion. In Computer Vision (ICCV), 2011 IEEE International Conference on , pages 802-809. IEEE, 2011. 5, 6, 7, 8
- [17] Onur C Hamsici, Paulo FU Gotardo, and Aleix M Martinez. Learning spatially-smooth mappings in non-rigid structure from motion. In European Conference on Computer Vision , pages 260-273. Springer, 2012. 5, 6, 7, 8
- [18] Chen Kong and Simon Lucey. Prior-less compressible structure from motion. Computer Vision and Pattern Recognition (CVPR) , 2016. 1, 2, 3, 4, 5, 6, 7, 8
- [19] Chen Kong, Rui Zhu, Hamed Kiani, and Simon Lucey. Structure from category: a generic and prior-less approach. International Conference on 3DVision (3DV) , 2016. 1, 2, 3, 5, 6, 8
- [20] Minsik Lee, Jungchan Cho, and Songhwai Oh. Consensus of non-rigid reconstructions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4670-4678, 2016. 5, 6, 7, 8
- [21] Yaoxin Li, Keyuan Qian, Tao Huang, and Jingkun Zhou. Depth estimation from monocular image and coarse depth points based on conditional gan. In MATEC Web of Conferences , volume 175, page 03055. EDP Sciences, 2018. 7
- [22] Yiyi Liao, Lichao Huang, Yue Wang, Sarath Kodagoda, Yinan Yu, and Yong Liu. Parse geometry from a line: Monocular depth estimation with partial laser observation. In Robotics and Automation (ICRA), 2017 IEEE International Conference on , pages 5059-5066. IEEE, 2017. 7
- [23] Joseph J. Lim, Hamed Pirsiavash, and Antonio Torralba. Parsing IKEA Objects: Fine Pose Estimation. ICCV , 2013. 5
- [24] Fangchang Mal and Sertac Karaman. Sparse-to-dense: Depth prediction from sparse depth samples and a single image. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1-8. IEEE, 2018. 7
- [25] Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research , 18(1):2887-2938, 2017. 2
- [26] Christopher J Rozell, Don H Johnson, Richard G Baraniuk, and Bruno A Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural computation , 20(10):2526-2563, 2008. 2
- [27] Hao Su, Charles R Qi, Yangyan Li, and Leonidas J Guibas. Render for cnn: Viewpoint estimation in images using cnns

trained with rendered 3d model views. In Proceedings of the IEEE International Conference on Computer Vision , pages 2686-2694, 2015. 1

[28] Jeremias Sulam, Vardan Papyan, Yaniv Romano, and Michael Elad. Multi-layer convolutional sparse modeling: Pursuit and dictionary learning. arXiv preprint arXiv:1708.08705 , 2017. 3

- [29] Carlo Tomasi and Takeo Kanade. Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision , 9(2):137-154, 1992. 2
- [30] Sara Vicente, Jo˜ ao Carreira, Lourdes Agapito, and Jorge Batista. Reconstructing pascal voc. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 41-48. IEEE, 2014. 1
- [31] Jiajun Wu, Tianfan Xue, Joseph J Lim, Yuandong Tian, Joshua B Tenenbaum, Antonio Torralba, and William T Freeman. Single image 3d interpreter network. European Conference on Computer Vision (ECCV) , 2016. 5
- [32] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1912-1920, 2015. 1
- [33] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision , pages 75-82. IEEE, 2014. 6
- [34] Xiaowei Zhou, Spyridon Leonardos, Xiaoyan Hu, and Kostas Daniilidis. 3d shape estimation from 2d landmarks: A convex relaxation approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4447-4455, 2015. 3, 5
- [35] Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Kosta Derpanis, and Kostas Daniilidis. Sparseness meets deepness: 3d human pose estimation from monocular video. arXiv preprint arXiv:1511.09439 , 2015. 2
- [36] Yingying Zhu, Dong Huang, Fernando De La Torre, and Simon Lucey. Complex non-rigid motion 3d reconstruction by union of subspaces. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on , pages 1542-1549. IEEE, 2014. 1, 2, 3, 4
- [37] Yingying Zhu and Simon Lucey. Convolutional sparse coding for trajectory reconstruction. Pattern Analysis and Machine Intelligence, IEEE Transactions on , 37(3):529-540, 2015. 3
Chen Kong Simon Lucey
Deep Non-Rigid Structure from Motion
Current non-rigid structure from motion (NRSfM) algorithms are mainly limited
with respect to: (i) the number of images, and (ii) the type of shape
variability they can handle. This has hampered the practical utility of NRSfM
for many applications within vision. In this paper we propose a novel deep
neural network to recover camera poses and 3D points solely from an ensemble of
2D image coordinates. The proposed neural network is mathematically
interpretable as a multi-layer block sparse dictionary learning problem, and
can handle problems of unprecedented scale and shape complexity. Extensive
experiments demonstrate the impressive performance of our approach where we
exhibit superior precision and robustness against all available
state-of-the-art works in the order of magnitude. We further propose a quality
measure (based on the network weights) which circumvents the need for 3D
ground-truth to ascertain the confidence we have in the reconstruction.