UNSUPERVISED DISCOVERY OF PARTS, STRUCTURE, AND DYNAMICS
Humans easily recognize object parts and their hierarchical structure by watching how they move; they can then predict how each part moves in the future. In this paper, we propose a novel formulation that simultaneously learns a hierarchical, disentangled object representation and a dynamics model for object parts from unlabeled videos. Our Parts, Structure, and Dynamics (PSD) model learns to, first, recognize the object parts via a layered image representation; second, predict hierarchy via a structural descriptor that composes low-level concepts into a hierarchical structure; and third, model the system dynamics by predicting the future. Experiments on multiple real and synthetic datasets demonstrate that our PSD model works well on all three tasks: segmenting object parts, building their hierarchical structure, and capturing their motion distributions.
==== Review Summary ====
The paper demonstrates an interesting and potentially useful idea.  But much of it is poorely explained, and experimental results are not strongly convincing.  The only numerical evaluations are on a simple dataset that the authors made themselves.  The most interesting claim - that this network can learn unsupervised hierarchical object segmentation based on unlabelled video data -  is not well supported by the paper.  
==== Paper Summary ====
This paper presents a deep neural network which learns object Segmentation, Structure, and Dynamics from unlabelled video.  The idea is quite useful, as it is a step towards learning models that can "discover" the concept of objects in a visual scene without any supervision.  
The central contributions of this paper are:
(1) To show how one can use coherent motion (the fact that different parts of an object move together) to learn unsupervised object segmentation.
(2) To show how once can learn the relation between objects (e.g. "arm" is part of "body") by observing relative motion between segments.
==== General Feedback ====
The paper would benefit a lot from better explanations and being better tied together (see "Details" below for examples).   Figure captions would benefit from much better explanations and integration with the text - each figure caption should at least describe what the figure is intended to demonstrate.  Variables such as ($\mathcal M$, $\mathcal S$, $\mathcal I$, $\mathcal L$) should be indicated in figures .  
Many things would benefit from being defined precisely with Equations.  For example I have no idea how the "soft" structural descriptor S is computed.  Is it (A) a parameter that is shared across data points and learned?  or (B) is it computed per-frame from the network?  And after it is calculated, how are the S_{ik} values (which fall between 0 and 1) used to combine the motion maps?  
==== Scientific Questions ===
I'm confused as to what the latent variables z "mean".  It seems strange that there is a 1-d latent variable representing the motion of each part.  Translation of a segment within an image is 2D.  3D if you include planar rotation, then there's scaling motion and out-of-plane rotations, so it seems an odd design choice that motion should be squeezed into a 1D representation.
I find it difficult to assess the quality of the "next frame predictions".  There's lots other literature on next-frame prediction to compare against (e.g. https://arxiv.org/abs/1605.08104).  At least you could compare to a naive baseline of simply shifting pixels based on the optical flow.  
I'm confused about how you are learning the "estimated flow".  My impression is that the input flow is calculated between the last 2 frames $\hat M = flow(I_{t-1}, I_t)$.  And that the "estimated" flow is an estimate of $flow(I_{t}, I_{t+1})$.  But in Section 4.3 you seem to indicate that the "estimated" flow is just trained to "reconstruct" the input flow.... In that case why not just feed the input flow directly into the Image Decoder?  What I guess you're doing is trying to Predict the next flow ($flow(I_{t}, I_{t+1})$) but if you're doing this neither Figure 3 nor Section 4.2 indicates this, hence my confusion.  
==== Details ====
Figure 3: 
----
The "shapes" example is odd, because it's not obvious that there's a structural hierarchy linking the shapes.  Maybe a "torso/left arm/right arm" would be better for illustrative purposes?
It would be helpful to put the variable names ($\mathcal M_k$, etc) on the figure.
Should add a second arrow coming into the (f) the Structural descriptor from a leaf-variable $p_k$
Also it would be helpful to indicate where the losses are evaluated.
"Next Frame" should probably be "Reconstruction" (though "Prediction" might be a more accurate word).
---
Section 4.2:
Notational point, it seems k can be from 1 to d.  But in Section 3 you say it can be from 1 to "n". Maybe it would be clearer to change both ("n" and "d") to "K" to emphasize that "k" is an index which goes up to "K".  (edit... now after reading 5.1: Latent Representation, I understand.  If there are n parts in the dataset you use d>n dimensions and the network learns to "drop" the extra ones... it would help to clarify that here).
Structural Descriptor: You say "in practice, we relax the binary constraints"... Yet this is important.. should write down the equation and say how the "soft" version of [i \in S] calculated.
Section 4.3
"elbo" seems like the wrong name for this loss, as it's not an Evidence Lower BOund.  The elbo would be the sum of this loss and the first component of L_{recon}.  It is a regularizing term, so you could call it L_reg.
It's not obvious that sparse local motion maps mean a heirarchical tree structure, but I see how it could help.  I suggest that without evidence for this loss you soften the claim to "this is intended to help encourage the model to learn a heirarchical tree structure"
Figure 4: 
It appears that each row indicates a different video in the dataset, but then in 4f you still have two rows but they appear to correspond to different algorithms... a vertical separator here might help show that the rows in 4f do not correspond to the rows in 4a-e.
"Next Frame" here appears to refer to ground truth, but in Figure 3 "Next Frame" appears to be the prediction (which you call reconstruction). 
Section 5.1
Future Prediction: No explanation either here or in Figure 4 of what it actually shows.  (What would a failure look like?)
Hierarchical structure... You binarize... how?
Figure 9:
What does this show?  The figure does not obviously demonstrate anything.  Maybe compare to ground-truth future frames?
Section 5.3:
Future Prediction: These images are from the test set, right?  If so, that is worth mentioning. 
Object Segmentation ("in several different dimensions"  -> "corresponding to the active latent dimensions?")
Object Segmentation: Visually, it looks nice, but I have now idea how good this segmentation is.  You compare verbally to R-NEM and PSD, but there're no numbers.
Human Studies... The line "For those whose predicted tree structures are not consistent with ours, they all agree with our results and believe ours are more reasonable than others" .. brings to mind an image of a room full of tortured experimental subjects not being allowed to leave until they sign a confession that their own tree structures were foolish mistakes and your tree structures are far superior.... So probably it should be removed because it sounds shady.