Within-class variation in a high-dimensional dataset can be modeled as being on a low-dimensional manifold due to the constraints of the physical processes producing that variation (e.g., translation, illumination, etc.). We desire a method for learning a representation of the manifolds induced by identity-preserving transformations that can be used to increase robustness, reduce the training burden, and encourage interpretability in machine learning tasks. In particular, what is needed is a representation of the transformation manifold that can robustly capture the shape of the manifold from the input data, generate new points on the manifold, and extend transformations outside of the training domain without significantly increasing the error. Previous work has proposed algorithms to efficiently learn analytic operators (called transport operators) that define the process of transporting one data point on a manifold to another. The main contribution of this paper is to define two transfer learning methods that use this generative manifold representation to learn natural transformations and incorporate them into new data. The first method uses this representation in a novel randomized approach to transfer learning that employs the learned generative model to map out unseen regions of the data space. These results are shown through demonstrations of transfer learning in a data augmentation task for few-shot image classification. The second method use of transport operators for injecting specific transformations into new data examples which allows for realistic image animation and informed data augmentation. These results are shown on stylized constructions using the classic swiss roll data structure and in demonstrations of transfer learning in a data augmentation task for few-shot image classification. We also propose the use of transport operators for injecting transformations into new data examples which allows for realistic image animation.
Overview:
The paper aim to model non-linear, intrinsically low-dimensional structure, in data by estimating "transport operators" that predict how points move along the manifold. This is an old idea, and the stated contribution of the paper is:
"The main contribution of this paper is to show that the manifold representation learned in the transport operators is valuable both as a probabilistic model to improve general machine learning tasks as well as for performing transfer learning in classification tasks." 
The paper provide nice illustrative experiments arguing why transport operators may be a useful modeling tool, but does not go beyond illustrative experiments.
While I follow the intuitions behind transport operators I am doubtful if they will generalize beyond very simple manifold structures (see detailed comments below).
Quality:
The paper is well-written and fairly easy to follow. In particular, I appreciate that the authors make no attempt to overclaim contributions. From a methodology point-of-view, the paper has limited novelty (transport operators, and learning thereof has been studied elsewhere), but there are some technical insights (likelihood model, use in data augmentation). Since the provided experiments are mostly illustrations, I would argue that the significance of the paper is limited. I'd say that to really convince a broader audience that this old idea is worth revisiting, the work must go beyond illustrations and apply to a real data problem.
Detailed Comments and Questions:
*) Equation 1 of the paper describe the key dynamics of the applied transport operators. Basically, the paper assume that the underlying data manifold is locally governed by a linear differential equation. This is a very suitable assumption, e.g., for the swiss roll data set, but it is unclear to this reader why it is a suitable assumption beyond such toy data. I would very much appreciate a detailed discussion of when this is a suitable modeling choice, and when it is not. My intuition is that this is mostly a suitable model when the data manifold appears due to simple transformations (e.g. rotations) of data. This is also exactly the type of data considered in the paper.
*) In Eq. 3, should it be "expm" instead of "exp" ?
*) The first two paragraphs of Sec. 2 are background material, whereas paragraph 3 and beyond describe material that is key to the paper. I would recommend introducing a \subsection (or something like it) to make this more clear.
*) The idea of working with transformations of data rather than the actual data is the corner-stone of Ulf Grenander's renowned "Pattern Theory". A citation to this seminal work would be appropriate.
*) In the first paragraph of the introduction links are drawn to the neuroscience literature; it would be appropriate to cite a suitable publication.
Pros(+) & Cons(-):
+ Well-written.
+ Good illustrative experiments.
- Real-life experiments are lacking.
- Limited methodology contribution.
- The assumed dynamics might be too simplistic (at least a discussing of this is missing).
For the AC:
The submitted paper acknowledges several grants (including grant numbers), which can directly be tied to the authors identity. This may be a violation of the double blind review policy. I did not use this information to determine the authors identity, though, so this review is still double blind.
Post-rebuttal comments:
The paper has improved with the incorporated revisions, but my main concerns remain. I find the Swiss Roll / rotated-USPS examples to be too contrived as the dynamics are exactly tailored to the linear ODE assumption. These are examples where the model assumptions are perfect. What is unclear is how the model behaves when the linear ODE assumption is not-quite-correct-but-also-not-totally-incorrect, i.e. how the model behaves in real life. I didn't get that from the newly added experiment. So, I'll keep my rating as is.