This paper proposes a representational model for image pair such as consecutive video frames that are related by local pixel displacements, in the hope that the model may shed light on motion perception in primary visual cortex (V1). The model couples the following two components. (1) The vector representations of local contents of images. (2) The matrix representations of local pixel displacements caused by the relative motions between the agent and the objects in the 3D scene. When the image frame undergoes changes due to local pixel displacements, the vectors are multiplied by the matrices that represent the local displacements. Our experiments show that our model can learn to infer local motions. Moreover, the model can learn Gabor-like filter pairs of quadrature phases.
The authors propose a model for learning local pixel motions between pairs of frames using local image representations and relative pixel displacements between agents and objects.  The model learned is compared to the ability of the primary visual cortex where adjacent simple cells share quadrature relationships and capture local motion.
"The representation theory underlies much of modern mathematics and holds the key to the quantum
theory (Zee, 2016)."
Can the relevance of this claim be elaborated on?
"Figure 1 illustrates the scheme of representation."
Please provide more detail here on what is happening in the figure.  The caption and reference here are not informative to what the figure is representing.
"We obtain the training data by collecting static images for (It) and simulate the
displacement field ...  We refer to this method as self-supervised learning"
This is not self-supervised learning.  In self-supervised learning the training label/signal is generated by the system.  In this case artificial data is being generated as the displacement between images is sampled.
Since the motion between images is artificially generated what guarantees are there that the model is learning to capture realistic motion behavior?  Why not use adjacent video frames?
"Note that those methods train deep and complicated neural networks with large scale datasets to
predict optical flows in supervised manners, while our model can be treated as a simple one-layer
network, accompanied by weight matrices representing motions."
Is there a comparison on execution times of the different approaches?
"by obtaining the pre-trained models and testing on V1Deform testing data"
Is this a fair comparison if the proposed approach was trained on V1Deform training data and the comparison methods were not.  A more appropriate comparison would be to apply all the methods to infer the displacement fields between video frames which is also a more natural application.  This can be controlled to contain small motions if needed.  Why nt use the MUG dataset here?
"Displacements at image border are leaved out" -> left out
Sections 5.4, 5.5 and 5.6 show only qualitative results with no comparison methods.  Can the authors provide reasons that other methods could not be used for evaluation?
I am not sure I understand the motivation for the approach.  Why do we need this over other methods that can better capture larger motions.  This needs to be more clear from the introduction.  Why do we care if the approach captures aspects of V1 for the tasks presented?
The work is sensible and the approach is clear but I found the evaluation and motivation lacking in key areas that I mention above.  The authors should revise and make it clear to the reader why we should care about this problem.  Aligning with V1 is interesting but it does not come into play in the applications of the approach or the analysis so I am not sure why I should care.  The evaluation also needs to be much more convincing before I could recommend acceptance.