UNIVERSAL MODAL EMBEDDING OF DYNAMICS IN VIDEOS AND ITS APPLICATIONS
Extracting underlying dynamics of objects in image sequences is one of the challenging problems in computer vision. On the other hand, dynamic mode decomposition (DMD) has recently attracted attention as a way of obtaining modal representations of nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. In this paper, we propose a convolutional autoencoder based DMD (CAE-DMD) that is an extended DMD (EDMD) approach, to extract underlying dynamics in videos. To this end, we develop a modified CAE model by incorporating DMD on the encoder, which gives a more meaningful compressed representation of input image sequences. On the reconstruction side, a decoder is used to minimize the reconstruction error after applying the DMD, which in result gives an accurate reconstruction of inputs. We empirically investigated the performance of CAE-DMD in two applications: background/foreground extraction and video classification, on publicly available datasets.
This paper presents an application of convolutional autoencoder networks and a nonlinear dynamic systems analysis method known as extended dynamic mode analysis (EDMD) to a data-driven analysis of multivariate time series. The DMD method appears to be well-known in the physics community but is outside my area of expertise and unfortunately I have limited time to make a quick study of it. However, from what I gather, it involves empirical approximation of a nonlinear dynamical system as a high-dimensional linear dynamical system, which in turn enables analysis in terms of eigendecomposition of the resulting linear operator, revealing basic modes of the dynamics. In the proposed method, DMD is used in the latent representations of a convolutional autoencoder for image sequences. The DMD objective is incorporated into the autoencoder training loss to minimize its reconstruction error. The DMD is also used, by conditioning on the eigenvalues, to split the reconstruction into high frequency (quickly varying) foreground modes and low-frequency (slowly varying) background modes. Although end-to-end training is mentioned, it is not made clear how the derivatives of the DMD decomposition are implemented, especially considering that the DMD involves an SVD, which can have unstable/ singular derivatives when two or more singular values are close to the same / exactly the same. The resulting methods are applied to a foreground extraction and a classification tasks, and compared with numerous baselines. It is not clear to me what the state of the art is on these tasks, but the proposed methods compare favorably to reported baselines, and the images of results look convincing. However the experimental results seem a little thin and I would expect a more thorough study. Overall the method looks very interesting. Some complaints: - the tables are a bit sloppy and should be formatted to fit in the document with normal sized fonts, - the images are too small to see well.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
UNIVERSAL MODAL EMBEDDING OF DYNAMICS IN VIDEOS AND ITS APPLICATIONS
Extracting underlying dynamics of objects in image sequences is one of the challenging problems in computer vision. On the other hand, dynamic mode decomposition (DMD) has recently attracted attention as a way of obtaining modal representations of nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. In this paper, we propose a convolutional autoencoder based DMD (CAE-DMD) that is an extended DMD (EDMD) approach, to extract underlying dynamics in videos. To this end, we develop a modified CAE model by incorporating DMD on the encoder, which gives a more meaningful compressed representation of input image sequences. On the reconstruction side, a decoder is used to minimize the reconstruction error after applying the DMD, which in result gives an accurate reconstruction of inputs. We empirically investigated the performance of CAE-DMD in two applications: background/foreground extraction and video classification, on publicly available datasets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents an application of convolutional autoencoder networks and a nonlinear dynamic systems analysis method known as extended dynamic mode analysis (EDMD) to a data-driven analysis of multivariate time series. The DMD method appears to be well-known in the physics community but is outside my area of expertise and unfortunately I have limited time to make a quick study of it. However, from what I gather, it involves empirical approximation of a nonlinear dynamical system as a high-dimensional linear dynamical system, which in turn enables analysis in terms of eigendecomposition of the resulting linear operator, revealing basic modes of the dynamics. In the proposed method, DMD is used in the latent representations of a convolutional autoencoder for image sequences. The DMD objective is incorporated into the autoencoder training loss to minimize its reconstruction error. The DMD is also used, by conditioning on the eigenvalues, to split the reconstruction into high frequency (quickly varying) foreground modes and low-frequency (slowly varying) background modes. Although end-to-end training is mentioned, it is not made clear how the derivatives of the DMD decomposition are implemented, especially considering that the DMD involves an SVD, which can have unstable/ singular derivatives when two or more singular values are close to the same / exactly the same. The resulting methods are applied to a foreground extraction and a classification tasks, and compared with numerous baselines. It is not clear to me what the state of the art is on these tasks, but the proposed methods compare favorably to reported baselines, and the images of results look convincing. However the experimental results seem a little thin and I would expect a more thorough study. Overall the method looks very interesting. Some complaints: - the tables are a bit sloppy and should be formatted to fit in the document with normal sized fonts, - the images are too small to see well.
assistant
This paper presents an application of convolutional autoencoder networks and a nonlinear dynamic systems analysis method known as extended dynamic mode analysis (EDMD) to a data-driven analysis of multivariate time series. The DMD method appears to be well-known in the physics community but is outside my area of expertise and unfortunately I have limited time to make a quick study of it. However, from what I gather, it involves empirical approximation of a nonlinear dynamical system as a high-dimensional linear dynamical system, which in turn enables analysis in terms of eigendecomposition of the resulting linear operator, revealing basic modes of the dynamics. In the proposed method, DMD is used in the latent representations of a convolutional autoencoder for image sequences. The DMD objective is incorporated into the autoencoder training loss to minimize its reconstruction error. The DMD is also used, by conditioning on the eigenvalues, to split the reconstruction into high frequency (quickly varying) foreground modes and low-frequency (slowly varying) background modes. Although end-to-end training is mentioned, it is not made clear how the derivatives of the DMD decomposition are implemented, especially considering that the DMD involves an SVD, which can have unstable/ singular derivatives when two or more singular values are close to the same / exactly the same. The resulting methods are applied to a foreground extraction and a classification tasks, and compared with numerous baselines. It is not clear to me what the state of the art is on these tasks, but the proposed methods compare favorably to reported baselines, and the images of results look convincing. However the experimental results seem a little thin and I would expect a more thorough study. Overall the method looks very interesting. Some complaints: - the tables are a bit sloppy and should be formatted to fit in the document with normal sized fonts, - the images are too small to see well.