Extracting relevant information, causally inferring and predicting the future states with high accuracy is a crucial task for modeling complex systems. The endeavor to address these tasks is made even more challenging when we have to deal with high-dimensional heterogeneous data streams. Such data streams often have higher-order inter-dependencies across spatial and temporal dimensions. We propose to perform a soft-clustering of the data and learn its dynamics to produce a compact dynamical model while still ensuring the original objectives of causal inference and accurate predictions. To efficiently and rigorously process the dynamics of soft-clustering, we advocate for an information theory inspired approach that incorporates stochastic calculus and seeks to determine a trade-off between the predictive accuracy and compactness of the mathematical representation. We cast the model construction as a maximization of the compression of the state variables such that the predictive ability and causal interdependence (relatedness) constraints between the original data streams and the compact model are closely bounded. We provide theoretical guarantees concerning the convergence of the proposed learning algorithm. To further test the proposed framework, we consider a high-dimensional Gaussian case study and describe an iterative scheme for updating the new model parameters. Using numerical experiments, we demonstrate the benefits on compression and prediction accuracy for a class of dynamical systems. Finally, we apply the proposed algorithm to the real-world dataset of multimodal sentiment intensity and show improvements in prediction with reduced dimensions.
The paper proposes a method to learn the conditional distribution of a random variable in order to minimize and maximize certain mutual information terms.  Interestingly, the proposed method can be applied to sentiment prediction and outperforms a 2018 method based on SVM.
Overall, the ideas seem intriguing, and the results seem promising, but I really cannot understand what the paper is saying, and I think the paper would be much stronger if it was written more clearly (to make individual sentences more clear, but also to make the broader picture more clear). Not only is the writing hard to understand (some sentences lack a verb!), but it is vague, and the notion of a "complex system" is never defined.  It seems that the technique can be applied to any (potentially non-stationary) Markov process?
Additionally, due to the lack of clarity in the writing and lack of mathematical rigor, Theorem 1 does not seem to be true as stated. I think this is an issue of stating the assumptions, and not due to a mistake in the derivation.  Right now, the actual conclusion of theorem 1 is not even clear to me.
Quality: poor/unclear
Clarity: very poor
Originality: unclear, perhaps high? Not clear how related it is to the methods of Tishby et al.
Significance: unclear, as clarity was poor, and there was minimal discussion of alternative methods.
Specific points:
- Eq (2), the first term is included because it is for the "information compression task", but I do not understand that. Where is the actual compression?  This is not traditional compression (turning a large vector into a smaller vector), but more like turning one PDF into a PDF with lower entropy?
- This paper seems to fall into the subfield of system identification (at which I am not an expert), so I'd expect to see some related literature in the field. The only compared method was the IF method of Tishby et al. from 18 years ago (and the current work seems to be a generalization of that).
- Equation (4): what exactly is the object being minimized? Is it a PDF/probability measure? Is it an *instance* of a random variable?  If it is a PDF, is it the PDF of B_k | X_{k-1} ?
- The statement of Theorem 1 is either too vague or wrong. To say "The solution... is given by" makes it sound like you are giving equations that define a unique solution. Perhaps you mean, "Any solution ... must necessarily satisfy..." ? And that is not clearly true without more work. You are basically saying that any minimizer must be a stationary point of the objective (since you are not assuming convexity). It seems everything is differentiable?  How do you know solutions even exist -- what if it is unbounded? In that case, these are not necessary conditions.
- Lemma 1: "The iterative procedure... is convergent."  The iterative procedure was never defined, so I don't even know what to make of this.
- Section 3.2: "As proved by prior work, the optimum solution obtained by a stochastic transformation that is jointly Gaussian with bottleneck's input."  I do not know what you are trying to say here. There's no predicate.
- Section 4 wasn't that interesting to me yet, since it was abstract and it seemed possible that you make a model to fit your framework well. But section 5 is much better, since you apply it to a real problem. However, what you are actually solving in section 5 is unclear. The entire setup is poorly described, so I am very confused.