Answers.Body: <p>I do not feel comfortable proclaiming that I would be a machine learning expert.
  But I want to point out that there is indeed interest in applying tensor networks in machine learning settings.
  Let me highlight three particular settings in the following.</p>
  <p>They can be used to find the governing equations behind dynamical systems analogous to the <a href="https://www.pnas.org/content/113/15/3932" rel="nofollow noreferrer">SINDy algorithm</a>.
  The reasoning is that the dynamics of your system may not be sparse but may indeed have a low-rank structure. (See <a href="https://arxiv.org/abs/1607.04444" rel="nofollow noreferrer">this paper</a> for a comparison of sparse and low-rank structures and <a href="https://epubs.siam.org/doi/10.1137/18M1183480" rel="nofollow noreferrer">this paper</a> for a discussion about why low-rank matrices so often appear in big data.)
  The resulting modification of SINDy is the appropriately named <a href="https://arxiv.org/abs/1809.02448" rel="nofollow noreferrer">MANDy algorithm</a>.
  But if your dynamical law exhibits additional structure (despite being low-rank) then this structure is not used. (Requiring you to collect unnecessary samples.)
  <a href="https://arxiv.org/abs/2002.12388" rel="nofollow noreferrer">This work</a> tackles this issue by allowing you to specify the structure that you expect your law to exhibit.</p>
  <p>Another area of interest is the application of tensor networks as a means of compression.
  Tensor networks can be used to approximate multivariate functions.
  Either directly (see <a href="https://arxiv.org/abs/2007.00118" rel="nofollow noreferrer">this article</a> or <a href="https://www.mis.mpg.de/fileadmin/uschmajew/uschmajew_approximationPreprint.pdf" rel="nofollow noreferrer">this article</a> for further details) or by representing their high-dimensional coefficient tensor (see e.g. <a href="https://link.springer.com/article/10.1007/s00211-016-0850-x" rel="nofollow noreferrer">here</a>).
  The last paper also highlights where such functions may occur: when solving stochastic or parametric PDEs.
  &quot;What does this have to do with machine learning?&quot; you may rightfully ask.
  These PDEs are hard to solve and it is often easier to minimize a residual than to apply a Galerkin method. The results can be shown to be equivalent with high probability. (See <a href="https://arxiv.org/abs/1810.01348" rel="nofollow noreferrer">this paper</a> for a proof of this statement and an application to some stochastic PDEs or <a href="https://arxiv.org/abs/1911.00279" rel="nofollow noreferrer">this paper</a> for an application to an optimal control problem.)</p>
  <p>Moreover, there is the problem of tensor completion - a generalization of matrix completion to higher dimensions. (<a href="https://arxiv.org/abs/1701.08045" rel="nofollow noreferrer">This</a> and <a href="https://arxiv.org/abs/1405.1773" rel="nofollow noreferrer">this</a> are references for algorithms.)
  These methods find application in data science where they can be used to decompose or denoise data. (Unfortunately, I don't have a reference with examples but I remember a presentation where these methods where used on EEG signals.)</p>
  <p>Additionally, I recently found this interesting symposium covering even more potential applications of tensor networks in machine learning:
  <a href="https://itsatcuny.org/calendar/quantum-inspired-machine-learning" rel="nofollow noreferrer">https://itsatcuny.org/calendar/quantum-inspired-machine-learning</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T23:32:31.263Z
Answers.LastActivityDate: 2021-01-18T23:58:29.090Z
Answers.LastEditDate: 2021-01-18T23:58:29.090Z
Answers.UserDisplayName: user2070206
Body: <p>Tensor networks (check <a href="https://arxiv.org/pdf/1812.04011.pdf" rel="nofollow noreferrer">this paper</a> for a review) are a numerical method originally introduced in condensed matter physics to model complex quantum systems. Roughly speaking, such systems are described by a very high-dimensional tensor (where the indices take a number of values scaling exponentially with the number of system constituents) and tensor networks provide an efficient representation of the latter as an outer product and contraction of many low-dimensional tensors.</p>
  
  <p>More recently, a specific kind of tensor network (called Matrix Product State in physics) found interesting applications in machine-learning through the so-called Tensor-Train decomposition (I do not know of a precise canonical reference in this context, so I will abstain from citing anything).</p>
  
  <p>Now, over the last few years, several works from the physics community seemed to push for a generalized use of tensor networks in machine learning (see <a href="https://arxiv.org/pdf/1605.05775.pdf" rel="nofollow noreferrer">this paper</a>, <a href="https://arxiv.org/pdf/1910.07425.pdf" rel="nofollow noreferrer">a second one</a> and <a href="https://arxiv.org/pdf/2001.08286.pdf" rel="nofollow noreferrer">a third one</a> and <a href="https://ai.googleblog.com/2019/06/introducing-tensornetwork-open-source.html" rel="nofollow noreferrer">this article</a> from Google AI for context). As a physicist, I am glad to learn that tools initially devised for physics may find interdisciplinary applications. However, at the same time, my critical mind tells me that from the machine learning research community's perspective, these results may not look that intriguing. After all, machine learning is now a very established field and it takes probably more than a suggestion for a new machine learning model and a basic benchmarking on a trivial dataset (as the MNIST one) -which is what the papers essentially do in my humble opinion- to attract any attention in the area. Besides, as I believe to know, there already exists quite a solid body of knowledge on tensor analysis techniques for machine learning (e.g.  tensor decompositions), which may cast doubt on the originality of the contribution.</p>
  
  <p>I would therefore be very curious to have the opinion of machine learning experts on this line of research: is it really an interesting direction to look into, or is it just about surfing on the current machine learning hype with a not-so-serious proposal?</p>
  
Tags: <neural-networks><machine-learning>
Title: Using tensor networks as machine learning models
UserDisplayName: drose
ThreadText: # Using tensor networks as machine learning models
  
  (Asked by: username_0 on 2020-02-22)
  
  Tensor networks (check [this paper](https://arxiv.org/pdf/1812.04011.pdf) for a review) are a numerical method originally introduced in condensed matter physics to model complex quantum systems. Roughly speaking, such systems are described by a very high-dimensional tensor (where the indices take a number of values scaling exponentially with the number of system constituents) and tensor networks provide an efficient representation of the latter as an outer product and contraction of many low-dimensional tensors.
  
  More recently, a specific kind of tensor network (called Matrix Product State in physics) found interesting applications in machine-learning through the so-called Tensor-Train decomposition (I do not know of a precise canonical reference in this context, so I will abstain from citing anything).
  
  Now, over the last few years, several works from the physics community seemed to push for a generalized use of tensor networks in machine learning (see [this paper](https://arxiv.org/pdf/1605.05775.pdf), [a second one](https://arxiv.org/pdf/1910.07425.pdf) and [a third one](https://arxiv.org/pdf/2001.08286.pdf) and [this article](https://ai.googleblog.com/2019/06/introducing-tensornetwork-open-source.html) from Google AI for context). As a physicist, I am glad to learn that tools initially devised for physics may find interdisciplinary applications. However, at the same time, my critical mind tells me that from the machine learning research community's perspective, these results may not look that intriguing. After all, machine learning is now a very established field and it takes probably more than a suggestion for a new machine learning model and a basic benchmarking on a trivial dataset (as the MNIST one) -which is what the papers essentially do in my humble opinion- to attract any attention in the area. Besides, as I believe to know, there already exists quite a solid body of knowledge on tensor analysis techniques for machine learning (e.g. tensor decompositions), which may cast doubt on the originality of the contribution.
  
  I would therefore be very curious to have the opinion of machine learning experts on this line of research: is it really an interesting direction to look into, or is it just about surfing on the current machine learning hype with a not-so-serious proposal?
  
  **Tags:** neural-networks, machine-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-18. Score: 1
  
  I do not feel comfortable proclaiming that I would be a machine learning expert.
  But I want to point out that there is indeed interest in applying tensor networks in machine learning settings.
  Let me highlight three particular settings in the following.
  
  They can be used to find the governing equations behind dynamical systems analogous to the [SINDy algorithm](https://www.pnas.org/content/113/15/3932).
  The reasoning is that the dynamics of your system may not be sparse but may indeed have a low-rank structure. (See [this paper](https://arxiv.org/abs/1607.04444) for a comparison of sparse and low-rank structures and [this paper](https://epubs.siam.org/doi/10.1137/18M1183480) for a discussion about why low-rank matrices so often appear in big data.)
  The resulting modification of SINDy is the appropriately named [MANDy algorithm](https://arxiv.org/abs/1809.02448).
  But if your dynamical law exhibits additional structure (despite being low-rank) then this structure is not used. (Requiring you to collect unnecessary samples.)
  [This work](https://arxiv.org/abs/2002.12388) tackles this issue by allowing you to specify the structure that you expect your law to exhibit.
  
  Another area of interest is the application of tensor networks as a means of compression.
  Tensor networks can be used to approximate multivariate functions.
  Either directly (see [this article](https://arxiv.org/abs/2007.00118) or [this article](https://www.mis.mpg.de/fileadmin/uschmajew/uschmajew_approximationPreprint.pdf) for further details) or by representing their high-dimensional coefficient tensor (see e.g. [here](https://link.springer.com/article/10.1007/s00211-016-0850-x)).
  The last paper also highlights where such functions may occur: when solving stochastic or parametric PDEs.
  "What does this have to do with machine learning?" you may rightfully ask.
  These PDEs are hard to solve and it is often easier to minimize a residual than to apply a Galerkin method. The results can be shown to be equivalent with high probability. (See [this paper](https://arxiv.org/abs/1810.01348) for a proof of this statement and an application to some stochastic PDEs or [this paper](https://arxiv.org/abs/1911.00279) for an application to an optimal control problem.)
  
  Moreover, there is the problem of tensor completion - a generalization of matrix completion to higher dimensions. ([This](https://arxiv.org/abs/1701.08045) and [this](https://arxiv.org/abs/1405.1773) are references for algorithms.)
  These methods find application in data science where they can be used to decompose or denoise data. (Unfortunately, I don't have a reference with examples but I remember a presentation where these methods where used on EEG signals.)
  
  Additionally, I recently found this interesting symposium covering even more potential applications of tensor networks in machine learning:
  <https://itsatcuny.org/calendar/quantum-inspired-machine-learning>
  
  
  ---
  
  