Supervised Learning with Tensor Networks
Tensor networks are approximations of high-order tensors which are efficient to work with and have been very successful for physics and mathematics applications. We demonstrate how algorithms for optimizing tensor networks can be adapted to supervised learning tasks by using matrix product states (tensor trains) to parameterize non-linear kernel learning models. For the MNIST data set we obtain less than 1% test set classification error. We discuss an interpretation of the additional structure imparted by the tensor network to the learned model.
This paper deals with the use of tensor networks to perform supervised learning. Specifically, the authors shows how tensor networks can be adapted to produce useful representation of data and solve classification problems. The main idea is to consider data feature maps that are built from the tensor product of local features and represent them as a higher-order tensor. A classification problem then consists in viewing the model parameters as a tensor network and learn them using a matrix-product state (MPS) decomposition. An optimization algorithm to find the MPS decomposition is described and an experiment on MNIST digit data set is given. Finally, a discussion on the class of functions in which the solution is searched is provided.
The idea of using tensor networks for machine learning problems is interesting. Unfortunately, there are many important issues that prevent the publication of the manuscript as it is now. My main concerns are the following:   1- Some parts of the paper are hard to read. The paper applies and adapts some idea about tensor networks that are used/developed by the Physics community to ML problems. The authors have done a good job in illustrating the idea and the intuition by many figures. However, I think that for a ML audience, giving more details on tensor networks, MPS decomposition algorithms and their use for Physics problems (in short, more details on the papers cited and published in Physics journals) will improve the readability and the accessibility of the paper. A section can be dedicated to this.  2- The notation is also heavy to follow and dos not help understanding. For example:  * In equation 2, what is the relation between x and x_j? Is x_j a component of the vector x? What is N: is it the number of attribute?  * In equation 3, \phi is indexed by s_j but depends only in x_j. For this example, is it the same \phi for all the s_j?  * A needs to be defined in equation 5  * line 68: ‘in classifying labeled data’ should be ‘unlabeled data’ (from a given training set)  * …  3- Experiments are weak. To assess the performance of the method, more data sets have to be used and not only the simple MNIST digit data. Also, some informations about the experimental setting are missing. For example, How the data set was divided in training and testing data?  4- I think that the choice of the local feature map is crucial. It is not clear how this feature map can be chosen. The one given in Eq. 3 seems to be adapted only for image data. The same for the parameters d and m. How these parameters can be chosen?  5- Is the MPS decomposition unique? If no, how this affects the learning process?  6- MPS decomposition for feature extraction and classification were used in [4]. I think that the novel contributions compared to [4] need to be clearly stated. Line 210 is too short and does not give enough details to see the difference between the two works.  7- ANOVA kernels are based on the product of a set of ‘local’ kernels. The product of the kernels allows to find a solution in a Hilbert space  constructed from the tensor product of the reproducing Hilbert spaces associated to each local kernel. I Think that it is interesting to study the link between the space of functions obtained by such kernels and the tensor network representation proposed here.