Higher-Order Factorization Machines
Factorization machines (FMs) are a supervised learning approach that can use second-order feature combinations even when the data is very high-dimensional. Unfortunately, despite increasing interest in FMs, there exists to date no efficient training algorithm for higher-order FMs (HOFMs). In this paper, we present the first generic yet efficient algorithms for training arbitrary-order HOFMs. We also present new variants of HOFMs with shared parameters, which greatly reduce model size and prediction times while maintaining similar accuracy. We demonstrate the proposed approaches on four different link prediction tasks.
The authors propose a learning algorithm for so called higher order FMs (HOFMs) based on ANOVA kernels and the use of HOFMs with shared parameters. A backpropagation like algorithm is used for training.
It is an interesting, reasoned and promising approach. But there a few issues which I would like to have clarified in the rebuttal to accept the paper. 1. The idea of the paper seems to strongly rely on the paper "Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms" by Blondel et al., where ANOVA kernels have already been used. Can you explain in more detail the difference and contributions in comparison to this paper? 2. There has been extensive work on tensor factorizations which I am missing here. E.g., many papers by Tamara Kolda. I'm wondering why such approaches cannot be applied to the given problem or why it is not better to adapt them to HOFMs. Such comparisons to state-of-the art tensor factorizations are accordingly also missing in the experiments. Are there reason for this? If so, please highlight them. 3. The experiments are not very extensive. There is just one quite specific setting based on the prediction of presence or absence of links in graphs which isn't explained in much detail. Moreover, your method does not consistently perform better than the baselines. Not just for GD, also the performance differences for the other data sets are low. For NIPS the low-rank bilinear regression comes close to the value attained by a HOFM. For Enzymes and Movielens a Polynomial Network comes very close. It should be noted that the values vary quite a lot for different orders m, even in a nonmonotonic way. For example, the difference between a Polynomial Network with m=3 and both m=2, m=4 is bigger than the difference between a Polynomial Network with m=2 (or m=4) and the best value for a HOFM. This makes me wonder whether those values didn't just happen by chance. You also seem to have tried more settings for HOFMs than for the baselines, so it could be man-made overfitting. Also the data sets are rather small which could raise scalability questions. More detailed experiments would be better in my opinion. At least these issues should be explained.