Probabilistic Matrix Factorization for Automated Machine Learning
In order to achieve state-of-the-art performance, modern machine learning techniques require careful data pre-processing and hyperparameter tuning. Moreover, given the ever increasing number of machine learning models being developed, model selection is becoming increasingly important. Automating the selection and tuning of machine learning pipelines, which can include different data preprocessing methods and machine learning models, has long been one of the goals of the machine learning community. In this paper, we propose to solve this meta-learning task by combining ideas from collaborative filtering and Bayesian optimization. Specifically, we use a probabilistic matrix factorization model to transfer knowledge across experiments performed in hundreds of different datasets and use an acquisition function to guide the exploration of the space of possible pipelines. In our experiments, we show that our approach quickly identifies highperforming pipelines across a wide range of datasets, significantly outperforming the current state-of-the-art.
This work presents an approach to deal with Automatic Machine Learning, optimizing not only the hyperparameters of a Machine Learning model but also different machine learning models to choose the best one, what it calls pipelines. This idea is implemented in Autoweka (SMAC), and also TPE can handle this task. In order to do so, it uses Probabilistic Matrix Factorization.  Related to:  Gaussian Process priors over this technique are used to predict the performance of pipelines in a new dataset. Mnih, Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems. 2008. The AutoML task is already solved: Kotthoff, Lars, et al. "Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA." The Journal of Machine Learning Research 18.1 (2017): 826-830. The surrogate model of AutoWeka is SMAC, which is proven to be a robust (and simple!) solution to this problem.  Strengths: -> Clever use of Probabilistic Matrix Factorization to solve this task. -> Experiments are performed in a plethora of datasets. -> Good ideas for further work.  Weaknesses: -> Contribution is low. No new models are presented. The contribution is employing Probabilistic Matrix Factorization to solve this problem. -> Clarity and organization. I think that this paper may be hard to read if you are not an expert of the area. Sometimes, variables are not provided an intuitive interpretation. -> The title is misleading. I would suggest to change it to Automatic Machine Learning via Collaborative Filtering. High-dimensional BO can exist in a problem with 10000 variables for example. -> In the experiments, some "hyper-hyperparameters" of this approach are given static values without justification. -> I think that this is a work in progress, at least not complete for a NIPS conference level. Many ideas remain to be implemented.  Does this submission add value to the NIPS community? : It has the potential, but not yet. I think that, besides the contribution is not as high as providing a new model, it is clever. Some of the presented further work must be added to this paper. I would highlight the addition of specific information of the datasets to the model.  Quality: Is this submission technically sound?: Not at all, borderline at least. Although it is clever to use Probabilistic Matrix Factorization, this task is already solved and only having taken into account the accuracy and not other features is a limitation. By adding these features that other approaches do not take into account, the soundness of the paper would be higher. Are claims well supported by theoretical analysis or experimental results?: No theoretical analysis is provided and experiments support well the results talking about quantity but I am not so sure about having set some of the variables of the experiments to a fixed value (Q=20, LR = 10-7, batch-size = 50). Is this a complete piece of work or work in progress?: Work in progress. Further work, really well explained, in the conclusions section must be presented in order for this paper to be accepted. Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?: I miss more analysis and explanations about the strenghts and weaknesses of the approach. Some are given, but it is not enough.  Clarity: Is the submission clearly written?: Not at all. This paper is hard to read for newcomers or even researchers that are not experts. Some areas are given too much detail (for example the explanation of the EI acquisition function, I would just cite it). With that gained space, I will clearly explain what does \mathbf{X}, \mathbf{x}_i and \mathbf{x}_j mean. There are also some quantities that are not explained, as const in \mathbf{NLL}, the implications of \eta, the learning rate, but above all, in the experiments section, an interpretation of Q, \eta and batch size values chosen must be provided. Is it well organized?: I would change some things to improve organization. Regarding the abstract, the first paragraph, until "learning community" is the standard literature of Bayesian Optimization that applies to all BO papers. In this case, all this information that does not describe this particular paper represents more than half of the abstract. Also, the other paragraph lacks cohesion with the first one. Regarding introduction, the third paragraph "Bayesian optimization techniques" should be a continuation of the first one, for coherence. Other critical problem is that, based on the title, I expected a complete different thing in the paper until I read from line 44 on advance. This information should be stated earlier or the title should be changed. I encourage to rewrite the abstract to make clear the pipeline topic. Another issue, section 3.1 is not expected when you finished reading section 3. I expected what is explained in section 3.2. I would change the last paragraph of section 3, introducing section 3.1, to add cohesion. Does it adequately inform the reader?: By solving the mentioned issues, I think so.  Originality: Are the tasks or methods new?: No. The methodology is built from well-known techniques. Is the work a novel combination of well-known techniques?: Yes, Probabilistic Matrix Factorization, Gaussian Processes and ML algorithms. Is it clear how this work differs from previous contributions?: Yes, I have not read nothing similar nor the related work presents nothing like the contribution of the authors. Is related work adequately cited?: Yes, the authors did a good job here.  Significance: Are the results important?: Currently not. But, if more features are added of dataset, it could be interesting to analyze that results. Are others likely to use the ideas or build on them?: Same answer as before. Does the submission address a difficult task in a better way than previous work?: Not in my opinion. I prefer the SMAC methodology. But I admit that this opinion can be subjective. I prefer that the model can handle all and here the addition of Probabilistic Matrix Factorization adds more complexity. Does it advance the state of the art in a demonstrable way?: Currently not. I am not sure about the fixed values of the experiments and the comparison with others. Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?: It does provide some unique conclusions, but further work must be implemented. Arguments for acceptance:  -> New methodology based on GPs and Probabilistic Matrix Factorization for Automatic Machine Learning.  Arguments against acceptance:  -> It is just a combination of well-known techniques that does not solve a new problem and its potential is not fully developed (further work). -> I would add more experiments. -> Title is misleading. -> Clarity and Quality are poor. Issues must be corrected. Line 138. Y -> \mathbf{Y}  -> Abstract, line 14: Name the state of the art techniques. -> Abstract, line 12: Define pipeline. -> Introduction, line 22: That is an assumption that you are considering. Define that fact as an assumption, parameter space should be formalized. -> Equation of line 27: Put a number and \mathbf{x} is a matrix, so capital letter. -> The instatiations of the pipelines that you are considering are a restriction for the experiments wrt the theory. State that this is a constraint. -> Line 57: Demonstrate empirically. -> Line 119: That "idea" is a hypothesis that you are considering. Define that idea as a hypothesis. -> Line 153: Why do you choose the Squared Exponential Kernel over the Matern kernel if the Matern kernel is preferred? -> Complexity analysis: How does this methodology scale wrt pipelines/datasets?  To sum up, I would consider giving a borderline to this paper if all these demands are satisfied.  If there are not satisfied I can not recommend this paper although I liked the idea of using Probabilistic Matrix Factorization.