Convex Deep Learning via Normalized Kernels
Deep learning has been a long standing pursuit in machine learning, which until recently was hampered by unreliable training methods before the discovery of improved heuristics for embedded layer training. A complementary research strategy is to develop alternative modeling architectures that admit efficient training methods while expanding the range of representable structures toward deep models. In this paper, we develop a new architecture for nested nonlinearities that allows arbitrarily deep compositions to be trained to global optimality. The approach admits both parametric and nonparametric forms through the use of normalized kernels to represent each latent layer. The outcome is a fully convex formulation that is able to capture compositions of trainable nonlinear layers to arbitrary depth.
The submission describes a convex deep learning formulation that leverages a number of key ideas.  First, a training objective is proposed that explicitly includes the outputs of hidden layers as variables to be inferred via optimization.  These are linked to linear responses via a loss function, and the net objective is the sum of these loss functions across the layers, plus some regularization terms.  Next, a number of changes of variables are performed in order to reparameterize the objective into a convex form, heavily leveraging the representer theorem and the idea of value regularization.  We are left with a convex objective in terms of three different matrices (per layer) to optimize.  In particular, one of these matrices is a nonparametric ‘normalized output kernel’ matrix, which takes the place of optimizing over the hidden layer outputs directly; however, this leads to a transductive method where we must simultaneously solve the optimization for training and test inputs.  Finally, a relaxation must be performed in order to obtain a truly convex formulation, since the set of valid kernel matrices is generally non convex (possibly discrete, given assumptions on the hidden layer values).  Experiments are shown comparing the method to a previous two-layer convex model on small datasets.Quality:The basic ideas presented seem technically sound, although I was unable to rigorously verify many aspects of the derivation.  It is no small technical feat to achieve a convex deep learning formulation, and the authors make use of several very sophisticated tricks, gyrations, assumptions, and clever observations to achieve this goal.  Most of the derivation of the method consists of reparameterizations of the objective that leverage the representer theorem.However impressive this technical feat may be, my overall impression is that the resulting method is more of an academic curiosity than a practical method, for a few reasons.  First, the derivation of the method makes a few critical assumptions and relaxations to achieve a convex objective, such that we are left wondering whether the resulting model really captures the essence of a traditional deep network.  The most significant of these is relaxing the set of output kernels to a convex set.  The original motivation for the objective seemed to be a Boltzmann-machine-style model, where the hidden units are restricted to take on binary values--except the submission proposes to infer the hidden states deterministically.  The convex relaxation of the output kernel then effectively relaxes this binary constraint on the hidden units, which brings us a little farther away from the original motivation, and raises doubts as to how the new objective might be interpreted.  A similar argument can be made for the loss, which seems to replace a saturating loss with a hinge loss.  The submission gives little intuition as to why this is justified.  The transductive nature of the algorithm is also disappointing, and doesn’t seem scalable to large datasets.Unfortunately, I found several important steps in the derivation to be beyond my ability to comprehend.  However, I have at least one technical doubt that the authors may want to address—the reparameterizations of the objective induce constraints on the matrices that are not obviously convex constraints: for example, S \in MRK in Eq. 8, where M and K are in convex sets, and R is an appropriate matrix.  It is not clear to me whether this set is convex.  Perhaps the authors could provide a reference.  Also, I think \Phi U in Eq. 5 should be U \Phi.The experiments are very weak and add to my doubts regarding the practicality of the method.  The datasets used are small subsets of the original sets, leading me to believe that scalability is indeed an issue.  The only comparison is to a two-layer version of the method.  An obvious missing comparison is to a traditional deep net.  For that matter, even a comparison to a shallow method should at least be included as a baseline.  Also, I would be interested to see how nonlinear optimization of the original non convex objective fares compared to the convex version.Clarity:A lack of clarity is perhaps the weakest point of the submission.  The authors seem to have in mind a particular mental model and rationalization for their approach that does not come through in the submission.  For example, in several places, passing remarks are made about the nature of the loss function (e.g., convexity in one slot or the other), but no such assumptions are clearly made up-front.  It would also very much help to clearly delineate the connection and inspiration from neural-net-like Bayes nets or Boltzmann machines in order better motivate the objective.  For example, what might the loss look like concretely for a Bayes net, and how does this compare to the actual loss used?  What effects does the difference have on the expressive capacity of the method?  The submission should makes these issues more clear.I also found several critical points in the derivation to be lacking any kind of meaningful explanation.  This is especially true in the vicinity of Eq. 5-7.  This is the core of the method, but very little explanation is provided besides a few references to value regularization and some recent papers on clustering.  This is very disappointing, because I found section 2 to be fairly readable in comparison, but that section essentially ends in a dead-end.  I would have preferred it if section 2 were much shorter, and section 3 were longer.Originality:Although this work seems to be inspired by earlier work on two-level convex networks, and is based on the same basic model, the trick to generalize to arbitrary nesting seems novel enough.  The proposed optimization algorithm also seems novel.Significance:As the paper itself acknowledges, obtaining a convex training formulation is not a universally accepted aim of deep learning researchers, nor do local minima seem to be a serious problem in real networks.  The proposed training procedure is fairly complicated and seems to scale poorly, judging by the experiments.  The formulation is fairly inflexible, working only with a narrowly defined set of loss functions, and the resulting algorithm is transductive, which further hurts performance and ultimately limits scalability.  In the end, even though the optimization is convex, this is only obtained by making a critical relaxation that limits our ability to constrain the domain of the hidden layer outputs.  For all these reasons, the immediate practical impact of this work is likely to be very low.On the other hand, it is possible that this work could stimulate further work on the subject, since the very idea that a convex formulation might capture the important aspects of a deep network is interesting by itself.  Future work might elucidate the intrinsic theoretical limitations of this formulation versus traditional deep nets.