Deepström Networks
Recent work has focused on combining kernel methods and deep learning. With this in mind, we introduce Deepström networks -- a new architecture of neural networks which we use to replace top dense layers of standard convolutional architectures with an approximation of a kernel function by relying on the Nyström approximation. Our approach is easy highly flexible. It is compatible with any kernel function and it allows exploiting multiple kernels. We show that Deepström networks reach state-of-the-art performance on standard datasets like SVHN and CIFAR100. One benefit of the method lies in its limited number of learnable parameters which make it particularly suited for small training set sizes, e.g. from 5 to 20 samples per class. Finally we illustrate two ways of using multiple kernels, including a multiple Deepström setting, that exploits a kernel on each feature map output by the convolutional part of the model.
 DEEPSTRÖM NETWORKS Quality: good Originality: original Significance: relevant for ICLR Pros: - interesting idea -- see detailed comments Cons: a number of open issues in the presentation -- see detailed comments The paper is based on some concepts to link non-linear data representations by means of non-linear kernel mappings, with the deep learning approach. The presented concept is compared to former methods going into this direction and random fourier feature approaches. Additionally an adaptive scheme is suggested where the internally used landmark matrix (which forms a kind of mahalanobis like weighting matrix to the data) is adapted. I think the main idea of the paper is interesting but the presentation lacks a number of details in the description/parametrization and the experimental design. - *Indeed, since the Nyström approximation uses an explicit feature map from the data kernel matrix, it is not restricted to a specific kernel function and not limited only to RBF kernels, as in Fastfood approximation.* -- well yes an no, Nystroem works only effectively if the intrinsic dimensionality of the data is reasonable small - otherwise you need a lot of disjunct landmarks. Additionally the parameterization - how many landmarks and which one is not so easy - in this sense random fourier features are easier to handle. Further the random fourier features are not restricted to RBF kernel and not even to shift-invariant ones - there are a few proposals - The *feature map* obtained by the Nystroem approximation is only helpful as long as the number of landmarks is reasonable small - otherwise an out of sample extension (mapping new test points) in the spanned feature space may not be very reliable - please correct the spell errors like *using ful training sets* (full); *In (24), wo architectures* -- two? - random subsampling of the landmarks can lead to rather high approximation errors if the number of landmarks is small - there are some proposal to address this using leverage score sampling and other see e.g. work of K. Zhang, Suykens, C. Musco - *it is restricted to the Gaussian kernel* - well this is not true - see comments above (but I agree that there may not have been done much work to show this in practice, and it is probably not possible to be used for - all - kinds of psd kernels) - *Singular Value Decomposition (SVD) of K 11 . In the case where the size of the subsample L, m,is large, the computational complexity of the SVD is O(m 3 ).* -- well - yes and no, with Nystroem you would expect that N is much ! larger than m - than the m^3 may still be somewhat ok. To learn the weightings on the fly in your adaptive approach will address the complexity issue but may in parts degenerate performance - now it is a question what the user is more interested on - You mainly ignore the point how the landmarks have to be selected - random is (often) not a very reasonable choice - see respective work around this In your datasets the problem may not show up because you have images only which have very strong characteristics and are intrinsically very low dimensional. --> I think you have to add another section here - evaluating either different sampling schemes or just use a more state of the art one like leverage score sampling (getting I think O(N^2) complexity); you should also address how you find a good guess for the number of landmarks - I suggest to use also additional non-image data. Basically one motivation in your approach is to go beyond the classical deep learning data representation - this may not be so interesting for image recognition problems - I may suggest to spend a small subsection to a complexity analysis (including parameter optimization) of the various methods - Figure 2 - what do you mean by number of parameters? - there are a few strong fluctuations (fully connected, addative deep linear) - why - please increase the level of details of the accuracy axis in Figure 2 - so far it basically has 10% steps - if I have not missed it you did not say how the parameter .sigma (or .gamma) in the rbf kernel has been determined (same for other parametric kernels) - multiple kernels - well how did you combine the kernels (there are many options) - there are number of typos in the refs (see e.g. ref 14); some other references are very incomplete e.g. ref 15 - plots should be (prefered) close to the respective text (the appendix is a bit strange here)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Deepström Networks
Recent work has focused on combining kernel methods and deep learning. With this in mind, we introduce Deepström networks -- a new architecture of neural networks which we use to replace top dense layers of standard convolutional architectures with an approximation of a kernel function by relying on the Nyström approximation. Our approach is easy highly flexible. It is compatible with any kernel function and it allows exploiting multiple kernels. We show that Deepström networks reach state-of-the-art performance on standard datasets like SVHN and CIFAR100. One benefit of the method lies in its limited number of learnable parameters which make it particularly suited for small training set sizes, e.g. from 5 to 20 samples per class. Finally we illustrate two ways of using multiple kernels, including a multiple Deepström setting, that exploits a kernel on each feature map output by the convolutional part of the model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 DEEPSTRÖM NETWORKS Quality: good Originality: original Significance: relevant for ICLR Pros: - interesting idea -- see detailed comments Cons: a number of open issues in the presentation -- see detailed comments The paper is based on some concepts to link non-linear data representations by means of non-linear kernel mappings, with the deep learning approach. The presented concept is compared to former methods going into this direction and random fourier feature approaches. Additionally an adaptive scheme is suggested where the internally used landmark matrix (which forms a kind of mahalanobis like weighting matrix to the data) is adapted. I think the main idea of the paper is interesting but the presentation lacks a number of details in the description/parametrization and the experimental design. - *Indeed, since the Nyström approximation uses an explicit feature map from the data kernel matrix, it is not restricted to a specific kernel function and not limited only to RBF kernels, as in Fastfood approximation.* -- well yes an no, Nystroem works only effectively if the intrinsic dimensionality of the data is reasonable small - otherwise you need a lot of disjunct landmarks. Additionally the parameterization - how many landmarks and which one is not so easy - in this sense random fourier features are easier to handle. Further the random fourier features are not restricted to RBF kernel and not even to shift-invariant ones - there are a few proposals - The *feature map* obtained by the Nystroem approximation is only helpful as long as the number of landmarks is reasonable small - otherwise an out of sample extension (mapping new test points) in the spanned feature space may not be very reliable - please correct the spell errors like *using ful training sets* (full); *In (24), wo architectures* -- two? - random subsampling of the landmarks can lead to rather high approximation errors if the number of landmarks is small - there are some proposal to address this using leverage score sampling and other see e.g. work of K. Zhang, Suykens, C. Musco - *it is restricted to the Gaussian kernel* - well this is not true - see comments above (but I agree that there may not have been done much work to show this in practice, and it is probably not possible to be used for - all - kinds of psd kernels) - *Singular Value Decomposition (SVD) of K 11 . In the case where the size of the subsample L, m,is large, the computational complexity of the SVD is O(m 3 ).* -- well - yes and no, with Nystroem you would expect that N is much ! larger than m - than the m^3 may still be somewhat ok. To learn the weightings on the fly in your adaptive approach will address the complexity issue but may in parts degenerate performance - now it is a question what the user is more interested on - You mainly ignore the point how the landmarks have to be selected - random is (often) not a very reasonable choice - see respective work around this In your datasets the problem may not show up because you have images only which have very strong characteristics and are intrinsically very low dimensional. --> I think you have to add another section here - evaluating either different sampling schemes or just use a more state of the art one like leverage score sampling (getting I think O(N^2) complexity); you should also address how you find a good guess for the number of landmarks - I suggest to use also additional non-image data. Basically one motivation in your approach is to go beyond the classical deep learning data representation - this may not be so interesting for image recognition problems - I may suggest to spend a small subsection to a complexity analysis (including parameter optimization) of the various methods - Figure 2 - what do you mean by number of parameters? - there are a few strong fluctuations (fully connected, addative deep linear) - why - please increase the level of details of the accuracy axis in Figure 2 - so far it basically has 10% steps - if I have not missed it you did not say how the parameter .sigma (or .gamma) in the rbf kernel has been determined (same for other parametric kernels) - multiple kernels - well how did you combine the kernels (there are many options) - there are number of typos in the refs (see e.g. ref 14); some other references are very incomplete e.g. ref 15 - plots should be (prefered) close to the respective text (the appendix is a bit strange here)
assistant
 DEEPSTRÖM NETWORKS Quality: good Originality: original Significance: relevant for ICLR Pros: - interesting idea -- see detailed comments Cons: a number of open issues in the presentation -- see detailed comments The paper is based on some concepts to link non-linear data representations by means of non-linear kernel mappings, with the deep learning approach. The presented concept is compared to former methods going into this direction and random fourier feature approaches. Additionally an adaptive scheme is suggested where the internally used landmark matrix (which forms a kind of mahalanobis like weighting matrix to the data) is adapted. I think the main idea of the paper is interesting but the presentation lacks a number of details in the description/parametrization and the experimental design. - *Indeed, since the Nyström approximation uses an explicit feature map from the data kernel matrix, it is not restricted to a specific kernel function and not limited only to RBF kernels, as in Fastfood approximation. * -- well yes an no, Nystroem works only effectively if the intrinsic dimensionality of the data is reasonable small - otherwise you need a lot of disjunct landmarks. Additionally the parameterization - how many landmarks and which one is not so easy - in this sense random fourier features are easier to handle. Further the random fourier features are not restricted to RBF kernel and not even to shift-invariant ones - there are a few proposals - The *feature map* obtained by the Nystroem approximation is only helpful as long as the number of landmarks is reasonable small - otherwise an out of sample extension (mapping new test points) in the spanned feature space may not be very reliable - please correct the spell errors like *using ful training sets* (full); *In (24), wo architectures* -- two? - random subsampling of the landmarks can lead to rather high approximation errors if the number of landmarks is small - there are some proposal to address this using leverage score sampling and other see e.g. work of K. Zhang, Suykens, C. Musco - *it is restricted to the Gaussian kernel* - well this is not true - see comments above (but I agree that there may not have been done much work to show this in practice, and it is probably not possible to be used for - all - kinds of psd kernels) - *Singular Value Decomposition (SVD) of K 11 . In the case where the size of the subsample L, m,is large, the computational complexity of the SVD is O(m 3 ). * -- well - yes and no, with Nystroem you would expect that N is much ! larger than m - than the m^3 may still be somewhat ok. To learn the weightings on the fly in your adaptive approach will address the complexity issue but may in parts degenerate performance - now it is a question what the user is more interested on - You mainly ignore the point how the landmarks have to be selected - random is (often) not a very reasonable choice - see respective work around this In your datasets the problem may not show up because you have images only which have very strong characteristics and are intrinsically very low dimensional. --> I think you have to add another section here - evaluating either different sampling schemes or just use a more state of the art one like leverage score sampling (getting I think O(N^2) complexity); you should also address how you find a good guess for the number of landmarks - I suggest to use also additional non-image data. Basically one motivation in your approach is to go beyond the classical deep learning data representation - this may not be so interesting for image recognition problems - I may suggest to spend a small subsection to a complexity analysis (including parameter optimization) of the various methods - Figure 2 - what do you mean by number of parameters? - there are a few strong fluctuations (fully connected, addative deep linear) - why - please increase the level of details of the accuracy axis in Figure 2 - so far it basically has 10% steps - if I have not missed it you did not say how the parameter .sigma (or .gamma) in the rbf kernel has been determined (same for other parametric kernels) - multiple kernels - well how did you combine the kernels (there are many options) - there are number of typos in the refs (see e.g. ref 14); some other references are very incomplete e.g. ref 15 - plots should be (prefered) close to the respective text (the appendix is a bit strange here)