Answers.Body: <p>State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask <em>when</em> you might prefer to try an SVM.</p>
  
  <p>SVMs have several advantages:</p>
  
  <ol>
  <li>Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear combinations of features, rather than the original feature set. In contrast, a more modern approach like a deep neural network will need to get deeper or wider to model the same patterns, which will increase its training time.</li>
  <li>SVMs have an inherent bias towards picking "conservative" hypotheses, that are less likely to overfit the data, because they try to find maximum margin hypotheses. In some sense, they "bake-in" Occam's razor.</li>
  <li>SVMs have only two hyperparameters (the choice of kernel and the regularization constant), so they are very easy to tune to specific problems. It is usually sufficient to tune them by performing a simple grid-search through the parameter space, which can be done automatically.</li>
  </ol>
  
  <p>SVMs also have some disadvantages:</p>
  
  <ol>
  <li>SVMs have a runtime that scales cubically in the number of datapoints you want to train on (i.e. <span class="math-container">$O(n^3)$</span> runtime)<sup>1</sup>. This does not compare well with, say, a typical training approach for a deep neural network which runs in <span class="math-container">$O(w*n*e)$</span> time, where <span class="math-container">$n$</span> is the number of data points, <span class="math-container">$e$</span> is the number of training epochs, and <span class="math-container">$w$</span> is the number of weights in the network. Generally <span class="math-container">$w, e &lt;&lt; n$</span>.</li>
  <li>To make use of the Kernel trick, SVMs cache a value for the kernelized "distance" between any two pairs of points. This means they need <span class="math-container">$O(n^2)$</span> memory. This is far, far, more trouble than the cubic runtime on most real-world sets. More than a few thousand datapoints will leave most modern servers <a href="https://en.wikipedia.org/wiki/Thrashing_(computer_science)" rel="noreferrer">thrashing</a>, which increases effective runtime by several orders of magnitude. Together with point 1, this means SVMs will tend to become unworkably slow for sets beyond maybe 5,000-10,000 datapoints, at the upper limit. </li>
  </ol>
  
  <p>All of these factors point to SVMs being relevant for exactly one use case: small datasets where the target pattern is thought, apriori, to be some regular, but highly non-linear, function of a large number of features. This use case actually arises fairly often. A recent example application where I found SVMs to be a natural approach was building predictive models for a target function that was known to be the result of interactions between pairs of features (specifically, communications between pairs of agents). An SVM with a quadratic kernel could therefore efficiently learn conservative, reasonable, guesses.</p>
  
  <hr>
  
  <p><sup>1</sup> There are approximate algorithms that will solve the SVM faster than this, as noted in the other answers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-23T15:31:36.133Z
Answers.LastActivityDate: 2019-09-26T23:10:13.717Z
Answers.LastEditDate: 2019-09-26T23:10:13.717Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>Deep Learning and Neural Networks are getting most of the focus because of recent advances in the field and most experts believe it to be the future of solving machine learning problems. </p>
  
  <p>But make no mistake, classical models still produce exceptional results and in certain problems, they can produce better results than deep learning. </p>
  
  <p>Linear Regression is still by far the most used machine learning algorithm in the world. </p>
  
  <p>It’s difficult to identify a specific domain where classical models always perform better as the accuracy is very much determined on the shape and quality of the input data. </p>
  
  <p>So algorithm and model selection is always a trade-off. It’s a somewhat accurate statement to make that classical models still perform better with smaller data sets. However, a lot of research is going into improving deep learning model performance on less data.</p>
  
  <p>Most classical models require less computational resources so if your goal is speed then its much better. </p>
  
  <p>Also, classical models are easier to implement and visualize which can be another indicator for performance, but it depends on your goals.</p>
  
  <p>If you have unlimited resources, a massive observable data set that is properly labeled and you implement it correctly within the problem domain then deep learning is likely going to give you better results in most cases. </p>
  
  <p>But in my experience, the real-world conditions are never this perfect</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-23T02:18:33.203Z
Answers.LastActivityDate: 2019-04-23T02:44:20.910Z
Answers.LastEditDate: 2019-04-23T02:44:20.910Z
Answers.UserDisplayName: IsakBosman
Answers.Body: <p>Totally agree with @John's answer. Will try and complement that with some more points.</p>
  
  <p>Some advantages of SVMs:</p>
  
  <p>a) SVM is defined by a convex optimisation problem for which there are <a href="http://cs229.stanford.edu/notes/cs229-notes3.pdf" rel="nofollow noreferrer">efficient methods to solve, like SMO</a>.</p>
  
  <p>b) Effective in high dimensional spaces and also in cases where number of dimensions is greater than the number of samples.</p>
  
  <p>c) Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.</p>
  
  <p>d) <a href="https://en.wikipedia.org/wiki/Kernel_method" rel="nofollow noreferrer">Different Kernel functions can be specified for the decision function.</a>. In its simplest form, the kernel trick means transforming data into another dimension that has a clear dividing margin between classes of data.</p>
  
  <p>The disadvantages of support vector machines include:</p>
  
  <p>a) If the number of features is much greater than the number of samples, avoiding over-fitting in choosing Kernel functions and regularization term is crucial. Kernel models can be quite sensitive to <a href="http://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley10a.pdf" rel="nofollow noreferrer">over-fitting the model selection criterion</a></p>
  
  <p>b) SVMs do not directly provide probability estimates. In many classification problems you actually want the probability of class membership, so it would be better to use a method like Logistic Regression, rather than post-process the output of the SVM to get probabilities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-27T11:11:03.260Z
Answers.LastActivityDate: 2019-09-28T14:35:26.720Z
Answers.LastEditDate: 2019-09-28T14:35:26.720Z
Answers.UserDisplayName: naive
Answers.Body: <p>For datasets of low-dimensional tabular data. DNN are not efficient on low-dimensional input because of huge overparametrisation. So even if dataset is huge in size, but each sample is low-dimensional SVM would beat DNN.</p>
  
  <p>More generally if data is tabular and the correlation between fields of the sample is weak and noisy, SVM may still beat DNN even for high dimensional data, but that depend on specific of data.</p>
  
  <p>Unfortunately I can't recall any specific papers on subject, so it's mostly common sense reasoning, you don't have to trust it. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-28T05:31:09.047Z
Answers.LastActivityDate: 2019-04-28T05:31:09.047Z
Answers.UserDisplayName: mirror2image
Body: <p>It seems that deep neural networks and other neural network based models are dominating many current areas like computer vision, object classification, reinforcement learning, etc.</p>
  
  <p>Are there domains where SVMs (or other models) are still producing state-of-the-art results?</p>
  
Tags: <machine-learning><support-vector-machine><state-of-the-art>
Title: What are the domains where SVMs are still state-of-the-art?
UserDisplayName: Steven Davis
ThreadText: # What are the domains where SVMs are still state-of-the-art?
  
  (Asked by: username_0 on 2019-04-23)
  
  It seems that deep neural networks and other neural network based models are dominating many current areas like computer vision, object classification, reinforcement learning, etc.
  
  Are there domains where SVMs (or other models) are still producing state-of-the-art results?
  
  **Tags:** machine-learning, support-vector-machine, state-of-the-art
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2019-04-23. Score: 14
  
  State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask *when* you might prefer to try an SVM.
  
  SVMs have several advantages:
  
  1. Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear combinations of features, rather than the original feature set. In contrast, a more modern approach like a deep neural network will need to get deeper or wider to model the same patterns, which will increase its training time.
  2. SVMs have an inherent bias towards picking "conservative" hypotheses, that are less likely to overfit the data, because they try to find maximum margin hypotheses. In some sense, they "bake-in" Occam's razor.
  3. SVMs have only two hyperparameters (the choice of kernel and the regularization constant), so they are very easy to tune to specific problems. It is usually sufficient to tune them by performing a simple grid-search through the parameter space, which can be done automatically.
  
  SVMs also have some disadvantages:
  
  1. SVMs have a runtime that scales cubically in the number of datapoints you want to train on (i.e. $O(n^3)$ runtime)1. This does not compare well with, say, a typical training approach for a deep neural network which runs in $O(w\*n\*e)$ time, where $n$ is the number of data points, $e$ is the number of training epochs, and $w$ is the number of weights in the network. Generally $w, e << n$.
  2. To make use of the Kernel trick, SVMs cache a value for the kernelized "distance" between any two pairs of points. This means they need $O(n^2)$ memory. This is far, far, more trouble than the cubic runtime on most real-world sets. More than a few thousand datapoints will leave most modern servers [thrashing](https://en.wikipedia.org/wiki/Thrashing_(computer_science)), which increases effective runtime by several orders of magnitude. Together with point 1, this means SVMs will tend to become unworkably slow for sets beyond maybe 5,000-10,000 datapoints, at the upper limit.
  
  All of these factors point to SVMs being relevant for exactly one use case: small datasets where the target pattern is thought, apriori, to be some regular, but highly non-linear, function of a large number of features. This use case actually arises fairly often. A recent example application where I found SVMs to be a natural approach was building predictive models for a target function that was known to be the result of interactions between pairs of features (specifically, communications between pairs of agents). An SVM with a quadratic kernel could therefore efficiently learn conservative, reasonable, guesses.
  
  ---
  
  1 There are approximate algorithms that will solve the SVM faster than this, as noted in the other answers.
  
  
  ---
  
  ### Answer by username_2 on 2019-04-23. Score: 7
  
  Deep Learning and Neural Networks are getting most of the focus because of recent advances in the field and most experts believe it to be the future of solving machine learning problems.
  
  But make no mistake, classical models still produce exceptional results and in certain problems, they can produce better results than deep learning.
  
  Linear Regression is still by far the most used machine learning algorithm in the world.
  
  It’s difficult to identify a specific domain where classical models always perform better as the accuracy is very much determined on the shape and quality of the input data.
  
  So algorithm and model selection is always a trade-off. It’s a somewhat accurate statement to make that classical models still perform better with smaller data sets. However, a lot of research is going into improving deep learning model performance on less data.
  
  Most classical models require less computational resources so if your goal is speed then its much better.
  
  Also, classical models are easier to implement and visualize which can be another indicator for performance, but it depends on your goals.
  
  If you have unlimited resources, a massive observable data set that is properly labeled and you implement it correctly within the problem domain then deep learning is likely going to give you better results in most cases.
  
  But in my experience, the real-world conditions are never this perfect
  
  
  ---
  
  ### Answer by username_3 on 2019-04-27. Score: 0
  
  Totally agree with @username_1's answer. Will try and complement that with some more points.
  
  Some advantages of SVMs:
  
  a) SVM is defined by a convex optimisation problem for which there are [efficient methods to solve, like SMO](http://cs229.stanford.edu/notes/cs229-notes3.pdf).
  
  b) Effective in high dimensional spaces and also in cases where number of dimensions is greater than the number of samples.
  
  c) Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
  
  d) [Different Kernel functions can be specified for the decision function.](https://en.wikipedia.org/wiki/Kernel_method). In its simplest form, the kernel trick means transforming data into another dimension that has a clear dividing margin between classes of data.
  
  The disadvantages of support vector machines include:
  
  a) If the number of features is much greater than the number of samples, avoiding over-fitting in choosing Kernel functions and regularization term is crucial. Kernel models can be quite sensitive to [over-fitting the model selection criterion](http://jmlr.csail.mit.edu/papers/volume11/cawley10a/cawley10a.pdf)
  
  b) SVMs do not directly provide probability estimates. In many classification problems you actually want the probability of class membership, so it would be better to use a method like Logistic Regression, rather than post-process the output of the SVM to get probabilities.
  
  
  ---
  
  ### Answer by username_4 on 2019-04-28. Score: -1
  
  For datasets of low-dimensional tabular data. DNN are not efficient on low-dimensional input because of huge overparametrisation. So even if dataset is huge in size, but each sample is low-dimensional SVM would beat DNN.
  
  More generally if data is tabular and the correlation between fields of the sample is weak and noisy, SVM may still beat DNN even for high dimensional data, but that depend on specific of data.
  
  Unfortunately I can't recall any specific papers on subject, so it's mostly common sense reasoning, you don't have to trust it.
  
  
  ---
  
  