Answers.Body: <p>The most likely explanation is that you're using too many training examples for your SVM implementation.</p>
  
  <p>SVMs are based around a <a href="https://en.wikipedia.org/wiki/Kernel_method" rel="noreferrer">kernel function</a>. Most implementations explicitly store this as an NxN matrix of distances between the training points to avoid computing entries over and over again.</p>
  
  <p>In your case, with 75% of 700,000 examples, this matrix will require approximately 250GB of RAM to store, which is more than you're likely to have in consumer hardware.</p>
  
  <p>If your SVM implementation can avoid caching the values, you might get a speedup that way, or you might not (you'll waste a lot of time recomputing them).</p>
  
  <p>A much better way to deal with this is to just <a href="https://stats.stackexchange.com/questions/122409/why-downsample">not use all</a> of the data, since most of it will be redundant from the SVM's perspective (it only benefits from having more data near the decision boundaries). A good starting place would be to randomly discard 90% of the training data, and see what performance looks like.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-08T08:27:54.757Z
Answers.LastActivityDate: 2018-08-08T12:56:56.550Z
Answers.LastEditDate: 2018-08-08T12:56:56.550Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>I think you should use a linear kernel, 'cause training SVM with a linear kernel is faster than with another kernel, especially for text classification. Good luck </p>
  
  <p><a href="https://www.svm-tutorial.com/2014/10/svm-linear-kernel-good-text-classification/" rel="nofollow noreferrer">https://www.svm-tutorial.com/2014/10/svm-linear-kernel-good-text-classification/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T08:01:41.357Z
Answers.LastActivityDate: 2020-04-21T08:01:41.357Z
Answers.UserDisplayName: Almas Bani Agam
Answers.Body: <p>To quickly train the SVM , you can try to Use Linear SVM or Use scaled data. </p>
  
  <p>sources: <a href="https://www.researchgate.net/publication/2926909_A_Practical_Guide_to_Support_Vector_Classification_Chih-Wei_Hsu_Chih-Chung_Chang_and_Chih-Jen_Lin" rel="nofollow noreferrer">https://www.researchgate.net/publication/2926909_A_Practical_Guide_to_Support_Vector_Classification_Chih-Wei_Hsu_Chih-Chung_Chang_and_Chih-Jen_Lin</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T09:15:55.050Z
Answers.Text: Hi, just  so you know, these links may 404 after a while, so if you can, just copy the important part and quote in your answer.
Answers.UserDisplayName: Anshuman Kumar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T10:23:30.840Z
Answers.LastActivityDate: 2020-04-21T10:23:30.840Z
Answers.UserDisplayName: Shinta Cyntia
Answers.Body: <p>Because you use various combinations of features whose dimensionality is between 1 and 14 features, You might try to use Linear SVM (linear Kernels) would be good for your problem. You could try LIBLINEAR library but the Data should be linearly separable, otherwise test accuracy would be very low.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T09:49:39.713Z
Answers.LastActivityDate: 2020-04-21T09:49:39.713Z
Answers.UserDisplayName: Dafa Prima
Answers.Body: <p>You can speed up the training time by doing several steps:</p>
  
  <ol>
  <li>scale the values of your features</li>
  <li>use only a limited number of features because this will affect the training time; i.e. when you use 14 features, it means your model has 14 dimensions and it makes computation more complex and take much time.</li>
  <li>choose a proper kernel, linear SVM kernel usually give the fastest result</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T10:27:37.920Z
Answers.LastActivityDate: 2020-04-21T12:49:27.637Z
Answers.LastEditDate: 2020-04-21T12:49:27.637Z
Answers.UserDisplayName: BnZ_Fighter
Answers.Body: <p>SVM scales rather badly with the number of training samples - from <span class="math-container">$O(n^2)$</span> to <span class="math-container">$O(n^3)$</span> as told in this answer <a href="https://stackoverflow.com/questions/16585465/training-complexity-of-linear-svm">https://stackoverflow.com/questions/16585465/training-complexity-of-linear-svm</a>.</p>
  <p>The vanilla approach requires inversion of <span class="math-container">$n \times n$</span> matrix, which is <span class="math-container">$O(n^3)$</span> operations in general.</p>
  <p>As suggested in the other answers, the most apparent way to reduce the computational and storage complexity is the reduction of number of training samples.</p>
  <p>I am even surprised that all this data fits into the memory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-26T16:00:40.003Z
Answers.LastActivityDate: 2021-08-26T16:00:40.003Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>The non-linear kernel SVMs can be slow if you have too many training samples. This is due to the fact that the algorithm creates an NxN matrix as @John Doucette answered.</p>
  <p>Now there are a few ways to speed up the non-linear kernel SVMs:</p>
  <ul>
  <li>Use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html?highlight=sgdclassifier#sklearn.linear_model.SGDClassifier" rel="nofollow noreferrer">SGDClassifier</a> instead and provide proper parameters for loss, penalty etc. to make it behave like an SVM. The optimisation process is different than libsvm though.</li>
  <li>Use a kernel approximator like <a href="https://scikit-learn.org/stable/modules/generated/sklearn.kernel_approximation.Nystroem.html#sklearn.kernel_approximation.Nystroem" rel="nofollow noreferrer">Nystroem</a></li>
  <li>Since you are yourself trying out feature combinations, a Linear SVM can also be good and fast :)</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-26T15:34:25.380Z
Answers.LastActivityDate: 2021-08-26T15:34:25.380Z
Answers.UserDisplayName: Soumyajit
Body: <p>I'm trying to create and test non-linear SVMs with various kernels (RBF, Sigmoid, Polynomial) in scikit-learn, to create a model which can classify anomalies and benign behaviors.</p>
  <p>My dataset includes 692703 records and I use a 75/25% training/testing split. Also, I use various combinations of features whose dimensionality is between 1 and 14 features. However, the training processes of the various SVMs take much too long. Is this reasonable?</p>
  <p>I have also examined the ensemble <code>BaggingClassifier</code> in combination with non-linear SVMs, by configuring the <code>n_jobs</code> parameter to <code>-1</code>; nevertheless, the training process proceeds again too slowly.</p>
  <p>How can I speed up the training processes?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-20T02:16:07.633Z
Comments.Text: Can you show some code and timing?  What do you have for computing power?  Have you gone through the following thread?  https://stackoverflow.com/questions/31681373/making-svm-run-faster-in-python
Comments.UserDisplayName: Brian O'Donnell
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-21T08:17:55.673Z
Comments.Text: This is my opinion and from my experience: you can use integers for speeding up your calculation and training data test with your data frame.
Comments.UserDisplayName: Danny Lukmana
Tags: <machine-learning><training><support-vector-machine>
Title: Why does training an SVM take so long? How can I speed it up?
UserDisplayName: Panagiotis
ThreadText: # Why does training an SVM take so long? How can I speed it up?
  
  (Asked by: username_0 on 2018-07-19)
  
  I'm trying to create and test non-linear SVMs with various kernels (RBF, Sigmoid, Polynomial) in scikit-learn, to create a model which can classify anomalies and benign behaviors.
  
  My dataset includes 692703 records and I use a 75/25% training/testing split. Also, I use various combinations of features whose dimensionality is between 1 and 14 features. However, the training processes of the various SVMs take much too long. Is this reasonable?
  
  I have also examined the ensemble `BaggingClassifier` in combination with non-linear SVMs, by configuring the `n_jobs` parameter to `-1`; nevertheless, the training process proceeds again too slowly.
  
  How can I speed up the training processes?
  
  **Tags:** machine-learning, training, support-vector-machine
  
  > **Comments:**
  > **username_1**: Can you show some code and timing?  What do you have for computing power?  Have you gone through the following thread?  https://stackoverflow.com/questions/31681373/making-svm-run-faster-in-python  
  > **username_2**: This is my opinion and from my experience: you can use integers for speeding up your calculation and training data test with your data frame.  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_3 on 2018-08-08. Score: 6
  
  The most likely explanation is that you're using too many training examples for your SVM implementation.
  
  SVMs are based around a [kernel function](https://en.wikipedia.org/wiki/Kernel_method). Most implementations explicitly store this as an NxN matrix of distances between the training points to avoid computing entries over and over again.
  
  In your case, with 75% of 700,000 examples, this matrix will require approximately 250GB of RAM to store, which is more than you're likely to have in consumer hardware.
  
  If your SVM implementation can avoid caching the values, you might get a speedup that way, or you might not (you'll waste a lot of time recomputing them).
  
  A much better way to deal with this is to just [not use all](https://stats.stackexchange.com/questions/122409/why-downsample) of the data, since most of it will be redundant from the SVM's perspective (it only benefits from having more data near the decision boundaries). A good starting place would be to randomly discard 90% of the training data, and see what performance looks like.
  
  
  ---
  
  ### Answer by username_4 on 2020-04-21. Score: 1
  
  I think you should use a linear kernel, 'cause training SVM with a linear kernel is faster than with another kernel, especially for text classification. Good luck
  
  <https://www.svm-tutorial.com/2014/10/svm-linear-kernel-good-text-classification/>
  
  
  ---
  
  ### Answer by username_5 on 2020-04-21. Score: 1
  
  To quickly train the SVM , you can try to Use Linear SVM or Use scaled data.
  
  sources: <https://www.researchgate.net/publication/2926909_A_Practical_Guide_to_Support_Vector_Classification_Chih-Wei_Hsu_Chih-Chung_Chang_and_Chih-Jen_Lin>
  
  > **Comments:**
  > **username_6**: Hi, just  so you know, these links may 404 after a while, so if you can, just copy the important part and quote in your answer.  
  
  ---
  
  ### Answer by username_7 on 2020-04-21. Score: 0
  
  Because you use various combinations of features whose dimensionality is between 1 and 14 features, You might try to use Linear SVM (linear Kernels) would be good for your problem. You could try LIBLINEAR library but the Data should be linearly separable, otherwise test accuracy would be very low.
  
  
  ---
  
  ### Answer by username_8 on 2020-04-21. Score: 0
  
  You can speed up the training time by doing several steps:
  
  1. scale the values of your features
  2. use only a limited number of features because this will affect the training time; i.e. when you use 14 features, it means your model has 14 dimensions and it makes computation more complex and take much time.
  3. choose a proper kernel, linear SVM kernel usually give the fastest result
  
  
  ---
  
  ### Answer by username_9 on 2021-08-26. Score: 0
  
  SVM scales rather badly with the number of training samples - from $O(n^2)$ to $O(n^3)$ as told in this answer <https://stackoverflow.com/questions/16585465/training-complexity-of-linear-svm>.
  
  The vanilla approach requires inversion of $n \times n$ matrix, which is $O(n^3)$ operations in general.
  
  As suggested in the other answers, the most apparent way to reduce the computational and storage complexity is the reduction of number of training samples.
  
  I am even surprised that all this data fits into the memory.
  
  
  ---
  
  ### Answer by username_10 on 2021-08-26. Score: 0
  
  The non-linear kernel SVMs can be slow if you have too many training samples. This is due to the fact that the algorithm creates an NxN matrix as @username_3 answered.
  
  Now there are a few ways to speed up the non-linear kernel SVMs:
  
  * Use the [SGDClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html?highlight=sgdclassifier#sklearn.linear_model.SGDClassifier) instead and provide proper parameters for loss, penalty etc. to make it behave like an SVM. The optimisation process is different than libsvm though.
  * Use a kernel approximator like [Nystroem](https://scikit-learn.org/stable/modules/generated/sklearn.kernel_approximation.Nystroem.html#sklearn.kernel_approximation.Nystroem)
  * Since you are yourself trying out feature combinations, a Linear SVM can also be good and fast :)
  
  
  ---
  
  