Answers.Body: <p>First of all, you mention that you have <strong>categorical</strong> data. I don't see <strong>how you can define similarity</strong> so that you can also define the distance between the predicted value and the ground truth (error). You can do that <strong>only if</strong> the data are <strong>ordinal</strong>.</p>
  
  <p>If you want to just classify between <strong>normal</strong> and <strong>anomalous</strong> points (<strong>binary classification</strong>), without caring about further classification of the anomaly types themselves, one of the most common algorithms is the <a href="http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/" rel="nofollow noreferrer">One-Class Support Vector Machine (OC-SVM)</a>.</p>
  
  <p>Anomalies are unpredictable in nature and sometimes hard to replicate and record. Therefore, there is usually lack of anomalous data and supervised learning approaches suffer because if you sacrifice some "precious" anomalous points to train the algorithm, you cannot use them to test it.</p>
  
  <p>The main advantage of OC-SVM is that it is semi-supervised learning, meaning that you train it only with normal data and then it can detect samples that deviate from the trained behaviour during testing and classify them as anomalous. Thus, you "save" all the rare anomalous points for testing purposes!</p>
  
  <p>Take a look at this short <a href="http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html" rel="nofollow noreferrer">Python example</a>, it has all you need :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-28T17:43:16.083Z
Answers.LastActivityDate: 2018-06-28T17:43:16.083Z
Answers.UserDisplayName: pcko1
Body: <p>I have a data-set with <span class="math-container">$m$</span> observations and <span class="math-container">$p$</span> categorical variables (nominal), each variable <span class="math-container">$X_1, X_2,\dots, X_p$</span> has several different possible values. </p>
  
  <p>Ultimately, I am looking for a way to find anomalies i.e. to identify rows for which the combination of values seems incorrect with respect to the data I saw so far. </p>
  
  <p>So far, I was thinking about building a model to predict the value for each column and then build some metric to evaluate how different the actual row is from the predicted row. </p>
  
  <p>I would greatly appreciate any help!</p>
  
Tags: <machine-learning><datasets><anomaly-detection>
Title: Find anomalies from records of categorical data
UserDisplayName: bat
ThreadText: # Find anomalies from records of categorical data
  
  (Asked by: username_0 on 2018-02-27)
  
  I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X\_1, X\_2,\dots, X\_p$ has several different possible values.
  
  Ultimately, I am looking for a way to find anomalies i.e. to identify rows for which the combination of values seems incorrect with respect to the data I saw so far.
  
  So far, I was thinking about building a model to predict the value for each column and then build some metric to evaluate how different the actual row is from the predicted row.
  
  I would greatly appreciate any help!
  
  **Tags:** machine-learning, datasets, anomaly-detection
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-06-28. Score: 2
  
  First of all, you mention that you have **categorical** data. I don't see **how you can define similarity** so that you can also define the distance between the predicted value and the ground truth (error). You can do that **only if** the data are **ordinal**.
  
  If you want to just classify between **normal** and **anomalous** points (**binary classification**), without caring about further classification of the anomaly types themselves, one of the most common algorithms is the [One-Class Support Vector Machine (OC-SVM)](http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/).
  
  Anomalies are unpredictable in nature and sometimes hard to replicate and record. Therefore, there is usually lack of anomalous data and supervised learning approaches suffer because if you sacrifice some "precious" anomalous points to train the algorithm, you cannot use them to test it.
  
  The main advantage of OC-SVM is that it is semi-supervised learning, meaning that you train it only with normal data and then it can detect samples that deviate from the trained behaviour during testing and classify them as anomalous. Thus, you "save" all the rare anomalous points for testing purposes!
  
  Take a look at this short [Python example](http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html), it has all you need :)
  
  
  ---
  
  