Answers.Body: <p>In the paper "<a href="https://www.sciencedirect.com/science/article/pii/S0925231217309864" rel="nofollow noreferrer">Unsupervised real-time anomaly detection for streaming data</a>" (by Subutai Ahmad, Alexander Lavin, Scott Purdy and Zuha Agha), 2017, an algorithm for anomaly detection (particularly suited for cases where a stream of data is continuously provided) is described. This algorithm is based on Numenta's Hierarchical Temporal Memory model.</p>
  
  <p>I've actually never used it, but I know that Numenta's work is particularly suited for anomaly detection. You can have a look at it and see if it fits your needs. Have also a look at the <a href="https://github.com/numenta/NAB" rel="nofollow noreferrer">Numenta Anomaly Benchmark</a> (NAB).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-24T20:01:49.983Z
Answers.Text: Thank you for the reference. I am new to machine learning related topics. I am a bit confused here about the datasets(logs) I have. Do I need to start by thinking about how to label them (anomalous/normal behavior)? Or can I perform anomaly detection on the unlabeled dataset?
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-24T20:19:06.353Z
Answers.Text: @Ben In this case, I don't think you will need labelled data. Anomaly detection only consists in finding certain patterns in the data (so this is a unsupervised learning technique).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T12:37:02.923Z
Answers.Text: Do you have suggestions for any other unsupervised learning techniques? The Numenta approach cannot be utilized for my work due to license issues.
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T12:38:24.100Z
Answers.Text: @Ben I would not have other suggestions (right now). But why is the license an issue? I think they have an open-source option.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T12:40:51.983Z
Answers.Text: Unfortunately, it is blacklisted for use in my company.
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T12:51:41.203Z
Answers.Text: @Ben I think you should report and discuss this issue at https://discourse.numenta.org/, if you have time. I also found Numenta's approach to licensing a little bit unclear. If you decide to post something on Numenta's forum, maybe link them to this question.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T12:58:48.297Z
Answers.Text: Can I not perform a clustering-based approach to recognize patterns of normal execution behavior from the logs and then identify anomalous behavior that does not adhere to the normal execution pattern? Or is this not an ideal solution to my scenario?
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T13:10:00.317Z
Answers.Text: @Ben I think that a clustering-based approach might help you to find certain groups of data points in your data. But how are you going to define an anomalous data point?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T13:25:50.600Z
Answers.Text: You are right. It is difficult to define an anomalous point. I believe it depends on how clearly we define normal behavior.
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-26T07:36:27.553Z
Answers.Text: There are a lot of ways to define this. Ultimately all machine learning is based upon an underlying set of assumptions and a model. The obvious thing is to fit some parametric model to your data and learn threshold for when things go wrong. Do you have examples of when things become anomalous?
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-24T17:47:18.337Z
Answers.LastActivityDate: 2019-02-24T17:47:18.337Z
Answers.UserDisplayName: nbro
Body: <p>I am developing an AI tool for <strong>anomaly detection in a distributed system</strong>.  The system supports an interface that combines several individual logs into a single log file generating approx. 7000 entries/min. The logs entries are partially system generated (d-Bus, IPC, ….)  and human written statements (Status not received, initialized successfully, ….). The developers use the generated log for debugging. The entries have been configured to have a similar format depending on the generated system (timestamp, ids, component, context, verbosity level, description, ….). </p>
  
  <p><em>Background:</em><br>
  1. The history of the identified anomalies is minimal and not archived.<br>
  2. Not many similar event templates in log files.<br>
  3. Software execution rules are not clearly documented.<br>
  4. The log events are co-related.  </p>
  
  <p>What are the recommended algorithms (Statistical, NLP, ML, Neural networks) that can be used to efficiently perform pattern extraction on the entries and identify existing and new anomalous behavior?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-24T16:28:33.370Z
Comments.Text: Are your categories only "anomalous" and "not anomalous" or you want to perform anomaly detection and categorisation (as two different tasks)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-24T17:28:25.627Z
Comments.Text: The main goal is to perform anomaly detection. By categorization, I meant the extraction of features from the log events relevant to the identification of the anomalous behavior.
Comments.UserDisplayName: Ben
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-24T17:29:57.573Z
Comments.Text: Is the data going to be continuously provided (that is, will you keep receiving a stream of data) or is the data contained in a set which will not change?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-24T17:34:13.933Z
Comments.Text: I am looking into the stream of data already present(offline). But in the future, it is desired to extend the method to perform on a stream of incoming data (online).
Comments.UserDisplayName: Ben
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-07T00:12:29.330Z
Comments.Text: What percentage of user log are expected attack?
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><machine-learning><natural-language-processing><pattern-recognition>
Title: Anomaly Detection in distributed system using generated log file
UserDisplayName: Ben
ThreadText: # Anomaly Detection in distributed system using generated log file
  
  (Asked by: username_0 on 2019-02-24)
  
  I am developing an AI tool for **anomaly detection in a distributed system**.  The system supports an interface that combines several individual logs into a single log file generating approx. 7000 entries/min. The logs entries are partially system generated (d-Bus, IPC, ….)  and human written statements (Status not received, initialized successfully, ….). The developers use the generated log for debugging. The entries have been configured to have a similar format depending on the generated system (timestamp, ids, component, context, verbosity level, description, ….).
  
  *Background:*  
  1. The history of the identified anomalies is minimal and not archived.  
  2. Not many similar event templates in log files.  
  3. Software execution rules are not clearly documented.  
  4. The log events are co-related.
  
  What are the recommended algorithms (Statistical, NLP, ML, Neural networks) that can be used to efficiently perform pattern extraction on the entries and identify existing and new anomalous behavior?
  
  **Tags:** neural-networks, machine-learning, natural-language-processing, pattern-recognition
  
  > **Comments:**
  > **username_1**: Are your categories only "anomalous" and "not anomalous" or you want to perform anomaly detection and categorisation (as two different tasks)?  
  > **username_0**: The main goal is to perform anomaly detection. By categorization, I meant the extraction of features from the log events relevant to the identification of the anomalous behavior.  
  > **username_1**: Is the data going to be continuously provided (that is, will you keep receiving a stream of data) or is the data contained in a set which will not change?  
  > **username_0**: I am looking into the stream of data already present(offline). But in the future, it is desired to extend the method to perform on a stream of incoming data (online).  
  > **username_2**: What percentage of user log are expected attack?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-24. Score: 0
  
  In the paper "[Unsupervised real-time anomaly detection for streaming data](https://www.sciencedirect.com/science/article/pii/S0925231217309864)" (by Subutai Ahmad, Alexander Lavin, Scott Purdy and Zuha Agha), 2017, an algorithm for anomaly detection (particularly suited for cases where a stream of data is continuously provided) is described. This algorithm is based on Numenta's Hierarchical Temporal Memory model.
  
  I've actually never used it, but I know that Numenta's work is particularly suited for anomaly detection. You can have a look at it and see if it fits your needs. Have also a look at the [Numenta Anomaly Benchmark](https://github.com/numenta/NAB) (NAB).
  
  > **Comments:**
  > **username_0**: Thank you for the reference. I am new to machine learning related topics. I am a bit confused here about the datasets(logs) I have. Do I need to start by thinking about how to label them (anomalous/normal behavior)? Or can I perform anomaly detection on the unlabeled dataset?  
  > **username_1**: @username_0 In this case, I don't think you will need labelled data. Anomaly detection only consists in finding certain patterns in the data (so this is a unsupervised learning technique).  
  > **username_0**: Do you have suggestions for any other unsupervised learning techniques? The Numenta approach cannot be utilized for my work due to license issues.  
  > **username_1**: @username_0 I would not have other suggestions (right now). But why is the license an issue? I think they have an open-source option.  
  > **username_0**: Unfortunately, it is blacklisted for use in my company.  
  > **username_1**: @username_0 I think you should report and discuss this issue at https://discourse.numenta.org/, if you have time. I also found Numenta's approach to licensing a little bit unclear. If you decide to post something on Numenta's forum, maybe link them to this question.  
  > **username_0**: Can I not perform a clustering-based approach to recognize patterns of normal execution behavior from the logs and then identify anomalous behavior that does not adhere to the normal execution pattern? Or is this not an ideal solution to my scenario?  
  > **username_1**: @username_0 I think that a clustering-based approach might help you to find certain groups of data points in your data. But how are you going to define an anomalous data point?  
  > **username_0**: You are right. It is difficult to define an anomalous point. I believe it depends on how clearly we define normal behavior.  
  > **username_3**: There are a lot of ways to define this. Ultimately all machine learning is based upon an underlying set of assumptions and a model. The obvious thing is to fit some parametric model to your data and learn threshold for when things go wrong. Do you have examples of when things become anomalous?  
  
  ---
  
  