Answers.Body: <p>Without knowing the kind of data and the process generating it it's hard to give a definite answer.
  In general, I would attempt a network that has as inputs the actual sensor readings, and outputs the expected readings. You train this network by presenting data with errors added as inputs, and correct readings as outputs.  It should learn to guess the correct (non-anomalous) readings from a set of actual readings, and you can find the sensors with anomalies by taking the difference between actual and predicted readings.
  Depending on the kind of anomalies you expect (misreadings such as fluke zero or max values, or actually anomalous states of the system being measured) your training data should be set up to have samples of such anomalies.
  If there are temporal correlations (for example, temperature readings that change slowly) using a RNN might be helpful but its dimensioning heavily depends on the nature of your measured system.
  External factors influencing the system (state of a heating mechanism, time of day/year, etc.) could be added at inputs for better predictions.
  At the end of the day, trial and error is your friend. Start with a simple network and see how well it behaves. Go for more complex networks when you see where the limitations of the simple solution are.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-08T10:12:23.283Z
Answers.Text: Actually the sensors have different idle values at different points.So an anomaly for one may be normal for other and we don't have data for anomalies at each point (at some certain points we have).
Answers.UserDisplayName: Fasty
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-08T10:20:41.417Z
Answers.Text: And can you just elaborate the point.Do we have both anomalies and non-anomalies in training? and what I understood is that we have the labels as 'non' anomalous values' ryt? And if the difference between predicted readings and normal  readings are high then as per your explanation is it anomaly or not anomaly?
Answers.UserDisplayName: Fasty
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-08T10:50:36.410Z
Answers.Text: It depends on what sort of output is more useful to you. If you simply want to identify anomalous values, then the network should be trained to output binary per-sensor labels. But if you actually want to "remove" the anomaly from the values then the network should be used for regression. I suspect the network for the former use-case (binary classification with simple anomalous/non-anomalous labels) will be easier to train, but this is just a shot in the dark given that I don't know the kind of data you're dealing with.
Answers.UserDisplayName: Ali250
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-08T07:38:42.023Z
Answers.LastActivityDate: 2019-03-08T07:38:42.023Z
Answers.UserDisplayName: Hans-Martin Mosner
Answers.Body: <p>Instead of using a neural network, simply sample as many non-anomalous readings from each sensor as you can. If the distribution of the readings from each sensor is approximately normal (check the <a href="https://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm" rel="nofollow noreferrer">skew and kurtosis</a> values for the samples from each sensor) then you can work out mean and standard deviation of the samples and, for any future samples, the value of a particular measurement on a normal probability distribution.</p>
  
  <p>(If your measurements do not have a normal distribution, then you can often apply a transform of some sort to the data make it normal).</p>
  
  <p>So, let's say that you have measured a few thousand typical samples for one of your sensors, confirmed that the distribution is normal and calculated the mean and standard deviation of those samples. You can now calculate where on the normal curve your new sample 'x' would be using:</p>
  
  <pre><code>def gaussian(x, mean, std):
      # check for very small standard deviations
      if(2 * std ** 2 == 0):
          return 0.0
  
      return (1.0/(math.sqrt(2*math.pi) * std)) * math.exp(-((x-mean)**2)/(2*std**2))
  </code></pre>
  
  <p>At this point, you can make a decision if the value of the of a particular measurement on that distribution is so low as to be anomalous, and inform the user.</p>
  
  <p>Now, you have probably realised that working out exactly <em>how</em> low a value should be to be considered anomalous might be tricky, and you are correct. The solution is to see if you can get some <em>real</em> anomalous data and use that to set the thresholds. Obviously, this might be rather a job if you have 4000 independent sensors...</p>
  
  <p>If you want to know more about Anomaly Detection then I recommend that you take a look at <a href="https://www.youtube.com/playlist?list=PLwgXNx7TiGV6UH3aEzmdZwzFRwvEnRb0N" rel="nofollow noreferrer">Andrew Ng's introductory lecture series.</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-08T10:56:08.900Z
Answers.LastActivityDate: 2019-03-12T16:20:50.353Z
Answers.LastEditDate: 2019-03-12T16:20:50.353Z
Answers.UserDisplayName: DrMcCleod
Body: <p>I have series of sensors (around 4k) and each sensor will measure the amplitudes at each point.Suppose I train the neural network with sufficent set of 4k values (N * 4k shape). The machine will find a pattern in the series of values.If the values stray away from the pattern (that is anomaly)  it can detect the point and will be able to say that anomaly is in the 'X'th  sensor.Is this possible.If so what kind of neural network should I use?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-07T18:20:32.363Z
Comments.Text: Questions. .. Are the sensors correlated in some way or all completely independent? Are they all required, or can you use a subset?
Comments.UserDisplayName: DrMcCleod
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-08T07:43:15.970Z
Comments.Text: Also, do you HAVE to use a neural network? There are easier ways to achieve this.
Comments.UserDisplayName: DrMcCleod
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-08T09:28:02.547Z
Comments.Text: Neural Network is not mandatory, any algorithm would do.As of now the sensors seem to be independent.
Comments.UserDisplayName: Fasty
Tags: <neural-networks><machine-learning><deep-learning><pattern-recognition>
Title: Finding anomaly detection by pattern matching in a set of continous data
UserDisplayName: Fasty
ThreadText: # Finding anomaly detection by pattern matching in a set of continous data
  
  (Asked by: username_0 on 2019-03-07)
  
  I have series of sensors (around 4k) and each sensor will measure the amplitudes at each point.Suppose I train the neural network with sufficent set of 4k values (N \* 4k shape). The machine will find a pattern in the series of values.If the values stray away from the pattern (that is anomaly) it can detect the point and will be able to say that anomaly is in the 'X'th sensor.Is this possible.If so what kind of neural network should I use?
  
  **Tags:** neural-networks, machine-learning, deep-learning, pattern-recognition
  
  > **Comments:**
  > **username_1**: Questions. .. Are the sensors correlated in some way or all completely independent? Are they all required, or can you use a subset?  
  > **username_1**: Also, do you HAVE to use a neural network? There are easier ways to achieve this.  
  > **username_0**: Neural Network is not mandatory, any algorithm would do.As of now the sensors seem to be independent.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-03-08. Score: 2
  
  Without knowing the kind of data and the process generating it it's hard to give a definite answer.
  In general, I would attempt a network that has as inputs the actual sensor readings, and outputs the expected readings. You train this network by presenting data with errors added as inputs, and correct readings as outputs. It should learn to guess the correct (non-anomalous) readings from a set of actual readings, and you can find the sensors with anomalies by taking the difference between actual and predicted readings.
  Depending on the kind of anomalies you expect (misreadings such as fluke zero or max values, or actually anomalous states of the system being measured) your training data should be set up to have samples of such anomalies.
  If there are temporal correlations (for example, temperature readings that change slowly) using a RNN might be helpful but its dimensioning heavily depends on the nature of your measured system.
  External factors influencing the system (state of a heating mechanism, time of day/year, etc.) could be added at inputs for better predictions.
  At the end of the day, trial and error is your friend. Start with a simple network and see how well it behaves. Go for more complex networks when you see where the limitations of the simple solution are.
  
  > **Comments:**
  > **username_0**: Actually the sensors have different idle values at different points.So an anomaly for one may be normal for other and we don't have data for anomalies at each point (at some certain points we have).  
  > **username_0**: And can you just elaborate the point.Do we have both anomalies and non-anomalies in training? and what I understood is that we have the labels as 'non' anomalous values' ryt? And if the difference between predicted readings and normal  readings are high then as per your explanation is it anomaly or not anomaly?  
  > **username_3**: It depends on what sort of output is more useful to you. If you simply want to identify anomalous values, then the network should be trained to output binary per-sensor labels. But if you actually want to "remove" the anomaly from the values then the network should be used for regression. I suspect the network for the former use-case (binary classification with simple anomalous/non-anomalous labels) will be easier to train, but this is just a shot in the dark given that I don't know the kind of data you're dealing with.  
  
  ---
  
  ### Answer by username_1 on 2019-03-08. Score: 2
  
  Instead of using a neural network, simply sample as many non-anomalous readings from each sensor as you can. If the distribution of the readings from each sensor is approximately normal (check the [skew and kurtosis](https://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm) values for the samples from each sensor) then you can work out mean and standard deviation of the samples and, for any future samples, the value of a particular measurement on a normal probability distribution.
  
  (If your measurements do not have a normal distribution, then you can often apply a transform of some sort to the data make it normal).
  
  So, let's say that you have measured a few thousand typical samples for one of your sensors, confirmed that the distribution is normal and calculated the mean and standard deviation of those samples. You can now calculate where on the normal curve your new sample 'x' would be using:
  
  ```
  def gaussian(x, mean, std):
      # check for very small standard deviations
      if(2 * std ** 2 == 0):
          return 0.0
  
      return (1.0/(math.sqrt(2*math.pi) * std)) * math.exp(-((x-mean)**2)/(2*std**2))
  
  ```
  
  At this point, you can make a decision if the value of the of a particular measurement on that distribution is so low as to be anomalous, and inform the user.
  
  Now, you have probably realised that working out exactly *how* low a value should be to be considered anomalous might be tricky, and you are correct. The solution is to see if you can get some *real* anomalous data and use that to set the thresholds. Obviously, this might be rather a job if you have 4000 independent sensors...
  
  If you want to know more about Anomaly Detection then I recommend that you take a look at [Andrew Ng's introductory lecture series.](https://www.youtube.com/playlist?list=PLwgXNx7TiGV6UH3aEzmdZwzFRwvEnRb0N)
  
  
  ---
  
  