Answers.Body: <p>Thresholding based on variance is one method for anomaly detection in time-series data. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:44:24.093Z
Answers.LastActivityDate: 2020-04-27T15:44:24.093Z
Answers.UserDisplayName: Brian Spiering
Body: <p>I have data that is collected from several different instruments simultaneously that is generally analyzed on a location-by-location basis.  A skilled interpreter can identify "markers" in the data that represent a certain change in conditions with depth - each marker only occurs once in each series of data.  However, it is possible that a maker is absent either due to missing data or that physical condition not existing.</p>
  
  <p>Often, there are dozens of these markers per location and thousands, if not 10's of thousands of measurements that need to be interpreted.  The task is not that difficult and there are many strong priors that can be used to guide interpretation.  E.g., if marker A is in location #1, and location #2 is very close to location #1, it is likely that marker A will be present in a very similar relative position.  Also, if you have markers A, B, and C they will always be in that order.  Although it could be that you have A/B/C or A/C or B/C, etc.</p>
  
  <p>I am including a hand-sketched example below with 4 example locations and one data stream (I normally have 4-5 data streams per location. </p>
  
  <p>I am looking for guidance on the type of algorithm to apply to this problem.  I have explored Dynamic Time warping, but the issue is that with 10-20k data samples per location, and thousands of locations, the problem becomes computationally challenging.</p>
  
  <p>Also, in general you may have 10000 locations, with maybe 100 that have been hand interpreted by an expert.</p>
  
  <p><a href="https://i.sstatic.net/OskMQ.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OskMQ.jpg" alt="hand sketch of example data"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-26T17:00:17.943Z
Comments.Text: Where do the markers come from? Are they in the data, or do you need to spot them? I can't quite make sense of what it is you're trying to achieve.
Comments.UserDisplayName: Oliver Mason
Tags: <pattern-recognition><time-series>
Title: What method to identify markers in data series via machine learning
UserDisplayName: user1563247
ThreadText: # What method to identify markers in data series via machine learning
  
  (Asked by: username_0 on 2020-04-19)
  
  I have data that is collected from several different instruments simultaneously that is generally analyzed on a location-by-location basis. A skilled interpreter can identify "markers" in the data that represent a certain change in conditions with depth - each marker only occurs once in each series of data. However, it is possible that a maker is absent either due to missing data or that physical condition not existing.
  
  Often, there are dozens of these markers per location and thousands, if not 10's of thousands of measurements that need to be interpreted. The task is not that difficult and there are many strong priors that can be used to guide interpretation. E.g., if marker A is in location #1, and location #2 is very close to location #1, it is likely that marker A will be present in a very similar relative position. Also, if you have markers A, B, and C they will always be in that order. Although it could be that you have A/B/C or A/C or B/C, etc.
  
  I am including a hand-sketched example below with 4 example locations and one data stream (I normally have 4-5 data streams per location.
  
  I am looking for guidance on the type of algorithm to apply to this problem. I have explored Dynamic Time warping, but the issue is that with 10-20k data samples per location, and thousands of locations, the problem becomes computationally challenging.
  
  Also, in general you may have 10000 locations, with maybe 100 that have been hand interpreted by an expert.
  
  [![hand sketch of example data](https://i.sstatic.net/OskMQ.jpg)](https://i.sstatic.net/OskMQ.jpg)
  
  **Tags:** pattern-recognition, time-series
  
  > **Comments:**
  > **username_1**: Where do the markers come from? Are they in the data, or do you need to spot them? I can't quite make sense of what it is you're trying to achieve.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-27. Score: 0
  
  Thresholding based on variance is one method for anomaly detection in time-series data.
  
  
  ---
  
  