Answers.Body: <p>So if I understood correctly:
  You have data from 2 sensors in time: Ar flow and BackGas Flow (SCCM, what is that?)
  You have that data for multiple products.</p>
  <p>1 - Since it is relatively low dimensional, you may try using raw data with K-Means or Self Organizing Maps.</p>
  <p>2 - If you searching for anomalies in time, you might try using feature engineering with things like flow change:</p>
  <ul>
  <li>i) Take 2 points in time and &quot;derivate&quot; (Point A - Point B)/(Time A - Time B)</li>
  <li>ii) Take the result for A and repeat the process as a second derivative, try that for many levels of &quot;derivative&quot;</li>
  <li>iii) Machine learning application for sensor failure detection in polymerization process. In: Simpósio Brasileiro de Telecomunicações e Processamento de Sinais.</li>
  </ul>
  <p>3 - You might want to check other Time-Series related research, even on regression and classification, since they may give you ideas on relevant features or approaches:</p>
  <ul>
  <li>i) An enseble approach to time dependent classification (10.1109/ICMLA.2018.00164)</li>
  <li>ii) Multivariate Time Series for Data-driven Endpoint Prediction in the Blast Oxygen Furnace (DOI: 10.1109/ICMLA.2018.00231)</li>
  </ul>
  <p>4 - Note that anomalies are outliers, so analysing data distant from the mean of any features you might engineer from the time. An model based on RBF kernels might pick up some information.</p>
  <p>5 - Since you don't know the amount of clusters you can try hierarchical clustering.</p>
  <p>6 - Don't forget to talk to people in your field about what to expect from this.</p>
  <p>If you can elaborate on the type of anomalies you're looking for I may come with better ideas. This is pretty general tips.</p>
  <p>I may have misunderstood what you wanted to cluster:</p>
  <p>7 - If you want to detect anomalies in:</p>
  <ul>
  <li>i) <strong>products behavior</strong> tips above were mostly for it and you might use all 23 features in short time intervals and cluster like this:</li>
  </ul>
  <pre><code>    For each product in dataset {
  
        samples = {} # empty set
  
        For each time interval in product {
  
          features in interval add to samples
  
        }
  
        perform clustering method using samples
  
      }
  </code></pre>
  <p>This algorithm would help cluster periods of time where the product behavied weirdly</p>
  <ul>
  <li><p>ii) <strong>sensors behavior</strong> then for something like that you might try tip 8</p>
  </li>
  <li><p>iii) <strong>products itself</strong> create a feature vector with all avaiable information for each product, then cluster that. Since it is composed of variable time interval and 23 features per product (plus time) you might want to use a bit of dimensionality reduction (PCA should work fine, since it gives you the direction of greater variance and you're look for that). Using a regressor to model products behavior and cluster that could all be usefull (create a regressor in time for each product to model its behavior, then cluster the weights of the regressors as the product representation)</p>
  </li>
  </ul>
  <p>8 - Let's say you want to identify sensors behavior, it might be helpful to model the sensors output as a function <strong>f</strong>(time, extravariables) and do regression over it (try linear, then try non-linear). Points in time that the regressor had bad performance predicting the answer might indicate that that behavior is anomalous</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T08:52:24.337Z
Answers.Text: Hey, Pedro. Thank you for your valuable tips. I shall try the things suggested by you, but I've put here only 2 parameters (ArFlow and BacksGas). In my dataset, I've about 23 different parameters. So do you think KMeans can still be applied for every single parameter separately? And sccm- is Standard Cubic Centimeter Flow
Answers.UserDisplayName: some_programmer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T11:01:03.120Z
Answers.Text: I'll add 2 more itens to the answers
Answers.UserDisplayName: Pedro Henrique Monforte
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T11:28:08.483Z
Answers.Text: if anyone can help me to make this algorithm readable I would thank hahaha
Answers.UserDisplayName: Pedro Henrique Monforte
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T11:31:02.377Z
Answers.Text: important to notice that tip 7 will find timeintervals where a product presented anomalous behavior to itself, if you want to detect it on other products that should have the same effect create a set using similar products to cluster
Answers.UserDisplayName: Pedro Henrique Monforte
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T02:15:13.447Z
Answers.LastActivityDate: 2020-09-18T07:26:47.107Z
Answers.LastEditDate: 2020-09-18T07:26:47.107Z
Answers.UserDisplayName: Pedro Henrique Monforte
Body: <p>I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines.</p>
  <p>My dataset looks like this:</p>
  <pre><code>ContextID   Time_ms Ar_Flow_sccm    BacksGas_Flow_sccm
  7289973 09:12:48.502    49.56054688 1.953125
  7289973 09:12:48.603    49.56054688 2.05078125
  7289973 09:12:48.934    99.85351563 2.05078125
  7289973 09:12:49.924    351.3183594 2.05078125
  7289973 09:12:50.924    382.8125    1.953125
  7289973 09:12:51.924    382.8125    1.7578125
  7289973 09:12:52.934    382.8125    1.7578125
  7289999 09:15:36.434    50.04882813 1.7578125
  7289999 09:15:36.654    50.04882813 1.7578125
  7289999 09:15:36.820    50.04882813 1.66015625
  7289999 09:15:37.904    333.2519531 1.85546875
  7289999 09:15:38.924    377.1972656 1.953125
  7289999 09:15:39.994    377.1972656 1.7578125
  7289999 09:15:41.94     388.671875  1.85546875
  7289999 09:15:42.136    388.671875  1.85546875
  7290025 09:18:00.429    381.5917969 1.85546875
  7290025 09:18:01.448    381.5917969 1.85546875
  7290025 09:18:02.488    381.5917969 1.953125
  7290025 09:18:03.549    381.5917969 14.453125
  7290025 09:18:04.589    381.5917969 46.77734375
  </code></pre>
  <p>What I have to do is to apply some unsupervised learning technique on each and every parameter column individually and find any anomalies that might exist in there. The <code>ContextID</code> is more like a product number.</p>
  <p>I would like to know which unsupervised learning techniques can be used for this kind of task at hand since the problem is a bit unique:</p>
  <ol>
  <li>It has temporal values.</li>
  <li>Since it has temporal values, each product will have many (similar or different) values as can be seen in the dataset above.</li>
  </ol>
  
Tags: <neural-networks><machine-learning><unsupervised-learning><time-series><anomaly-detection>
Title: Which unsupervised learning technique can be used for anomaly detection in a time series?
UserDisplayName: some_programmer
ThreadText: # Which unsupervised learning technique can be used for anomaly detection in a time series?
  
  (Asked by: username_0 on 2019-03-22)
  
  I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines.
  My dataset looks like this:
  ContextID   Time_ms Ar_Flow_sccm    BacksGas_Flow_sccm
  7289973 09:12:48.502    49.56054688 1.953125
  7289973 09:12:48.603    49.56054688 2.05078125
  7289973 09:12:48.934    99.85351563 2.05078125
  7289973 09:12:49.924    351.3183594 2.05078125
  7289973 09:12:50.924    382.8125    1.953125
  7289973 09:12:51.924    382.8125    1.7578125
  7289973 09:12:52.934    382.8125    1.7578125
  7289999 09:15:36.434    50.04882813 1.7578125
  7289999 09:15:36.654    50.04882813 1.7578125
  7289999 09:15:36.820    50.04882813 1.66015625
  7289999 09:15:37.904    333.2519531 1.85546875
  7289999 09:15:38.924    377.1972656 1.953125
  7289999 09:15:39.994    377.1972656 1.7578125
  7289999 09:15:41.94     388.671875  1.85546875
  7289999 09:15:42.136    388.671875  1.85546875
  7290025 09:18:00.429    381.5917969 1.85546875
  7290025 09:18:01.448    381.5917969 1.85546875
  7290025 09:18:02.488    381.5917969 1.953125
  7290025 09:18:03.549    381.5917969 14.453125
  7290025 09:18:04.589    381.5917969 46.77734375
  
  What I have to do is to apply some unsupervised learning technique on each and every parameter column individually and find any anomalies that might exist in there. The ContextID is more like a product number.
  I would like to know which unsupervised learning techniques can be used for this kind of task at hand since the problem is a bit unique:
  
  It has temporal values.
  Since it has temporal values, each product will have many (similar or different) values as can be seen in the dataset above.
  
  **Tags:** neural-networks, machine-learning, unsupervised-learning, time-series, anomaly-detection
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-23. Score: 4 | Accepted answer
  
  So if I understood correctly:
  You have data from 2 sensors in time: Ar flow and BackGas Flow (SCCM, what is that?)
  You have that data for multiple products.
  1 - Since it is relatively low dimensional, you may try using raw data with K-Means or Self Organizing Maps.
  2 - If you searching for anomalies in time, you might try using feature engineering with things like flow change:
  
  i) Take 2 points in time and "derivate" (Point A - Point B)/(Time A - Time B)
  ii) Take the result for A and repeat the process as a second derivative, try that for many levels of "derivative"
  iii) Machine learning application for sensor failure detection in polymerization process. In: Simpósio Brasileiro de Telecomunicações e Processamento de Sinais.
  
  3 - You might want to check other Time-Series related research, even on regression and classification, since they may give you ideas on relevant features or approaches:
  
  i) An enseble approach to time dependent classification (10.1109/ICMLA.2018.00164)
  ii) Multivariate Time Series for Data-driven Endpoint Prediction in the Blast Oxygen Furnace (DOI: 10.1109/ICMLA.2018.00231)
  
  4 - Note that anomalies are outliers, so analysing data distant from the mean of any features you might engineer from the time. An model based on RBF kernels might pick up some information.
  5 - Since you don't know the amount of clusters you can try hierarchical clustering.
  6 - Don't forget to talk to people in your field about what to expect from this.
  If you can elaborate on the type of anomalies you're looking for I may come with better ideas. This is pretty general tips.
  I may have misunderstood what you wanted to cluster:
  7 - If you want to detect anomalies in:
  
  i) products behavior tips above were mostly for it and you might use all 23 features in short time intervals and cluster like this:
  
      For each product in dataset {
  
        samples = {} # empty set
  
        For each time interval in product {
  
          features in interval add to samples
  
        }
  
        perform clustering method using samples
  
      }
  
  This algorithm would help cluster periods of time where the product behavied weirdly
  
  ii) sensors behavior then for something like that you might try tip 8
  
  iii) products itself create a feature vector with all avaiable information for each product, then cluster that. Since it is composed of variable time interval and 23 features per product (plus time) you might want to use a bit of dimensionality reduction (PCA should work fine, since it gives you the direction of greater variance and you're look for that). Using a regressor to model products behavior and cluster that could all be usefull (create a regressor in time for each product to model its behavior, then cluster the weights of the regressors as the product representation)
  
  
  8 - Let's say you want to identify sensors behavior, it might be helpful to model the sensors output as a function f(time, extravariables) and do regression over it (try linear, then try non-linear). Points in time that the regressor had bad performance predicting the answer might indicate that that behavior is anomalous
  
  > **Comments:**
  > **username_0**: Hey, Pedro. Thank you for your valuable tips. I shall try the things suggested by you, but I've put here only 2 parameters (ArFlow and BacksGas). In my dataset, I've about 23 different parameters. So do you think KMeans can still be applied for every single parameter separately? And sccm- is Standard Cubic Centimeter Flow  
  > **username_1**: I'll add 2 more itens to the answers  
  > **username_1**: if anyone can help me to make this algorithm readable I would thank hahaha  
  > **username_1**: important to notice that tip 7 will find timeintervals where a product presented anomalous behavior to itself, if you want to detect it on other products that should have the same effect create a set using similar products to cluster  
  
  ---
  
  