Answers.Body: <p>Since your data has lots of repeated patterns, and thus correlations, I suggest you try using <a href="https://en.wikipedia.org/wiki/Singular_spectrum_analysis#Main_algorithm" rel="nofollow noreferrer">Singular spectrum analysis</a>. The core idea is that you re-structure your timeseries of <span class="math-container">$N$</span> points into a matrix of size <span class="math-container">$N - L + 1 \times L$</span>. The first row has (one-indexed) samples from <span class="math-container">$1$</span> to <span class="math-container">$L$</span> from the series, the second has samples <span class="math-container">$2$</span> to <span class="math-container">$L+1$</span> etc. See Wikipedia for an example matrix.</p>
  <p>Then you apply PCA (or SVD) to this, projecting your samples to a lower dimension. Maybe you'll get neat clusters even in 2D, and can apply for example the simple K-nearest neighbors algorithm. The benefit of running SSA first, is to visualize your data, and you can run kNN in a much lower dimensional space.</p>
  <p>And since I'm a big fan of linear methods, an other option is to apply <a href="https://en.wikipedia.org/wiki/Linear_discriminant_analysis" rel="nofollow noreferrer">Linear discriminant analysis</a> to the &quot;time-shifted&quot; matrix. It will learn to project your time-series samples into a lower dimensional space, with maximal margin between classes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-31T11:31:16.797Z
Answers.LastActivityDate: 2024-05-31T11:31:16.797Z
Answers.UserDisplayName: NikoNyrh
Answers.Body: <p>There are a number of possibilities you can consider:</p>
  <h2>simple classification</h2>
  <p>Following what you have suggested as your problem definition, you can train a model for those labels, given a hisotry of some window size (your hyper-parameter). In that case, common timeseries models can be utilized (LSTM, <a href="https://keras.io/examples/timeseries/timeseries_classification_from_scratch/" rel="nofollow noreferrer">TCN</a>, <a href="https://arxiv.org/abs/1603.06995" rel="nofollow noreferrer">MCNN</a> ). I prefer the MCNN method for it is a nice model especially when fixing a proper window-size is a challenge. It probably outperforms LSTM, so chances are you'll have better results. Moreover, LSTM must be improved with Attention mechanism, as you said the main challenge of your problem is long-term dependencies.</p>
  <h2>Kernel-methods classification</h2>
  <p>Since you already know that meaningful patterns exist in your data, you can base your classification on similarity measures. Note that this is still a classification model like the one above, but now instead of raw timepoints, you provide a similarity measure for signals (or <em>shapelets</em> so to speak) such as Dynamic Time Warping (DTW) and do classification using kernel methods. This still requires the window-size as one of the hyper-parameters)</p>
  <h2>Switching models and timeseries segmentation</h2>
  <p>The way you have associated those patterns to a label, I think you can assume that your problem is an example of switching state space models, where you have <span class="math-container">$k$</span> states, each defined by a state-space model, and you switch among those states. you can assume the switching mechanism depends on the data, or the duration of the current state, which makes it interesting for your problem, since you have the challenge of long-term dependencies. switching model can learn the expected dwelling time on states, thus it will form a prior belief, not to switch too often, if it repeatedly sees rare switches. Nice thing about this model is that you can do timeseries segmentation, which marks for you your &quot;class&quot; regions. the learning is a bit tricky though. for that purpose, you can checkout <a href="https://arxiv.org/abs/2110.13878" rel="nofollow noreferrer">this paper</a>.</p>
  <h2>conclusion</h2>
  <p>In timeseries problems, the problem definition is often very important and deciding. Sometimes you aim for an impossible problem to solve, whereas with a change of perspective, you can solve it. As an example, imagine a simple linear trend with an additive noise component. if you define your goal as &quot;predicting the next timepoint&quot; you might not achieve satisfying &quot;regression&quot; results because you're basically trying to learn an independent noise. Whereas you can instead be interested in &quot;if the timeseries will go down or up in the next timepoint&quot; then the expected value your label is simply the slope of the trend, which is way more doable than regression. My point is, maybe you can get better result if you think of your problem as another form of research question such as segmentation, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T12:40:43.980Z
Answers.Text: Hello, thanks. It's been a little while since I was thinking about DTW or Shapelets. If I remember correctly You compute distance matrix between points of the serie your're studying and one or more series of references in a database for example (I my case it would be the subpart that compose my curve like in fig.2). Then you send the result of the calculation to a model (svm, ...) ? I had also investigated image-transformation to turn time serie into an image - very interesting but until now it gave me only average results.
Answers.UserDisplayName: Tiffany L
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T15:42:47.253Z
Answers.Text: yes, a kernel method accepts the pair-wise distance matrix. for a more basic method, you can also do KNN for example, if your sample size allows for that.
  
  your "average" results might be due to lack of information in shapelets. maybe you can do some summary statistics to see how diverse your signals are, or calculate TFIDF on your shapelets. I'd start with the most basic models to check for some trivial results
Answers.UserDisplayName: Alireza
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T15:59:28.893Z
Answers.Text: Hi, I have indeed started by extracted on shapelet statistical and temporal features (haven't try spectral features yet) such as Mode, Skewness, signal_distance, and so on. Then I used features selection to keep only the most relevant one. I then trained a SVM model  on it and achieved a pretty good précsion on test set. But once I try on a stream version with different windows size I end up with the same class prediction (class 2). I'm reading for the switching model its look very interesting.
Answers.UserDisplayName: Tiffany L
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T09:51:53.633Z
Answers.LastActivityDate: 2022-05-10T09:51:53.633Z
Answers.UserDisplayName: Alireza
Body: <p>I would like to submit you a problem with which I struggle.</p>
  <p>Suppose I have this kind of record over time in a dataframe:</p>
  <p><a href="https://i.sstatic.net/vdFFu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vdFFu.png" alt="enter image description here" /></a>
  fig.1</p>
  <p>If we zoom in a bit we see such shape:</p>
  <p><a href="https://i.sstatic.net/eINf0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eINf0.png" alt="enter image description here" /></a>
  fig.2</p>
  <p>We see that the general pattern is a increase with a pick (very high or very thin sometime even flat) follow by an almost flat part with vibration then a decrease, the we go back to zero (almost) for a time (like in the middle of fig.1) or we start an other cycle</p>
  <p>Some have very high peak, some are more flat, some have a more longuer part before to decrease.</p>
  <p>I have 4 classes :</p>
  <ul>
  <li>increase time - 1</li>
  <li>working time  - 2</li>
  <li>decrease time - 3</li>
  <li>rest time (no activity) - 0</li>
  </ul>
  <p>Now assume in my dataframe I have columns that tell to what class belong each point in time.</p>
  <p><a href="https://i.sstatic.net/GsfEf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GsfEf.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/fXiaF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fXiaF.png" alt="enter image description here" /></a></p>
  <p><strong>I would like to build a model that can recognize those 4 class when it see it on stream data . Imagine that our stream data is fig.1 and that we read N points (on a sliding windows) over time. What model could allow me to classify correctly each point or subpart data point in this window according to a certain pattern (hope I'm clear)</strong>
  Regarding the fact that in reallity I could be in rest time for a very long time or in working time a very long time also.
  It may also depend a lot of the sliding window, we for exemple see the beginning of the increase time on the first window then it end on the next window.</p>
  <p>I first try to use LSTM or 1D-CNN, problem I have is that it tend to see this general pattern even when it's not present.</p>
  <p>--- UPDATE : Chillston</p>
  <pre><code>from sklearn.model_selection import train_test_split
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=777, shuffle=False)
  
  # input N x T x D
  X_train.shape, y_train.shape, X_test.shape, y_test.shape,
  
  ((17562, 1000, 1), (17562, 1000, 4), (4391, 1000, 1), (4391, 1000, 4))
  </code></pre>
  <p>I then pass it to 1d-CNN model, I have try many Architecture even resnet-cnn version, here's it's just more classical one.</p>
  <pre><code>def build_res1dcnn(n_classes):
  
      input_shape = (X_train.shape[1], 1)
  
      inputs = Input(shape = input_shape, name = 'input')
  
      # Stage 1
      x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu', 
                         kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                         bias_regularizer = 'l2')(inputs)
      x = BatchNormalization()(x)
      x = MaxPooling1D(pool_size=(2))(x)
      x = Dropout(0.2)(x)
  
      x = Conv1D(128, kernel_size= 5, strides = 2, padding = 'same', activation = 'relu', 
                         kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                         bias_regularizer = 'l2')(inputs)
      x = BatchNormalization()(x)
      x = MaxPooling1D(pool_size=(2))(x)
      x = Dropout(0.2)(x)
  
      x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu', 
                         kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                         bias_regularizer = 'l2')(inputs)
      x = BatchNormalization()(x)
      x = MaxPooling1D(pool_size=(2))(x)
      x = Dropout(0.2)(x)
      
      x = GlobalMaxPooling1D()(x)
      
      # Here I want T x n_class for a T X 1 input sequence
      outputs = []
  
      # https://stackoverflow.com/questions/51397484/appending-tensors-in-keras
      Ty = X_train.shape[1]
      for i in range(Ty):
          out = Dense(n_classes, activation = &quot;softmax&quot;)(x)
          outputs.append(out)
  
      output = Concatenate()(outputs)
      output = Reshape([Ty, n_classes])(output)
      
      
      model = Model(inputs = inputs, outputs=output)
      
      model.compile(optimizer = Adam(learning_rate = 0.1), 
                    loss = 'categorical_crossentropy', 
                    metrics = ['categorical_accuracy'])
  
      
      #model.summary()
      
      return model
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-09T21:47:26.967Z
Comments.Text: As your data is coming in as a stream, I would try a model that, given some number of historic samples (in your case you already have a sliding window $w_{t-n:t}$), classifies only the last sample of that window. That way the model can base the prediction on the $n$ previous samples. And from the looks a 1D-CNN should work fine with that data. Maybe you can provide a bit more info on what you have tried and what exactly didn't work?
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-11T12:47:49.750Z
Comments.Text: @Chillston Hello, sure. I have update my post. First I start by cutting 1000 points in the fig.1 with a step of 500 points (before I was cutting sequence in fig.1 so to have many sequence like those on fig.2) - maybe what cause the model to see this general patttern everywhere). I also get the label associated with those point in "y" variable, then I one-hot encode "y". after that I send it to 1d-cnn that predict (N x T X n_class). I guess the real problem is the model architecture+hyperparam combine with windows size from train and test.
Comments.UserDisplayName: Tiffany L
Tags: <classification><time-series><pattern-recognition>
Title: Pattern recognition for live stream Time serie
UserDisplayName: Tiffany L
ThreadText: # Pattern recognition for live stream Time serie
  
  (Asked by: username_0 on 2022-05-09)
  
  I would like to submit you a problem with which I struggle.
  Suppose I have this kind of record over time in a dataframe:
  
  fig.1
  If we zoom in a bit we see such shape:
  
  fig.2
  We see that the general pattern is a increase with a pick (very high or very thin sometime even flat) follow by an almost flat part with vibration then a decrease, the we go back to zero (almost) for a time (like in the middle of fig.1) or we start an other cycle
  Some have very high peak, some are more flat, some have a more longuer part before to decrease.
  I have 4 classes :
  
  increase time - 1
  working time  - 2
  decrease time - 3
  rest time (no activity) - 0
  
  Now assume in my dataframe I have columns that tell to what class belong each point in time.
  
  
  I would like to build a model that can recognize those 4 class when it see it on stream data . Imagine that our stream data is fig.1 and that we read N points (on a sliding windows) over time. What model could allow me to classify correctly each point or subpart data point in this window according to a certain pattern (hope I'm clear)
  Regarding the fact that in reallity I could be in rest time for a very long time or in working time a very long time also.
  It may also depend a lot of the sliding window, we for exemple see the beginning of the increase time on the first window then it end on the next window.
  I first try to use LSTM or 1D-CNN, problem I have is that it tend to see this general pattern even when it's not present.
  --- UPDATE : Chillston
  from sklearn.model_selection import train_test_split
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=777, shuffle=False)
  
  # input N x T x D
  X_train.shape, y_train.shape, X_test.shape, y_test.shape,
  
  ((17562, 1000, 1), (17562, 1000, 4), (4391, 1000, 1), (4391, 1000, 4))
  
  I then pass it to 1d-CNN model, I have try many Architecture even resnet-cnn version, here's it's just more classical one.
  def build_res1dcnn(n_classes):
  
      input_shape = (X_train.shape[1], 1)
  
      inputs = Input(shape = input_shape, name = 'input')
  
      # Stage 1
      x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu', 
                         kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                         bias_regularizer = 'l2')(inputs)
      x = BatchNormalization()(x)
      x = MaxPooling1D(pool_size=(2))(x)
      x = Dropout(0.2)(x)
  
      x = Conv1D(128, kernel_size= 5, strides = 2, padding = 'same', activation = 'relu', 
                         kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                         bias_regularizer = 'l2')(inputs)
      x = BatchNormalization()(x)
      x = MaxPooling1D(pool_size=(2))(x)
      x = Dropout(0.2)(x)
  
      x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu', 
                         kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                         bias_regularizer = 'l2')(inputs)
      x = BatchNormalization()(x)
      x = MaxPooling1D(pool_size=(2))(x)
      x = Dropout(0.2)(x)
      
      x = GlobalMaxPooling1D()(x)
      
      # Here I want T x n_class for a T X 1 input sequence
      outputs = []
  
      # https://stackoverflow.com/questions/51397484/appending-tensors-in-keras
      Ty = X_train.shape[1]
      for i in range(Ty):
          out = Dense(n_classes, activation = "softmax")(x)
          outputs.append(out)
  
      output = Concatenate()(outputs)
      output = Reshape([Ty, n_classes])(output)
      
      
      model = Model(inputs = inputs, outputs=output)
      
      model.compile(optimizer = Adam(learning_rate = 0.1), 
                    loss = 'categorical_crossentropy', 
                    metrics = ['categorical_accuracy'])
  
      
      #model.summary()
      
      return model
  
  **Tags:** classification, time-series, pattern-recognition
  > **Comments:**
  > **username_1**: As your data is coming in as a stream, I would try a model that, given some number of historic samples (in your case you already have a sliding window $w_{t-n:t}$), classifies only the last sample of that window. That way the model can base the prediction on the $n$ previous samples. And from the looks a 1D-CNN should work fine with that data. Maybe you can provide a bit more info on what you have tried and what exactly didn't work?  
  > **username_0**: @Chillston Hello, sure. I have update my post. First I start by cutting 1000 points in the fig.1 with a step of 500 points (before I was cutting sequence in fig.1 so to have many sequence like those on fig.2) - maybe what cause the model to see this general patttern everywhere). I also get the label associated with those point in "y" variable, then I one-hot encode "y". after that I send it to 1d-cnn that predict (N x T X n_class). I guess the real problem is the model architecture+hyperparam combine with windows size from train and test.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-05-31. Score: 0
  
  Since your data has lots of repeated patterns, and thus correlations, I suggest you try using [Singular spectrum analysis](https://en.wikipedia.org/wiki/Singular_spectrum_analysis#Main_algorithm). The core idea is that you re-structure your timeseries of $N$ points into a matrix of size $N - L + 1 \times L$. The first row has (one-indexed) samples from $1$ to $L$ from the series, the second has samples $2$ to $L+1$ etc. See Wikipedia for an example matrix.
  
  Then you apply PCA (or SVD) to this, projecting your samples to a lower dimension. Maybe you'll get neat clusters even in 2D, and can apply for example the simple K-nearest neighbors algorithm. The benefit of running SSA first, is to visualize your data, and you can run kNN in a much lower dimensional space.
  
  And since I'm a big fan of linear methods, an other option is to apply [Linear discriminant analysis](https://en.wikipedia.org/wiki/Linear_discriminant_analysis) to the "time-shifted" matrix. It will learn to project your time-series samples into a lower dimensional space, with maximal margin between classes.
  
  
  ---
  
  ### Answer by username_3 on 2022-05-10. Score: 0
  
  There are a number of possibilities you can consider:
  
  ## simple classification
  
  Following what you have suggested as your problem definition, you can train a model for those labels, given a hisotry of some window size (your hyper-parameter). In that case, common timeseries models can be utilized (LSTM, [TCN](https://keras.io/examples/timeseries/timeseries_classification_from_scratch/), [MCNN](https://arxiv.org/abs/1603.06995) ). I prefer the MCNN method for it is a nice model especially when fixing a proper window-size is a challenge. It probably outperforms LSTM, so chances are you'll have better results. Moreover, LSTM must be improved with Attention mechanism, as you said the main challenge of your problem is long-term dependencies.
  
  ## Kernel-methods classification
  
  Since you already know that meaningful patterns exist in your data, you can base your classification on similarity measures. Note that this is still a classification model like the one above, but now instead of raw timepoints, you provide a similarity measure for signals (or *shapelets* so to speak) such as Dynamic Time Warping (DTW) and do classification using kernel methods. This still requires the window-size as one of the hyper-parameters)
  
  ## Switching models and timeseries segmentation
  
  The way you have associated those patterns to a label, I think you can assume that your problem is an example of switching state space models, where you have $k$ states, each defined by a state-space model, and you switch among those states. you can assume the switching mechanism depends on the data, or the duration of the current state, which makes it interesting for your problem, since you have the challenge of long-term dependencies. switching model can learn the expected dwelling time on states, thus it will form a prior belief, not to switch too often, if it repeatedly sees rare switches. Nice thing about this model is that you can do timeseries segmentation, which marks for you your "class" regions. the learning is a bit tricky though. for that purpose, you can checkout [this paper](https://arxiv.org/abs/2110.13878).
  
  ## conclusion
  
  In timeseries problems, the problem definition is often very important and deciding. Sometimes you aim for an impossible problem to solve, whereas with a change of perspective, you can solve it. As an example, imagine a simple linear trend with an additive noise component. if you define your goal as "predicting the next timepoint" you might not achieve satisfying "regression" results because you're basically trying to learn an independent noise. Whereas you can instead be interested in "if the timeseries will go down or up in the next timepoint" then the expected value your label is simply the slope of the trend, which is way more doable than regression. My point is, maybe you can get better result if you think of your problem as another form of research question such as segmentation, etc.
  
  > **Comments:**
  > **username_0**: Hello, thanks. It's been a little while since I was thinking about DTW or Shapelets. If I remember correctly You compute distance matrix between points of the serie your're studying and one or more series of references in a database for example (I my case it would be the subpart that compose my curve like in fig.2). Then you send the result of the calculation to a model (svm, ...) ? I had also investigated image-transformation to turn time serie into an image - very interesting but until now it gave me only average results.  
  > **username_3**: yes, a kernel method accepts the pair-wise distance matrix. for a more basic method, you can also do KNN for example, if your sample size allows for that.  your "average" results might be due to lack of information in shapelets. maybe you can do some summary statistics to see how diverse your signals are, or calculate TFIDF on your shapelets. I'd start with the most basic models to check for some trivial results  
  > **username_0**: Hi, I have indeed started by extracted on shapelet statistical and temporal features (haven't try spectral features yet) such as Mode, Skewness, signal_distance, and so on. Then I used features selection to keep only the most relevant one. I then trained a SVM model  on it and achieved a pretty good précsion on test set. But once I try on a stream version with different windows size I end up with the same class prediction (class 2). I'm reading for the switching model its look very interesting.  
  
  ---
  
  