Body: <p>A little bit of context. I have a classification algorithm based on mathematical discriminator and I am not applying any machine learning or AI technique, just moving window and several relative comparisons. This is a <strong>signal segmentation</strong> problem, where <strong>I need to detect three states based on their mathematical properties or morphology</strong>. In addition, I need to perform this with <strong>minimum delay and almost real-time</strong>.</p>
  <p>The output of my algorithm is a labelled signal with the different sections of it labelleded depending on those mathematical descriptors. The reference model was labelled by professionals that tagged the data for me and I am in charge of creating an automatic labelling tool for it.</p>
  <p>Here you can see on the top the reference model and on the bottom the output of my algorithm.</p>
  <p><a href="https://i.sstatic.net/Rt3F8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Rt3F8.png" alt="enter image description here" /></a></p>
  <p>My question is, I have defined a measure of success function that compares sample by sample the two signals but it is not very accurate and I want to know what is the state of the art in this type of classification problem.</p>
  <p>My &quot;measure of success&quot; function:</p>
  <pre><code>def measure_of_success(signal_size, result_labels, model_labels):
  &quot;&quot;&quot; Measure of success function &quot;&quot;&quot;
  incorrect_matching = 0
  for index in range(signal_size):
      expected_section = identify_section_from_sample(model_labels, index)
      result_section = identify_section_from_sample(result_labels, index)
      if expected_section != result_section:
          incorrect_matching += 1
  matching = (1 - (incorrect_matching / signal_size)) * 100
  return matching
  </code></pre>
  <p>This function performs a sample-by-sample comparison. My problem is: the labels from the upper picture were set by hand but, I does not matter how accurate the section is determined but just to know that the event has been detected for a considerable time. This is a bit confusing but basically, if I detect the same &quot;b&quot; event and it coincides in time with the majority of the reference model, it should be 100% ok. On the other hand, I feel this measure of success is a bit simplistic and there might be better ways like a confusion matrix or something like that.</p>
  <p>What would you suggest me to do in order to evaluate the success of the classification?</p>
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-30T11:18:37.223Z
Comments.Text: Welcome to AI stack exchange. I suggest you share your "measure of success function" since this is key to understanding how you are evaluating. If you want to compare to state-of-the-art, you will need to give more information about the problem domain and what you already know about state-of-the-art algorithms (this looks like a signal sequence, so I would guess some kind of RNN or CNN might do well - if it is a sequence please make clear whether prediction needs to happen at the moment of sampling for each record or can be done retrospectively). Use [edit] to add details to the question
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-31T10:20:30.857Z
Comments.Text: I have added some information in the post! Thanks for pointing that out
Comments.UserDisplayName: GGChe
Tags: <classification><regression><evaluation-functions>
Title: How can evaluate the success of my algoritm?
UserDisplayName: GGChe
ThreadText: # How can evaluate the success of my algoritm?
  
  (Asked by: username_0 on 2022-07-30)
  
  A little bit of context. I have a classification algorithm based on mathematical discriminator and I am not applying any machine learning or AI technique, just moving window and several relative comparisons. This is a **signal segmentation** problem, where **I need to detect three states based on their mathematical properties or morphology**. In addition, I need to perform this with **minimum delay and almost real-time**.
  
  The output of my algorithm is a labelled signal with the different sections of it labelleded depending on those mathematical descriptors. The reference model was labelled by professionals that tagged the data for me and I am in charge of creating an automatic labelling tool for it.
  
  Here you can see on the top the reference model and on the bottom the output of my algorithm.
  
  [![enter image description here](https://i.sstatic.net/Rt3F8.png)](https://i.sstatic.net/Rt3F8.png)
  
  My question is, I have defined a measure of success function that compares sample by sample the two signals but it is not very accurate and I want to know what is the state of the art in this type of classification problem.
  
  My "measure of success" function:
  
  ```
  def measure_of_success(signal_size, result_labels, model_labels):
  """ Measure of success function """
  incorrect_matching = 0
  for index in range(signal_size):
      expected_section = identify_section_from_sample(model_labels, index)
      result_section = identify_section_from_sample(result_labels, index)
      if expected_section != result_section:
          incorrect_matching += 1
  matching = (1 - (incorrect_matching / signal_size)) * 100
  return matching
  
  ```
  
  This function performs a sample-by-sample comparison. My problem is: the labels from the upper picture were set by hand but, I does not matter how accurate the section is determined but just to know that the event has been detected for a considerable time. This is a bit confusing but basically, if I detect the same "b" event and it coincides in time with the majority of the reference model, it should be 100% ok. On the other hand, I feel this measure of success is a bit simplistic and there might be better ways like a confusion matrix or something like that.
  
  What would you suggest me to do in order to evaluate the success of the classification?
  
  Thanks!
  
  **Tags:** classification, regression, evaluation-functions
  
  > **Comments:**
  > **username_1**: Welcome to AI stack exchange. I suggest you share your "measure of success function" since this is key to understanding how you are evaluating. If you want to compare to state-of-the-art, you will need to give more information about the problem domain and what you already know about state-of-the-art algorithms (this looks like a signal sequence, so I would guess some kind of RNN or CNN might do well - if it is a sequence please make clear whether prediction needs to happen at the moment of sampling for each record or can be done retrospectively). Use [edit] to add details to the question  
  > **username_0**: I have added some information in the post! Thanks for pointing that out  
  
  ---
  
  ## 0 answers
  
  