Body: <p>A little bit of context. I have a classification algorithm based on mathematical discriminator and I am not applying any machine learning or AI technique, just moving window and several relative comparisons. This is a <strong>signal segmentation</strong> problem, where <strong>I need to detect three states based on their mathematical properties or morphology</strong>. In addition, I need to perform this with <strong>minimum delay and almost real-time</strong>.</p>
  <p>The output of my algorithm is a labelled signal with the different sections of it labelleded depending on those mathematical descriptors. The reference model was labelled by professionals that tagged the data for me and I am in charge of creating an automatic labelling tool for it.</p>
  <p>Here you can see on the top the reference model and on the bottom the output of my algorithm.</p>
  <p><a href="https://i.sstatic.net/Rt3F8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Rt3F8.png" alt="enter image description here" /></a></p>
  <p>My question is, I have defined a measure of success function that compares sample by sample the two signals but it is not very accurate and I want to know what is the state of the art in this type of classification problem.</p>
  <p>My &quot;measure of success&quot; function:</p>
  <pre><code>def measure_of_success(signal_size, result_labels, model_labels):
  &quot;&quot;&quot; Measure of success function &quot;&quot;&quot;
  incorrect_matching = 0
  for index in range(signal_size):
      expected_section = identify_section_from_sample(model_labels, index)
      result_section = identify_section_from_sample(result_labels, index)
      if expected_section != result_section:
          incorrect_matching += 1
  matching = (1 - (incorrect_matching / signal_size)) * 100
  return matching
  </code></pre>
  <p>This function performs a sample-by-sample comparison. My problem is: the labels from the upper picture were set by hand but, I does not matter how accurate the section is determined but just to know that the event has been detected for a considerable time. This is a bit confusing but basically, if I detect the same &quot;b&quot; event and it coincides in time with the majority of the reference model, it should be 100% ok. On the other hand, I feel this measure of success is a bit simplistic and there might be better ways like a confusion matrix or something like that.</p>
  <p>What would you suggest me to do in order to evaluate the success of the classification?</p>
  <p>Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-30T11:18:37.223Z
Comments.Text: Welcome to AI stack exchange. I suggest you share your "measure of success function" since this is key to understanding how you are evaluating. If you want to compare to state-of-the-art, you will need to give more information about the problem domain and what you already know about state-of-the-art algorithms (this looks like a signal sequence, so I would guess some kind of RNN or CNN might do well - if it is a sequence please make clear whether prediction needs to happen at the moment of sampling for each record or can be done retrospectively). Use [edit] to add details to the question
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-31T10:20:30.857Z
Comments.Text: I have added some information in the post! Thanks for pointing that out
Comments.UserDisplayName: GGChe
Tags: <classification><regression><evaluation-functions>
Title: How can evaluate the success of my algoritm?
UserDisplayName: GGChe
ThreadText: # How can evaluate the success of my algoritm?
  
  (Asked by: username_0 on 2022-07-30)
  
  A little bit of context. I have a classification algorithm based on mathematical discriminator and I am not applying any machine learning or AI technique, just moving window and several relative comparisons. This is a signal segmentation problem, where I need to detect three states based on their mathematical properties or morphology. In addition, I need to perform this with minimum delay and almost real-time.
  The output of my algorithm is a labelled signal with the different sections of it labelleded depending on those mathematical descriptors. The reference model was labelled by professionals that tagged the data for me and I am in charge of creating an automatic labelling tool for it.
  Here you can see on the top the reference model and on the bottom the output of my algorithm.
  
  My question is, I have defined a measure of success function that compares sample by sample the two signals but it is not very accurate and I want to know what is the state of the art in this type of classification problem.
  My "measure of success" function:
  def measure_of_success(signal_size, result_labels, model_labels):
  """ Measure of success function """
  incorrect_matching = 0
  for index in range(signal_size):
      expected_section = identify_section_from_sample(model_labels, index)
      result_section = identify_section_from_sample(result_labels, index)
      if expected_section != result_section:
          incorrect_matching += 1
  matching = (1 - (incorrect_matching / signal_size)) * 100
  return matching
  
  This function performs a sample-by-sample comparison. My problem is: the labels from the upper picture were set by hand but, I does not matter how accurate the section is determined but just to know that the event has been detected for a considerable time. This is a bit confusing but basically, if I detect the same "b" event and it coincides in time with the majority of the reference model, it should be 100% ok. On the other hand, I feel this measure of success is a bit simplistic and there might be better ways like a confusion matrix or something like that.
  What would you suggest me to do in order to evaluate the success of the classification?
  Thanks!
  
  **Tags:** classification, regression, evaluation-functions
  > **Comments:**
  > **username_1**: Welcome to AI stack exchange. I suggest you share your "measure of success function" since this is key to understanding how you are evaluating. If you want to compare to state-of-the-art, you will need to give more information about the problem domain and what you already know about state-of-the-art algorithms (this looks like a signal sequence, so I would guess some kind of RNN or CNN might do well - if it is a sequence please make clear whether prediction needs to happen at the moment of sampling for each record or can be done retrospectively). Use [edit] to add details to the question  
  > **username_0**: I have added some information in the post! Thanks for pointing that out  
  
  ---
  
  ## 0 answers
  
  