Answers.Body: <p>(This comment got too long)</p>
  <blockquote>
  <p>positions in a predicted protein chain</p>
  </blockquote>
  <p>You do mean <em>position</em> in SS sequence, not <em>coordinates</em> in tertiary, yes?
  From the sounds of it you are talking about SS —not tertiary— and want the analysis to be independent of tertiary.</p>
  <p>Therefore, you are comparing for each protein two 1D arrays of sequential elements containing between 3 and 6 states, one for each residue. Technically, you could add another state, a chain break, and have two colossally long 1D arrays. This very much becomes a <strong>statistics</strong> classification topic.</p>
  <p>The properties of these arrays are:</p>
  <ul>
  <li>The values are sequential.</li>
  <li>Neighbouring elements influence each other as a span of residues makes a SS element. Thus a correct classification will propagate. This will mean that all metrics will be hugely inflated.</li>
  <li>Assumption: the divergence is likely on the ends of a span. This could be tested and plotted!</li>
  </ul>
  <p>The simplest thing would be collapse the states into a binary loop vs non loop and classify each element of the two arrays and do a <strong>typical confusion matrix</strong> and its metrics (see <a href="https://en.wikipedia.org/wiki/Confusion_matrix" rel="nofollow noreferrer">Wikipedia</a>). This would have the issue that neighbours influence each other.</p>
  <p>For a figure, given N states, one could simply plot a <em>N</em> × <em>N</em> heatmap of the true states vs. predicted states. This would show a hot diagonal of the correct allocations, but will show which states are more prone to misallocation. The frequency of the states may need normalising... but this would still be affected by the length of the SS spans. Therefore, a more sophisticated <em>model</em> may be needed for better normalisation.</p>
  <p>For more advanced classifications, more advanced algorithms would be needed, but it still remains a stats problem. It might be worth asking the question again under that discussion framework or look up a relevant multi-state classifier.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T09:41:33.870Z
Answers.Text: I already prepared confusion matrices. I was looking for something graphical like chart or histogram, etc..
Answers.UserDisplayName: user366312
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T09:50:05.660Z
Answers.Text: If so, given _N_ states, wouldn't a simple _N_&times;_N_ heatmap of the true states vs. predicted states suffice? This would show a hot diagonal of the correct allocations, but will show which states are more prone to misallocation.
Answers.UserDisplayName: Matteo Ferla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T09:52:00.917Z
Answers.Text: Please, add your last comment to your answer.
Answers.UserDisplayName: user366312
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T10:10:43.073Z
Answers.Text: As said, the above is a comment, advising to rephrase or repost the whole Q as a statistical problem as there's a lot of stats know-how here, way way more than compbiochem...
Answers.UserDisplayName: Matteo Ferla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T10:45:18.597Z
Answers.Text: One problem still remains: the proteins are not the same in length. So, how can I align them in the heatmap?
Answers.UserDisplayName: user366312
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T10:55:46.950Z
Answers.Text: The size of the suggested heatmap is N x N, where N is the number of states (alpha-Helix, Loop, shEet, Pi-helix, Turn etc.). The tally would be on the concatentaed SS sequence as you did for the confusion matrix, but without collapsing the states to binary.
Answers.UserDisplayName: Matteo Ferla
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T09:38:20.567Z
Answers.LastActivityDate: 2022-03-07T10:20:06.323Z
Answers.LastEditDate: 2022-03-07T10:20:06.323Z
Answers.UserDisplayName: Matteo Ferla
Body: <p>I have 6000 predicted protein structures.</p>
  <p>I am comparing secondary structures predicted by an algorithm with the correct secondary structure assignments.</p>
  <p>I need to show which positions in a predicted protein chain have the most differences as to the true protein chain.</p>
  <p>How can I do that?</p>
  <p>How to represent a summary of disagreement between two secondary structure predictors given predictions for many proteins?</p>
  <p>I already prepared confusion matrices. I was looking for something graphical like chart or histogram, etc..</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T22:40:04.883Z
Comments.Text: Are you asking:
  a. How to represent the disagreement between two secondary structure predictors on one protein
  or
  b. How to represent a summary of disagreement between two secondary structure predictors given predictions for many proteins
  or
  c. If there is any common way and simple tool for representing a or b? (kind of yes for a, not to my knowledge for b)
Comments.UserDisplayName: O.Laprevote
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T05:43:14.700Z
Comments.Text: @O.Laprevote, I am asking (b).
Comments.UserDisplayName: user366312
Tags: <protein-structure>
Title: How to represent a summary of disagreement between two secondary structure predictors given predictions for many proteins?
ThreadText: # How to represent a summary of disagreement between two secondary structure predictors given predictions for many proteins?
  
  (Asked by: username_0 on 2022-03-06)
  
  I have 6000 predicted protein structures.
  
  I am comparing secondary structures predicted by an algorithm with the correct secondary structure assignments.
  
  I need to show which positions in a predicted protein chain have the most differences as to the true protein chain.
  
  How can I do that?
  
  How to represent a summary of disagreement between two secondary structure predictors given predictions for many proteins?
  
  I already prepared confusion matrices. I was looking for something graphical like chart or histogram, etc..
  
  **Tags:** protein-structure
  
  > **Comments:**
  > **username_1**: Are you asking: a. How to represent the disagreement between two secondary structure predictors on one protein or b. How to represent a summary of disagreement between two secondary structure predictors given predictions for many proteins or c. If there is any common way and simple tool for representing a or b? (kind of yes for a, not to my knowledge for b)  
  > **username_0**: @username_1, I am asking (b).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-07. Score: 3 | Accepted answer
  
  (This comment got too long)
  
  > positions in a predicted protein chain
  
  You do mean *position* in SS sequence, not *coordinates* in tertiary, yes?
  From the sounds of it you are talking about SS —not tertiary— and want the analysis to be independent of tertiary.
  
  Therefore, you are comparing for each protein two 1D arrays of sequential elements containing between 3 and 6 states, one for each residue. Technically, you could add another state, a chain break, and have two colossally long 1D arrays. This very much becomes a **statistics** classification topic.
  
  The properties of these arrays are:
  
  * The values are sequential.
  * Neighbouring elements influence each other as a span of residues makes a SS element. Thus a correct classification will propagate. This will mean that all metrics will be hugely inflated.
  * Assumption: the divergence is likely on the ends of a span. This could be tested and plotted!
  
  The simplest thing would be collapse the states into a binary loop vs non loop and classify each element of the two arrays and do a **typical confusion matrix** and its metrics (see [Wikipedia](https://en.wikipedia.org/wiki/Confusion_matrix)). This would have the issue that neighbours influence each other.
  
  For a figure, given N states, one could simply plot a *N* × *N* heatmap of the true states vs. predicted states. This would show a hot diagonal of the correct allocations, but will show which states are more prone to misallocation. The frequency of the states may need normalising... but this would still be affected by the length of the SS spans. Therefore, a more sophisticated *model* may be needed for better normalisation.
  
  For more advanced classifications, more advanced algorithms would be needed, but it still remains a stats problem. It might be worth asking the question again under that discussion framework or look up a relevant multi-state classifier.
  
  > **Comments:**
  > **username_0**: I already prepared confusion matrices. I was looking for something graphical like chart or histogram, etc..  
  > **username_2**: If so, given _N_ states, wouldn't a simple _N_&times;_N_ heatmap of the true states vs. predicted states suffice? This would show a hot diagonal of the correct allocations, but will show which states are more prone to misallocation.  
  > **username_0**: Please, add your last comment to your answer.  
  > **username_2**: As said, the above is a comment, advising to rephrase or repost the whole Q as a statistical problem as there's a lot of stats know-how here, way way more than compbiochem...  
  > **username_0**: One problem still remains: the proteins are not the same in length. So, how can I align them in the heatmap?  
  > **username_2**: The size of the suggested heatmap is N x N, where N is the number of states (alpha-Helix, Loop, shEet, Pi-helix, Turn etc.). The tally would be on the concatentaed SS sequence as you did for the confusion matrix, but without collapsing the states to binary.  
  
  ---
  
  