Body: <p>I am working on a novel protein secondary structure detection algorithm using machine learning.</p>
  <p>As far as I understand, online protein databanks (e.g., RCSB, etc.) have a huge collection of proteins (several millions). All proteins are not useful for research purposes as they contain redundant, duplicate, or incomplete information.</p>
  <p>So, we have to choose the minimum number of proteins so that we have enough proteins to cover all the required information for our specific purpose.</p>
  <p>This is called &quot;protein culling.&quot;</p>
  <p>Google search shows that there were some servers like PISCES, ProDaMa, etc. However, as usual, all of them are down.</p>
  <p>Now the question is, how do I do protein-culling myself in the context of protein secondary structure detection?</p>
  <p><strong>N.B.</strong> Proteins with similar folds or secondary structure elements can be considered redundant, even if their sequence identity is low.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T02:39:44.770Z
Comments.Text: How good are your data engineering skills? Removing duplicate, redundant or incomplete stuff isn't hard and a database of several million is honestly pretty small. You can probably do this in pandas2 definitely polars, but after that it gets more specialised.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T02:42:26.790Z
Comments.Text: @M__, I work on secondary and tertiary structures. I need a minimum set of individual PDB files.
Comments.UserDisplayName: user366312
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T02:43:33.687Z
Comments.Text: @M__, PIESCES server had 5022 PDB files.
Comments.UserDisplayName: user366312
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T02:44:40.587Z
Comments.Text: It's still applicable to data engineering for the criteria you are considering. I agree that it may not fit inside pandas2. If you were considering genetic distance ... thats more complicated.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T11:11:06.820Z
Comments.Text: This depends entirely on what, exactly, you are trying to do. Please [edit] your question and specify the biological question you are trying to answer.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T15:44:40.850Z
Comments.Text: @terdon, I am working on a novel protein secondary and tertiary structure detection algorithm using machine learning.
Comments.UserDisplayName: user366312
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T15:48:11.763Z
Comments.Text: OK, please add that to your question but, again, what do you consider not useful here? Presumably, you don't care so much about sequence identity as about tertiary structure similarity, right? So what would you like to cull and what would you like to keep?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T15:50:28.427Z
Comments.Text: @terdon, Proteins with similar folds or secondary or tertiary structure elements can be considered redundant, even if their sequence identity is low.
Comments.UserDisplayName: user366312
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T15:54:34.217Z
Comments.Text: Have you considered looking at this in terms of protein domains instead of proteins? Say starting with the Pfam data and getting representative examples of each domain? (random shot in the dark here, I have no experience with tertiary structure analysis)
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-13T15:55:18.547Z
Comments.Text: @terdon, I am a beginner at level zero in Bioinformatics.
Comments.UserDisplayName: user366312
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-14T11:19:50.687Z
Comments.Text: If you were willing to download the data yourself, you could use a tool like [FoldSeek](https://github.com/steineggerlab/foldseek) to cluster on various thresholds to remove redundancy. Though I realise this is tertiary structure, rather than secondary structure, but it still could work with some tweaking.
Comments.UserDisplayName: dthorbur
Tags: <proteins>
Title: How do I do "protein culling"?
ThreadText: # How do I do "protein culling"?
  
  (Asked by: username_0 on 2024-03-13)
  
  I am working on a novel protein secondary structure detection algorithm using machine learning.
  
  As far as I understand, online protein databanks (e.g., RCSB, etc.) have a huge collection of proteins (several millions). All proteins are not useful for research purposes as they contain redundant, duplicate, or incomplete information.
  
  So, we have to choose the minimum number of proteins so that we have enough proteins to cover all the required information for our specific purpose.
  
  This is called "protein culling."
  
  Google search shows that there were some servers like PISCES, ProDaMa, etc. However, as usual, all of them are down.
  
  Now the question is, how do I do protein-culling myself in the context of protein secondary structure detection?
  
  **N.B.** Proteins with similar folds or secondary structure elements can be considered redundant, even if their sequence identity is low.
  
  **Tags:** proteins
  
  > **Comments:**
  > **username_1**: How good are your data engineering skills? Removing duplicate, redundant or incomplete stuff isn't hard and a database of several million is honestly pretty small. You can probably do this in pandas2 definitely polars, but after that it gets more specialised.  
  > **username_0**: @username_1, I work on secondary and tertiary structures. I need a minimum set of individual PDB files.  
  > **username_0**: @username_1, PIESCES server had 5022 PDB files.  
  > **username_1**: It's still applicable to data engineering for the criteria you are considering. I agree that it may not fit inside pandas2. If you were considering genetic distance ... thats more complicated.  
  > **username_2**: This depends entirely on what, exactly, you are trying to do. Please [edit] your question and specify the biological question you are trying to answer.  
  > **username_0**: @username_2, I am working on a novel protein secondary and tertiary structure detection algorithm using machine learning.  
  > **username_2**: OK, please add that to your question but, again, what do you consider not useful here? Presumably, you don't care so much about sequence identity as about tertiary structure similarity, right? So what would you like to cull and what would you like to keep?  
  > **username_0**: @username_2, Proteins with similar folds or secondary or tertiary structure elements can be considered redundant, even if their sequence identity is low.  
  > **username_2**: Have you considered looking at this in terms of protein domains instead of proteins? Say starting with the Pfam data and getting representative examples of each domain? (random shot in the dark here, I have no experience with tertiary structure analysis)  
  > **username_0**: @username_2, I am a beginner at level zero in Bioinformatics.  
  > **username_3**: If you were willing to download the data yourself, you could use a tool like [FoldSeek](https://github.com/steineggerlab/foldseek) to cluster on various thresholds to remove redundancy. Though I realise this is tertiary structure, rather than secondary structure, but it still could work with some tweaking.  
  
  ---
  
  ## 0 answers
  
  