Answers.Body: <p>I don't know lipids, but if 1/3 of your data set comprises duplication thats a skewed distribution with a low mean and high variance (of the duplications). It is likely to follow a negative binomial distribution. You need to plot a histogram of your data and you will find that 0 (no duplicates) is massively represented, 1 duplicate is next in frequency, 2 duplicates are lower frequency ... and on until a very small proportion of the data comprises a large number of duplicates. This is a very common distribution throughout biology.</p>
  <p>The advice is:</p>
  <ul>
  <li>Do not remove the duplicates because they are a vital part of the distribution, i.e. biological information.</li>
  <li>Construct a histogram of each data set.</li>
  <li>Don't take &quot;an average&quot;, i.e. don't take the mean of skewed data, thats very important if it has a high variance.</li>
  </ul>
  <p><strong>Easy solution</strong> The easiest way to deal with this data is take a median and construct upper and lower quartiles around the median (IQR), especially the frequency (i.e. the distribution duplicates). This will give an approximate comparison between data sets. This alone might work and certainly gives you &quot;cut-offs&quot;.</p>
  <p>If it is a negative binomial distribution then testing the significance between two negative binomial distributions beyond the IQR is an advanced area of statistics and a separate question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T08:10:15.497Z
Answers.LastActivityDate: 2023-07-09T15:36:21.173Z
Answers.LastEditDate: 2023-07-09T15:36:21.173Z
Answers.UserDisplayName: M__
Body: <p>I have a MS-lipidomics dataset - from human samples split into a treatment group (n = 2) and control (n = 2) - which contains several detected targets that matched to the same lipid. I don't have access to any of the raw data, only the Tier 1 and Tier 2-matched lipid names and ID numbers.</p>
  <p>There are around 300 duplicates out of a total of 1000 detected markers, so I'm reluctant to exclude them entirely, but I couldn't find any descriptions of best-practices for what to do in this situation. I'm considering testing the correlation between duplicates and removing the ones lie below some threshold (e.g: 0.5), then averaging the remaining duplicates - but I'm not sure if this would be a valid way of approaching this.</p>
  <p>Are there any good ways of addressing these duplicate values?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-08T07:47:47.143Z
Comments.Text: Sorry how is the threshold derived thats quite key here?
Comments.UserDisplayName: M__
Tags: <proteomics><multi-omics>
Title: Handling duplicates in untargeted lipidomics
ThreadText: # Handling duplicates in untargeted lipidomics
  
  (Asked by: username_0 on 2023-07-01)
  
  I have a MS-lipidomics dataset - from human samples split into a treatment group (n = 2) and control (n = 2) - which contains several detected targets that matched to the same lipid. I don't have access to any of the raw data, only the Tier 1 and Tier 2-matched lipid names and ID numbers.
  
  There are around 300 duplicates out of a total of 1000 detected markers, so I'm reluctant to exclude them entirely, but I couldn't find any descriptions of best-practices for what to do in this situation. I'm considering testing the correlation between duplicates and removing the ones lie below some threshold (e.g: 0.5), then averaging the remaining duplicates - but I'm not sure if this would be a valid way of approaching this.
  
  Are there any good ways of addressing these duplicate values?
  
  **Tags:** proteomics, multi-omics
  
  > **Comments:**
  > **username_1**: Sorry how is the threshold derived thats quite key here?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-08. Score: 0
  
  I don't know lipids, but if 1/3 of your data set comprises duplication thats a skewed distribution with a low mean and high variance (of the duplications). It is likely to follow a negative binomial distribution. You need to plot a histogram of your data and you will find that 0 (no duplicates) is massively represented, 1 duplicate is next in frequency, 2 duplicates are lower frequency ... and on until a very small proportion of the data comprises a large number of duplicates. This is a very common distribution throughout biology.
  
  The advice is:
  
  * Do not remove the duplicates because they are a vital part of the distribution, i.e. biological information.
  * Construct a histogram of each data set.
  * Don't take "an average", i.e. don't take the mean of skewed data, thats very important if it has a high variance.
  
  **Easy solution** The easiest way to deal with this data is take a median and construct upper and lower quartiles around the median (IQR), especially the frequency (i.e. the distribution duplicates). This will give an approximate comparison between data sets. This alone might work and certainly gives you "cut-offs".
  
  If it is a negative binomial distribution then testing the significance between two negative binomial distributions beyond the IQR is an advanced area of statistics and a separate question.
  
  
  ---
  
  