Answers.Body: <p>Okay I see. Basically there's a threshold for removing low intensity signal that can't be distinguished from background. That threshold seems tight at 1% of the mean of the normal distributio. Only the signal that is highly significantly above background signal is retained, i.e. subject to further analysis.</p>
  <p>The way it works is that for a given CpG site  there is a) methylated allele probe and b) unmethylated allele probe. The beta-value is a representation of those two intensities. What they want to do is filter out the background signal to avoid false positives.</p>
  <p>Here's the manual <a href="https://bioconductor.org/packages/devel/bioc/vignettes/minfi/inst/doc/minfi.html#5_Quality_control" rel="nofollow noreferrer">https://bioconductor.org/packages/devel/bioc/vignettes/minfi/inst/doc/minfi.html#5_Quality_control</a></p>
  <p>Here's a basic code (I stress the word <em>basic</em>) <strong>HOWEVER</strong> see the note on Noob in minfi below:</p>
  <pre><code>library(minfi)
  
  myStuff &lt;- read.metharray.exp(&quot;/Users/username/Desktop/myfile&quot;)
  rgSet &lt;- preprocessQuantile(myStuff)
  
  M &lt;- getM(rgSet)
  U &lt;- getU(rgSet)
  
  beta &lt;- M / (M + U)
  
  filterProbesByZTest &lt;- function(M, U, p.threshold = 0.01) {
    n &lt;- ncol(M)
    filteredProbes &lt;- rep(TRUE, ncol(M)) 
    
    for (i in 1:ncol(M)) {
      probeM &lt;- M[, i]
      probeU &lt;- U[, i]
      
      meanM &lt;- mean(probeM, na.rm = TRUE)
      meanU &lt;- mean(probeU, na.rm = TRUE)
      sdM &lt;- sd(probeM, na.rm = TRUE)
      sdU &lt;- sd(probeU, na.rm = TRUE)
      
      z_stat &lt;- (meanM - meanU) / sqrt((sdM^2 / length(probeM)) + (sdU^2 / length(probeU)))
      p_value &lt;- 2 * pnorm(-abs(z_stat))
      
      if (p_value &gt; p.threshold) {
        filteredProbes[i] &lt;- FALSE
      }
    }
    
    return(filteredProbes)
  }
  
  filteredProbes &lt;- filterProbesByZTest(M, U, p.threshold = 0.01)
  betaFiltered &lt;- beta[, filteredProbes]
  </code></pre>
  <hr />
  <p>Note apart from being <em>basic</em>. the code isn't an exact replication of the authors method because they used Noob</p>
  <blockquote>
  <p>ll data processing was performed in R (version 3.6.1; R Foundation for Statistical Computing, Vienna, Austria). Using the minfi library [26], raw IDAT files were imported; Noob normalization was performed [27]; and beta values were computed. Beta values whose corresponding intensities on probe-level failed a z-test with a P value threshold of 0.01 against background signal were masked.</p>
  </blockquote>
  <p><code>preprocessNoob</code> by <a href="https://doi.org/10.1093/nar/gkt090" rel="nofollow noreferrer">Triche et al (2013)</a></p>
  <p>My suspicion is that <code>preprocessNoob</code> comprises the above function. I'd need to read the paper in detail.</p>
  <p><code>preprocessQuantile</code> was used above and was part of the original <a href="https://doi.org/10.1093/bioinformatics/btu049" rel="nofollow noreferrer">minfi package</a></p>
  <hr />
  <p><strong>Note</strong> A key minfi paper describing the implementation is:
  <a href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0503-2" rel="nofollow noreferrer">https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0503-2</a></p>
  <p>The most advanced method I'm aware of is <a href="https://clinicalepigeneticsjournal.biomedcentral.com/articles/10.1186/s13148-019-0615-3" rel="nofollow noreferrer">Heiss and Just (2019)</a>
  There guys have developed a different tool set <strong><a href="https://github.com/hhhh5/ewastools" rel="nofollow noreferrer">ewastools</a></strong>  ... if its compatible.</p>
  <p>Getting involved what is a periphery topic for me at the cutting edge of the field is something I'd rather avoid, admittedly I'm heavily into cancer stuff.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-14T11:54:02.480Z
Answers.Text: @Munk Try the above but read up on Noob.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-13T19:59:08.420Z
Answers.LastActivityDate: 2024-12-14T12:30:49.947Z
Answers.LastEditDate: 2024-12-14T12:30:49.947Z
Answers.UserDisplayName: M__
Body: <p>I'm trying to replicate this paper (<a href="https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.5845" rel="nofollow noreferrer">https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.5845</a>) and I don't understand this bit from the &quot;Data Preprocessing&quot; section on page 4:</p>
  <blockquote>
  <p>Beta values whose corre-sponding intensities on probe-level failed a z-test with
  a P value threshold of 0.01 against background signal
  were masked.</p>
  </blockquote>
  <p>Can someone please explain what this means? I'm using R, mainly minfi.</p>
  <p>Paper details:</p>
  <blockquote>
  <p>Title: Machine learning models predict the primary sites of head and neck squamous cell carcinoma metastases based on DNA methylation</p>
  </blockquote>
  <blockquote>
  <p>Details: It takes idat files from a few geo numbers for preprocessing and then models them. I have retrieved the idats from GEO used in the paper, fed them into minfi. My question is with the preprocessing while this data is in minfi.</p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-13T17:31:35.660Z
Comments.Text: Please [edit] your question and give us more context. Ideally, make it so we only need the paper if we are particularly curious. If we need to read the paper to answer your question, that makes it far more unlikely that people will have the time to do so. So include enough for us to get the context. Also clarify exactly what part of this is confusing you. What is it you do not understand?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-14T07:15:56.633Z
Comments.Text: @terdon added paper description to my best abilities. Forgive for I'm still a beginner
Comments.UserDisplayName: Munk
Tags: <r><bioconductor><machine-learning><methylation>
Title: Problem while replicating methylation predicting paper, looking for explanation
ThreadText: # Problem while replicating methylation predicting paper, looking for explanation
  
  (Asked by: username_0 on 2024-12-13)
  
  I'm trying to replicate this paper (<https://pathsocjournals.onlinelibrary.wiley.com/doi/10.1002/path.5845>) and I don't understand this bit from the "Data Preprocessing" section on page 4:
  
  > Beta values whose corre-sponding intensities on probe-level failed a z-test with
  > a P value threshold of 0.01 against background signal
  > were masked.
  
  Can someone please explain what this means? I'm using R, mainly minfi.
  
  Paper details:
  
  > Title: Machine learning models predict the primary sites of head and neck squamous cell carcinoma metastases based on DNA methylation
  
  > Details: It takes idat files from a few geo numbers for preprocessing and then models them. I have retrieved the idats from GEO used in the paper, fed them into minfi. My question is with the preprocessing while this data is in minfi.
  
  **Tags:** r, bioconductor, machine-learning, methylation
  
  > **Comments:**
  > **username_1**: Please [edit] your question and give us more context. Ideally, make it so we only need the paper if we are particularly curious. If we need to read the paper to answer your question, that makes it far more unlikely that people will have the time to do so. So include enough for us to get the context. Also clarify exactly what part of this is confusing you. What is it you do not understand?  
  > **username_0**: @username_1 added paper description to my best abilities. Forgive for I'm still a beginner  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-13. Score: 1 | Accepted answer
  
  Okay I see. Basically there's a threshold for removing low intensity signal that can't be distinguished from background. That threshold seems tight at 1% of the mean of the normal distributio. Only the signal that is highly significantly above background signal is retained, i.e. subject to further analysis.
  
  The way it works is that for a given CpG site there is a) methylated allele probe and b) unmethylated allele probe. The beta-value is a representation of those two intensities. What they want to do is filter out the background signal to avoid false positives.
  
  Here's the manual <https://bioconductor.org/packages/devel/bioc/vignettes/minfi/inst/doc/minfi.html#5_Quality_control>
  
  Here's a basic code (I stress the word *basic*) **HOWEVER** see the note on Noob in minfi below:
  
  ```
  library(minfi)
  
  myStuff <- read.metharray.exp("/Users/username/Desktop/myfile")
  rgSet <- preprocessQuantile(myStuff)
  
  M <- getM(rgSet)
  U <- getU(rgSet)
  
  beta <- M / (M + U)
  
  filterProbesByZTest <- function(M, U, p.threshold = 0.01) {
    n <- ncol(M)
    filteredProbes <- rep(TRUE, ncol(M)) 
    
    for (i in 1:ncol(M)) {
      probeM <- M[, i]
      probeU <- U[, i]
      
      meanM <- mean(probeM, na.rm = TRUE)
      meanU <- mean(probeU, na.rm = TRUE)
      sdM <- sd(probeM, na.rm = TRUE)
      sdU <- sd(probeU, na.rm = TRUE)
      
      z_stat <- (meanM - meanU) / sqrt((sdM^2 / length(probeM)) + (sdU^2 / length(probeU)))
      p_value <- 2 * pnorm(-abs(z_stat))
      
      if (p_value > p.threshold) {
        filteredProbes[i] <- FALSE
      }
    }
    
    return(filteredProbes)
  }
  
  filteredProbes <- filterProbesByZTest(M, U, p.threshold = 0.01)
  betaFiltered <- beta[, filteredProbes]
  
  ```
  
  ---
  
  Note apart from being *basic*. the code isn't an exact replication of the authors method because they used Noob
  
  > ll data processing was performed in R (version 3.6.1; R Foundation for Statistical Computing, Vienna, Austria). Using the minfi library [26], raw IDAT files were imported; Noob normalization was performed [27]; and beta values were computed. Beta values whose corresponding intensities on probe-level failed a z-test with a P value threshold of 0.01 against background signal were masked.
  
  `preprocessNoob` by [Triche et al (2013)](https://doi.org/10.1093/nar/gkt090)
  
  My suspicion is that `preprocessNoob` comprises the above function. I'd need to read the paper in detail.
  
  `preprocessQuantile` was used above and was part of the original [minfi package](https://doi.org/10.1093/bioinformatics/btu049)
  
  ---
  
  **Note** A key minfi paper describing the implementation is:
  <https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0503-2>
  
  The most advanced method I'm aware of is [Heiss and Just (2019)](https://clinicalepigeneticsjournal.biomedcentral.com/articles/10.1186/s13148-019-0615-3)
  There guys have developed a different tool set **[ewastools](https://github.com/hhhh5/ewastools)** ... if its compatible.
  
  Getting involved what is a periphery topic for me at the cutting edge of the field is something I'd rather avoid, admittedly I'm heavily into cancer stuff.
  
  > **Comments:**
  > **username_2**: @username_0 Try the above but read up on Noob.  
  
  ---
  
  