Answers.Body: <p>Negative values are to be expected as you have log-transformed the data. Any data value below 2 will have a negative log2.</p>
  
  <p>There is a difference between normalising for analysis and truncating for plotting. The RMA normalisation or similar is intended for differential expression analysis. I would recommend using this normalisation first.</p>
  
  <p>When plotting, the normalised data may still contain outliers which skew the plotting parameters (such as the colour scale). Due to the large number of genes, there may be some far beyond 3 standard deviations and plotting these at their true value will make it hard to discern trends in the middle. Truncating the data for a heatmap is common, typically at |z| > 3 but quantiles should work much the same. This is a matter of visualisation, not clustering analysis. It would not be needed unless you were plotting the data.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-05T09:08:21.453Z
Answers.Text: Hello Tom, Yes I want to plot these data. You mean I should use the normalisation which is mentioned above in my question?
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-12T07:15:03.427Z
Answers.Text: You should normalise the data first as you wish to analyse it further. The visualisations should as closely matched the data you've used, to adjust for batch effects, etc. If necessary to plot the values on a sensible range further standardisation (such as the z-transform) may be done. While there are best practices, expression analysis is not an exact science and involves a lot of decisions, depending on the variation you want to focus on, for what you can show from the data.
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-26T03:26:46.203Z
Answers.LastActivityDate: 2017-11-26T03:26:46.203Z
Answers.UserDisplayName: Tom Kelly ケリー・トム
Answers.Body: <p>You see negative values with your function because you're setting the average of each row to 0 and its standard deviation to 1.</p>
  
  <p>In general, I would trust a standard normalization method (rma in this case) more than some random "truncate and then scale the rows" method. Your method isn't even doing any between-array normalization, which is the benefit of rma.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-06T20:51:42.013Z
Answers.Text: Thankyou for the reply. So, I will use the rma normalisation method.                                                 ty <- backgroundCorrect(data); normdata <- normalize(ty); And then using the normdata I can generate a clustering heat map. Am I right?
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T10:40:36.833Z
Answers.Text: Could you please tell me whether what I mentioned in my previous comment is right or not?
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T10:47:57.103Z
Answers.Text: That or `rma()` directly.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:22:59.000Z
Answers.Text: Ok. I did the normalisation like I mentioned in my previous comment. I also generated a clustering heat map. Could please have a look and tell me whether this is fine or not for clustering? https://imgur.com/3vZsO4Y
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:24:27.660Z
Answers.Text: It looks like a clustered heatmap, whether it's "fine" is something that only you can answer.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:27:33.553Z
Answers.Text: Yes, I know what I mean is whether there is anything needs to be added for clustering heat map. Thank you !!
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:29:24.950Z
Answers.Text: There isn't anything else absolutely required. It might be nice to not split by `Type`, since I imagine someone would like to see if the samples cluster according to it.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:37:25.987Z
Answers.Text: Ok. But when I don't use split it looks like this. https://imgur.com/a/MJotE
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:38:33.780Z
Answers.Text: That's telling you something that's rather important to know.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:42:28.167Z
Answers.Text: This means samples are not clustered. Am I right?
Answers.UserDisplayName: beginner
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-07T12:43:21.060Z
Answers.Text: They don't cluster by `Type`, at least.
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-06T20:35:15.903Z
Answers.LastActivityDate: 2017-11-06T20:35:15.903Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I wanted to generate a clustering heat map for the microarray data. This is the first time I'm working on Microarray data. I read some tutorials but have few doubts.</p>
  
  <p>I'm using microarray (Affymetrix SNP 6.0 data) gene expression data. For example the data looks like following:</p>
  
  <pre><code>ProbeID S1  S2  S3  S4  S5  S6  S7
  10008   131.4311    369.4926    222.0441    687.4181    176.8892    258.1233    316.5573
  10010   78.73022    83.97501    81.56039    86.11443    78.09758    81.88231    84.17101
  10014   90.02816    95.07267    101.1761    93.35585    81.96468    94.3553 93.89527
  10017   79.86837    81.63064    88.19524    79.47265    76.3437 101.6351    93.71674
  10019   99.03493    109.1835    104.97  102.7423    108.3677    98.93459    101.4052
  10020   79.58075    84.28915    90.53562    74.47786    75.96112    96.39649    95.8828
  10021   121.5373    149.9351    146.5956    122.8523    110.5759    132.4268    130.4409
  10025   616.5994    1326.735    1358.187    2315.851    1068.745    3229.759    4435.021
  10035   70.44073    69.56772    68.25446    68.35857    70.86771    74.3843 67.93569
  </code></pre>
  
  <p>I created expression matrix and did log2 conversion. Now the data looks like following.</p>
  
  <pre><code>            S1  S2  S3  S4  S5  S6  S7
  Gene1   6.429276339 6.338451158 6.333760753 6.419191996 6.503471181 6.329103499 6.211373601
  Gene2   6.379471993 6.296018518 6.237316465 6.2696332   6.329489132 6.359770303 6.240070336
  Gene3   12.84498365 12.00265682 13.92741965 12.553162   13.39001307 13.8933423  12.58695704
  Gene4   8.661860382 6.723004202 6.300975176 7.661012019 7.905219709 6.957578023 6.70945883
  Gene5   6.945382967 6.814979733 6.575916303 6.63460198  6.627380524 6.733926424 6.280618235
  Gene6   9.280222581 8.81560969  9.683073561 9.480038673 8.801438707 9.190943942 7.743705471
  Gene7   6.593564871 6.63502488  6.389962535 6.511360029 6.694572404 6.6492763   6.527199544
  Gene8   6.431615372 6.309515078 6.248153876 6.329288965 6.46768078  6.355268547 6.384284754
  Gene9   7.513349406 7.234654595 7.490935892 6.801368215 7.323811386 7.018733196 7.055932044
  </code></pre>
  
  <p>Before clustering for normalisation:</p>
  
  <p>I applied a function for normalisation </p>
  
  <pre><code>data = t(apply(data, 1, function(x) {
    q10 = quantile(x, 0.1)
    q90 = quantile(x, 0.9)
    x[x &lt; q10] = q10
    x[x &gt; q90] = q90
    scale(x)
  }))
  </code></pre>
  
  <p>When I apply the above function I see there are values which are positive and also negative. </p>
  
  <p>And when I used "rma" for background correction and "normalise" function from "oligo" package I don't see any negative values.</p>
  
  <p>I don't understand. </p>
  
  <p>Which one should I use to check the clustering among samples? Any help is appreciated.</p>
  
Tags: <r><bioconductor><normalization><microarray>
Title: Normalizing microarray data for clustering heat map
ThreadText: # Normalizing microarray data for clustering heat map
  
  (Asked by: username_0 on 2017-11-06)
  
  I wanted to generate a clustering heat map for the microarray data. This is the first time I'm working on Microarray data. I read some tutorials but have few doubts.
  
  I'm using microarray (Affymetrix SNP 6.0 data) gene expression data. For example the data looks like following:
  
  ```
  ProbeID S1  S2  S3  S4  S5  S6  S7
  10008   131.4311    369.4926    222.0441    687.4181    176.8892    258.1233    316.5573
  10010   78.73022    83.97501    81.56039    86.11443    78.09758    81.88231    84.17101
  10014   90.02816    95.07267    101.1761    93.35585    81.96468    94.3553 93.89527
  10017   79.86837    81.63064    88.19524    79.47265    76.3437 101.6351    93.71674
  10019   99.03493    109.1835    104.97  102.7423    108.3677    98.93459    101.4052
  10020   79.58075    84.28915    90.53562    74.47786    75.96112    96.39649    95.8828
  10021   121.5373    149.9351    146.5956    122.8523    110.5759    132.4268    130.4409
  10025   616.5994    1326.735    1358.187    2315.851    1068.745    3229.759    4435.021
  10035   70.44073    69.56772    68.25446    68.35857    70.86771    74.3843 67.93569
  
  ```
  
  I created expression matrix and did log2 conversion. Now the data looks like following.
  
  ```
              S1  S2  S3  S4  S5  S6  S7
  Gene1   6.429276339 6.338451158 6.333760753 6.419191996 6.503471181 6.329103499 6.211373601
  Gene2   6.379471993 6.296018518 6.237316465 6.2696332   6.329489132 6.359770303 6.240070336
  Gene3   12.84498365 12.00265682 13.92741965 12.553162   13.39001307 13.8933423  12.58695704
  Gene4   8.661860382 6.723004202 6.300975176 7.661012019 7.905219709 6.957578023 6.70945883
  Gene5   6.945382967 6.814979733 6.575916303 6.63460198  6.627380524 6.733926424 6.280618235
  Gene6   9.280222581 8.81560969  9.683073561 9.480038673 8.801438707 9.190943942 7.743705471
  Gene7   6.593564871 6.63502488  6.389962535 6.511360029 6.694572404 6.6492763   6.527199544
  Gene8   6.431615372 6.309515078 6.248153876 6.329288965 6.46768078  6.355268547 6.384284754
  Gene9   7.513349406 7.234654595 7.490935892 6.801368215 7.323811386 7.018733196 7.055932044
  
  ```
  
  Before clustering for normalisation:
  
  I applied a function for normalisation
  
  ```
  data = t(apply(data, 1, function(x) {
    q10 = quantile(x, 0.1)
    q90 = quantile(x, 0.9)
    x[x < q10] = q10
    x[x > q90] = q90
    scale(x)
  }))
  
  ```
  
  When I apply the above function I see there are values which are positive and also negative.
  
  And when I used "rma" for background correction and "normalise" function from "oligo" package I don't see any negative values.
  
  I don't understand.
  
  Which one should I use to check the clustering among samples? Any help is appreciated.
  
  **Tags:** r, bioconductor, normalization, microarray
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-11-06. Score: 5 | Accepted answer
  
  You see negative values with your function because you're setting the average of each row to 0 and its standard deviation to 1.
  
  In general, I would trust a standard normalization method (rma in this case) more than some random "truncate and then scale the rows" method. Your method isn't even doing any between-array normalization, which is the benefit of rma.
  
  > **Comments:**
  > **username_0**: Thankyou for the reply. So, I will use the rma normalisation method.                                                 ty <- backgroundCorrect(data); normdata <- normalize(ty); And then using the normdata I can generate a clustering heat map. Am I right?  
  > **username_0**: Could you please tell me whether what I mentioned in my previous comment is right or not?  
  > **username_2**: That or `rma()` directly.  
  > **username_0**: Ok. I did the normalisation like I mentioned in my previous comment. I also generated a clustering heat map. Could please have a look and tell me whether this is fine or not for clustering? https://imgur.com/3vZsO4Y  
  > **username_2**: It looks like a clustered heatmap, whether it's "fine" is something that only you can answer.  
  > **username_0**: Yes, I know what I mean is whether there is anything needs to be added for clustering heat map. Thank you !!  
  > **username_2**: There isn't anything else absolutely required. It might be nice to not split by `Type`, since I imagine someone would like to see if the samples cluster according to it.  
  > **username_0**: Ok. But when I don't use split it looks like this. https://imgur.com/a/MJotE  
  > **username_2**: That's telling you something that's rather important to know.  
  > **username_0**: This means samples are not clustered. Am I right?  
  > **username_2**: They don't cluster by `Type`, at least.  
  
  ---
  
  ### Answer by username_1 on 2017-11-26. Score: 1
  
  Negative values are to be expected as you have log-transformed the data. Any data value below 2 will have a negative log2.
  
  There is a difference between normalising for analysis and truncating for plotting. The RMA normalisation or similar is intended for differential expression analysis. I would recommend using this normalisation first.
  
  When plotting, the normalised data may still contain outliers which skew the plotting parameters (such as the colour scale). Due to the large number of genes, there may be some far beyond 3 standard deviations and plotting these at their true value will make it hard to discern trends in the middle. Truncating the data for a heatmap is common, typically at |z| > 3 but quantiles should work much the same. This is a matter of visualisation, not clustering analysis. It would not be needed unless you were plotting the data.
  
  > **Comments:**
  > **username_0**: Hello Tom, Yes I want to plot these data. You mean I should use the normalisation which is mentioned above in my question?  
  > **username_1**: You should normalise the data first as you wish to analyse it further. The visualisations should as closely matched the data you've used, to adjust for batch effects, etc. If necessary to plot the values on a sensible range further standardisation (such as the z-transform) may be done. While there are best practices, expression analysis is not an exact science and involves a lot of decisions, depending on the variation you want to focus on, for what you can show from the data.  
  
  ---
  
  