Answers.Body: <p>I am not sure how much you know about bioinformatics already, can you use R? For a bioinformatician looking at QC for microarrays should not be a big deal, at least for me it would take maybe a day (or two) to get this done. However, if you never used R and want to start from scratch, it depends on how quickly you learn how to deal with arrays and QC. There are many tutorials and review papers about normalization and QC of arrays. Try there and see if you can easily understand what these papers are telling you.</p>
  
  <p>Read the limma user's guide, and try to follow the tutorials therein. If you understand that quickly you can try a real experiment of your own. If you don't understand what's described in there, you might want to look for another project for your thesis.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-02T12:06:40.227Z
Answers.LastActivityDate: 2018-02-02T12:23:44.583Z
Answers.LastEditDate: 2018-02-02T12:23:44.583Z
Answers.UserDisplayName: benn
Body: <p>I am a PhD student working on developing ideas for my dissertation papers. One of my planned papers will be working with some (human) gene expression data.</p>
  
  <p>I have had a class on working with gene expression data, but no practical experience. From what I remember from my project in that class, you do a bunch of pre-checks and QC, then you remove batch effects, then you summarize expression across probes and normalize. I thank that part of my class project took me about three days.</p>
  
  <p>But when I talk to other people about it, they say that cleaning/QCing/normalizing gene expression data is really hard, that it will take forever to get it working, and that if I do this dissertation paper I am going to end up taking extra years to graduate.</p>
  
  <p>Am I missing something here? Why do the people I talk to seem to think this is so hard and time consuming -- is it really that bad? Should I avoid having a dissertation paper that uses gene expression data if I want to graduate on time?</p>
  
  <p>(The data will be from Affymetrix microarrays. There are enough samples to use two 96-well plates, so batch effects will be more of a concern than they were in my class project.)</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-01T22:29:28.907Z
Comments.Text: Most of the problems with normalizing data are due to a lack of proper design. How is the experiment designed? What do you plan to do if your QC fails? Will it take months to repeat the same experiment? Or it will take 1k$ more to repeat it and 2 weeks? (Who is running with the costs?) What do you expect to learn from this experiment? Aren't there already public datasets that answer your question (I don't know what do you need to do for your dissertation paper)? Why do you do NGS instead of microarrays? (Sorry for the questions, but they will give context about why your idea)
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-01T23:19:08.067Z
Comments.Text: Thanks for commenting! The samples are all collected and frozen, and haven't been run yet. There is a correlation structure in the data (we sampled the same people repeatedly over time) that might complicate things. Because the samples haven't been run yet, we're hoping we can organize our plating so that all samples from the same person end up in the same column on the same plate to minimize batch effects. We are using microarrays for cost reasons. If lots of samples fail QC, the project dies, we won't have money to do it again.
Comments.UserDisplayName: bluemouse
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-01T23:23:20.690Z
Comments.Text: If you put all the samples of an individual on the same plate you will not be able to distinguish between batch effects and individual effects!! That's the thing that's tricky when performing those analysis (Realizing the implications and later problems you will face when analyzing the data). How many samples do you have? Is there any other condition involved (origin of the sample/hospital/date/treatment/age/sex...) that could affect your plates? If it is your first time check your **design** with more experienced people of your lab before running anything (or spending the samples)!
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-01T23:23:33.530Z
Comments.Text: More info: We'll be looking at within-individual changes over time, and between-individual differences at the same timepoint.
Comments.UserDisplayName: bluemouse
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-01T23:31:49.037Z
Comments.Text: Very good point on the batch vs. individual effects Llopis!!! I will be discussing this with more experienced people before this gets plated/run. We do have differences in age, sex, and race. My main concern at this point is "do I really want this to be a dissertation paper, or is it going to turn into a big mess and make me a 7-year PhD student?"... so far the two more experienced people I've talked to seem to think this should NOT be a dissertation paper, but my advisor (who hasn't done gene expression before) **really** wants me to do it...
Comments.UserDisplayName: bluemouse
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-02T07:53:27.267Z
Comments.Text: I was about to write an answer to your questions, but your main question is if you should do it or not, and I can't answer that. If you want perhaps you can change the question to what aspects to you need to take into account when designing a microarray experiment (which would be a very nice question with many references) which could be answered in the format of this platform. Good luck with your disseration
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-02T09:56:22.273Z
Comments.Text: It sounds like you're facing an academic, rather than a technical issue. I suggest you [edit] your question here to focus only on the technical aspects. Describe the experiment you want to run in detail and ask how long you should expect the QC step to take. The question of whether to do it and whether it should be a part of your dissertation can then be asked on [academia.se].
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-03T10:48:25.640Z
Comments.Text: @terdon The main question (time scale) is mostly dependent on technical aspects of the experimental design and I do not think that other communities would be able to properly discuss it.
  
  At the same time I agree with terdon that you should [edit] your question and add all this spicy details you wrote in the comments. Also, note that your question should be answerable, which is something I doubt a bit.
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-02-14T10:02:50.513Z
Comments.Text: I'm voting to close this question as off-topic because it's not a real technical question
Comments.UserDisplayName: mgalardini
Tags: <microarray><quality-control><data-preprocessing><batch-effects>
Title: How hard is it to clean and QC gene expression microarray data?
ThreadText: # How hard is it to clean and QC gene expression microarray data?
  
  (Asked by: username_0 on 2018-02-01)
  
  I am a PhD student working on developing ideas for my dissertation papers. One of my planned papers will be working with some (human) gene expression data.
  
  I have had a class on working with gene expression data, but no practical experience. From what I remember from my project in that class, you do a bunch of pre-checks and QC, then you remove batch effects, then you summarize expression across probes and normalize. I thank that part of my class project took me about three days.
  
  But when I talk to other people about it, they say that cleaning/QCing/normalizing gene expression data is really hard, that it will take forever to get it working, and that if I do this dissertation paper I am going to end up taking extra years to graduate.
  
  Am I missing something here? Why do the people I talk to seem to think this is so hard and time consuming -- is it really that bad? Should I avoid having a dissertation paper that uses gene expression data if I want to graduate on time?
  
  (The data will be from Affymetrix microarrays. There are enough samples to use two 96-well plates, so batch effects will be more of a concern than they were in my class project.)
  
  **Tags:** microarray, quality-control, data-preprocessing, batch-effects
  
  > **Comments:**
  > **username_1**: Most of the problems with normalizing data are due to a lack of proper design. How is the experiment designed? What do you plan to do if your QC fails? Will it take months to repeat the same experiment? Or it will take 1k$ more to repeat it and 2 weeks? (Who is running with the costs?) What do you expect to learn from this experiment? Aren't there already public datasets that answer your question (I don't know what do you need to do for your dissertation paper)? Why do you do NGS instead of microarrays? (Sorry for the questions, but they will give context about why your idea)  
  > **username_0**: Thanks for commenting! The samples are all collected and frozen, and haven't been run yet. There is a correlation structure in the data (we sampled the same people repeatedly over time) that might complicate things. Because the samples haven't been run yet, we're hoping we can organize our plating so that all samples from the same person end up in the same column on the same plate to minimize batch effects. We are using microarrays for cost reasons. If lots of samples fail QC, the project dies, we won't have money to do it again.  
  > **username_1**: If you put all the samples of an individual on the same plate you will not be able to distinguish between batch effects and individual effects!! That's the thing that's tricky when performing those analysis (Realizing the implications and later problems you will face when analyzing the data). How many samples do you have? Is there any other condition involved (origin of the sample/hospital/date/treatment/age/sex...) that could affect your plates? If it is your first time check your **design** with more experienced people of your lab before running anything (or spending the samples)!  
  > **username_0**: More info: We'll be looking at within-individual changes over time, and between-individual differences at the same timepoint.  
  > **username_0**: Very good point on the batch vs. individual effects Llopis!!! I will be discussing this with more experienced people before this gets plated/run. We do have differences in age, sex, and race. My main concern at this point is "do I really want this to be a dissertation paper, or is it going to turn into a big mess and make me a 7-year PhD student?"... so far the two more experienced people I've talked to seem to think this should NOT be a dissertation paper, but my advisor (who hasn't done gene expression before) **really** wants me to do it...  
  > **username_1**: I was about to write an answer to your questions, but your main question is if you should do it or not, and I can't answer that. If you want perhaps you can change the question to what aspects to you need to take into account when designing a microarray experiment (which would be a very nice question with many references) which could be answered in the format of this platform. Good luck with your disseration  
  > **username_2**: It sounds like you're facing an academic, rather than a technical issue. I suggest you [edit] your question here to focus only on the technical aspects. Describe the experiment you want to run in detail and ask how long you should expect the QC step to take. The question of whether to do it and whether it should be a part of your dissertation can then be asked on [academia.se].  
  > **username_3**: @username_2 The main question (time scale) is mostly dependent on technical aspects of the experimental design and I do not think that other communities would be able to properly discuss it.  At the same time I agree with username_2 that you should [edit] your question and add all this spicy details you wrote in the comments. Also, note that your question should be answerable, which is something I doubt a bit.  
  > **username_4**: I'm voting to close this question as off-topic because it's not a real technical question  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_5 on 2018-02-02. Score: 3 | Accepted answer
  
  I am not sure how much you know about bioinformatics already, can you use R? For a bioinformatician looking at QC for microarrays should not be a big deal, at least for me it would take maybe a day (or two) to get this done. However, if you never used R and want to start from scratch, it depends on how quickly you learn how to deal with arrays and QC. There are many tutorials and review papers about normalization and QC of arrays. Try there and see if you can easily understand what these papers are telling you.
  
  Read the limma user's guide, and try to follow the tutorials therein. If you understand that quickly you can try a real experiment of your own. If you don't understand what's described in there, you might want to look for another project for your thesis.
  
  
  ---
  
  