Answers.Body: <p>I prefer to treat software tools and computers in a similar fashion to laboratory equipment, and in some sense biology in general. Biologists are used to unexpected things happening in their experiments, and it's not uncommon for a new discovery to change the way that people look at something. Things break down, cells die off quicker on a Wednesday afternoon, results are inconsistent, and that third reviewer keeps on about doing that thing that's been done a hundred times before without anything surprising happening (just not <em>this</em> time). It's a good idea to record as much as can be thought of that might influence an experiment, and for software that includes any input data or command line options, and especially software version numbers.</p>
  
  <p>In this sense, a discovered software bug can be treated as a new discovery of how the world works. If the discovery is made public, and other people consider that it's important enough, then some people might revisit old research to see if it changes things.</p>
  
  <p>Of course, the nice thing about software is that bugs can be reported back to the creators of programs, and possibly fixed, resulting in an improved version of the software at a later date. If the bug itself doesn't spark interest and the program gets fixed anyway, people unknowingly use newer versions, and there might be a bit more confusion and discussion about why results don't match similar studies carried out before the software change.</p>
  
  <p>If you want a bit of an idea of the biological equivalent of a major software bug, have a look at the barcode <a href="https://www.biorxiv.org/content/early/2017/04/09/125724" rel="noreferrer">index switching</a> issue, or the <a href="https://blogs.biomedcentral.com/on-biology/2016/03/22/mix-mayhem-cell-line-contamination-biological-research/" rel="noreferrer">cell line contamination</a> issue.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T12:04:57.783Z
Answers.Text: Great answer (+1), except for introducing the IMHO unnecessary and confusing distinction between 'corner' and 'non-corner' case bugs. A code is not like a lab equipment in that you do not have a general understanding over the state-space of how the tool is used (e.g. you do not need to label all lab equipment with 'this tool might not work in outer space').  Another way to put this: all bugs are necessarily corner cases to the person that wrote the code.
Answers.UserDisplayName: user189035
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T18:48:02.510Z
Answers.Text: One person's bug could be another person's (feature)[https://xkcd.com/1172/]. A person doesn't need to know (and often can't know) everything about software in order to use it, and developers don't need to fix every known bug in order for a program to be useful.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-15T02:49:51.350Z
Answers.Text: Thanks for those two links @gringer, definitely an interesting read.
Answers.UserDisplayName: Nic Barker
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-15T08:18:35.960Z
Answers.Text: "people unknowingly use newer versions, and there might be a bit more confusion and discussion about why results don't match similar studies carried out before the software change." Which is why you should note the versions of the software you used in your publications. However small and however subtle you do it, make sure it's in there.
Answers.UserDisplayName: Mast
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T02:13:27.307Z
Answers.LastActivityDate: 2017-11-15T08:41:59.447Z
Answers.LastEditDate: 2017-11-15T08:41:59.447Z
Answers.UserDisplayName: gringer
Answers.Body: <p>While I agree with <a href="https://bioinformatics.stackexchange.com/a/2836/48">gringer's answer</a> I would first report the bug to the maintainers with a test case (if possible) in order to  have the bug corrected. </p>
  
  <p>The bug might be an erroneous implementation in a corner case or in a more common use case of the software. But it is a good opportunity to collaborate to increase the quality of the software, either by introducing tests (or expanding them), by re-evaluating the state of the art with other tools, or to communicate with users of other software that depends on the tool.</p>
  
  <p>Depending on the bug and the time one is able to spend, do a collaboration to study the effect of such a bug in the published studies using the software.</p>
  
  <p>Sometimes the bug doesn't affect much, but sometimes it can have quite an impact which can be difficult to estimate. For instance, I <a href="https://support.bioconductor.org/p/92979/" rel="nofollow noreferrer">found</a> that the reactome.db package didn't have the same annotations that the available online database (in version 1.58.0), due to an incorrect mapping. However, it was impossible for me to evaluate the damage of such an error on the published literature, because gene set enrichment tools (usually) don't report the data they are based on and the versions of the databases used are often under reported in the literature.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T11:26:29.600Z
Answers.Text: Don't worry about it! That's the beauty of a collaboratively edited site! And I cheat: I'm a native speaker :)
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T08:15:17.873Z
Answers.LastActivityDate: 2017-11-15T09:32:32.397Z
Answers.LastEditDate: 2017-11-15T09:32:32.397Z
Answers.UserDisplayName: llrs
Answers.Body: <p>It depends also where the package is available from. I mean if it is from <a href="https://bioconductor.org/" rel="nofollow noreferrer">bioconductor</a>, authors and maintainers are responsible to <a href="https://www.bioconductor.org/developers/package-guidelines/#responsibilities" rel="nofollow noreferrer">respond to bug</a> reports. It could be reported at <a href="https://support.bioconductor.org/" rel="nofollow noreferrer">support sites</a>, so others are also aware of the bug. If the maintainer doesn't fix it, it might lose its spot in bioconductor. However, other sites, such as CRAN or Sourgeforge might be less stringent.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T10:39:09.147Z
Answers.Text: I always wonder what the "Response to bug reports" means, but that would be for the bioconductor mailing list. I read it as they don't need to be corrected, but just have a (reasonable) response. And AFAIK it is not enforced, I reported several bugs and I didn't get any response from the maintainers :(
Answers.UserDisplayName: llrs
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T10:43:29.597Z
Answers.Text: In my opinion an adequate response would be a bug fix. Not just a "thanks, you'll hear from us" response on the support site. It's a pity to hear that you don't even got a response, I thought bioconductor was stringent is these matters.
Answers.UserDisplayName: benn
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T10:27:49.730Z
Answers.LastActivityDate: 2017-11-14T10:27:49.730Z
Answers.UserDisplayName: benn
Answers.Body: <p>If you have a software development background you probably know about filing bug reports. Your question seems to be more about how to correct the scientific literature which has been affected by the bug. </p>
  
  <p>The traditional way that this has been done is by writing an paper that explains the bug and gives examples of how it may impact published results (see gringer's examples of index switching and HELA contamination). This way the community will be aware that older results using the implicated method need to be verified before building upon them. Unfortunately, awareness of these issues doesn't always diffuse through the community as quickly as it should, but it's probably still the best tool we have to ensure that science is self-correcting.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-15T07:39:30.290Z
Answers.Text: I wonder if you could publish enough papers purely from bug-hunting to earn your way to a normal career in academia ;)
Answers.UserDisplayName: curiousdannii
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-14T12:39:54.510Z
Answers.LastActivityDate: 2017-11-14T12:39:54.510Z
Answers.UserDisplayName: heathobrien
Answers.Body: <p>In extreme cases, it can destroy careers:</p>
  
  <blockquote>
    <p>One of the most spectacular flameouts in science happened last year.
    In a short letter (barely over 300 words long) published in Science in
    the very last issue of 2006, Geoffrey Chang, a crystallographer,
    retracted 3 Science articles, a Nature article, a PNAS article and a
    JMB article. The sum of 5 years of work was destroyed, apparently,
    over a single sign error in a data-processing program.
    – <a href="http://boscoh.com/protein/a-sign-a-flipped-structure-and-a-scientific-flameout-of-epic-proportions.html" rel="noreferrer">A Sign, a Flipped Structure, and a Scientific Flameout of Epic Proportions</a></p>
  </blockquote>
  
  <p>Most cases are not that extreme. In fact, realistically, I would guess that <em>most</em> papers have some data processing errors of some kinds, yet still we muddle on towards truth. And this isn't unique to software – the whole idea of independent replication in science is that we're always probably messing things up in ways we don't realize, so we need to re-do things and hope that the errors are different this time. But software is unusual in how tiny errors can cause very large effects, and often the effects are hard to detect. If you drop a test tube on the floor, you'll probably notice. If a piece of new software gives incorrect results... well, we didn't know what the results were supposed to look like, that's what makes it science! There are things you can do to <a href="https://www.nature.com/news/blind-analysis-hide-results-to-seek-the-truth-1.18510" rel="noreferrer">mitigate this somewhat</a>, but to some extent it's just inherent to the scientific process.</p>
  
  <p>I still like to show that blog post to new grad students to scare them into writing tests. I actually think it's a good habit to, whenever you start using a new software package, start by writing some tests for it – it's a good way to check your assumptions and understanding about what it actually does, and even if it's a popular package by eminent researchers, you'll still sometimes be shocked by what you find.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-11-15T12:05:23.593Z
Answers.LastActivityDate: 2017-11-15T12:05:23.593Z
Answers.UserDisplayName: Nathaniel J. Smith
Body: <p>Yesterday I was debugging some things in R trying to get a popular Flow Cytometry tool to work on our data. After a few hours of digging into the package I discovered that our data was hitting an edge case, and it seems like the algorithm wouldn't work correctly under certain circumstances.</p>
  
  <p>This bug is not neccessarily the most significant issue, but I'm confident that it has occurred for other users before me, perhaps in a more insidious and less obvious way.</p>
  
  <p>Given that since this tool was released, literature has been published that utiltised it, <strong>how does the profession handle the discovery of bugs in these widely used packages?</strong></p>
  
  <p>I'm sure that given the piecemeal nature of a lot of the libraries out there, this is going to happen in a much more significant way at some point (i.e. invalidating a large number of published results)</p>
  
  <p>For some context I'm a programmer who's been working across various other parts of software development, and is quite new to Bioinformatics.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-14T03:27:45.230Z
Comments.Text: Have you reported the bug to the developers?  Is the tool still maintained?  Really it depends on the developers how the bug will be handled.  Sometime it will go out in the next point release and sometime it will be determined to be a low risk and low probability occurrence and the potential fix would introduce great risk so it will stay as a known issue.
Comments.UserDisplayName: Bioathlete
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-14T08:19:03.503Z
Comments.Text: I think this question might benefit from a more general tag. This might come to other packages/libraries in C++, python, perl, S, or whatever language. Maybe a good tag would be [tag:software-quality]?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-14T14:29:40.193Z
Comments.Text: Might also be a relevant question over on Academics.SE
Comments.UserDisplayName: Wayne Werner
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-14T14:39:10.817Z
Comments.Text: @WayneWerner there is a relevant [question there](https://academia.stackexchange.com/q/89599/18238)
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-14T15:18:06.073Z
Comments.Text: I'm not sure I understood the question. "it seems like the algorithm wouldn't work correctly under certain circumstances" - do you mean that the algorithm is incorrect from a biologist's point of view, but was implemented correctly in the package, or that the general algorithm would compute everything properly, but its R implementation in the specific package does not meet the specification?
Comments.UserDisplayName: rumtscho
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-15T02:22:45.753Z
Comments.Text: @Llopis I wanted to create a new general tag, unfortunately I didn't have the reputation at the time. I'll have a think about it and retag if I can.
Comments.UserDisplayName: Nic Barker
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-15T02:27:25.757Z
Comments.Text: @Bioathlete Yep, I probably should have been more clear - I've reported and also fixed the bug, but I was musing about the larger implications on public literature in a more general sense.
Comments.UserDisplayName: Nic Barker
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-15T05:40:25.517Z
Comments.Text: @rumtscho What I mean by that statement is that for some cases it works as expected, and for other cases it doesn't. I.e. for certain data sets the results would be erroneous, but for other data sets it works fine.
Comments.UserDisplayName: Nic Barker
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-15T07:54:13.337Z
Comments.Text: @NicBarker You seem to have created issues on the repository package but you haven't arised these concerns in the support site of Bioconductor. I would first report there your findings. You can always fork the most updated version and modify the code yourself.
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-11-16T00:25:10.223Z
Comments.Text: Thanks for the tip @Llopis, I'm still finding my feet and figuring out best practise in the field.
Comments.UserDisplayName: Nic Barker
Tags: <r><software-quality>
Title: What happens if a major bug is discovered in a bioinformatic package that has been used in published literature?
ThreadText: # What happens if a major bug is discovered in a bioinformatic package that has been used in published literature?
  
  (Asked by: username_0 on 2017-11-14)
  
  Yesterday I was debugging some things in R trying to get a popular Flow Cytometry tool to work on our data. After a few hours of digging into the package I discovered that our data was hitting an edge case, and it seems like the algorithm wouldn't work correctly under certain circumstances.
  
  This bug is not neccessarily the most significant issue, but I'm confident that it has occurred for other users before me, perhaps in a more insidious and less obvious way.
  
  Given that since this tool was released, literature has been published that utiltised it, **how does the profession handle the discovery of bugs in these widely used packages?**
  
  I'm sure that given the piecemeal nature of a lot of the libraries out there, this is going to happen in a much more significant way at some point (i.e. invalidating a large number of published results)
  
  For some context I'm a programmer who's been working across various other parts of software development, and is quite new to Bioinformatics.
  
  **Tags:** r, software-quality
  
  > **Comments:**
  > **username_1**: Have you reported the bug to the developers?  Is the tool still maintained?  Really it depends on the developers how the bug will be handled.  Sometime it will go out in the next point release and sometime it will be determined to be a low risk and low probability occurrence and the potential fix would introduce great risk so it will stay as a known issue.  
  > **username_2**: I think this question might benefit from a more general tag. This might come to other packages/libraries in C++, python, perl, S, or whatever language. Maybe a good tag would be [tag:software-quality]?  
  > **username_3**: Might also be a relevant question over on Academics.SE  
  > **username_2**: @username_3 there is a relevant [question there](https://academia.stackexchange.com/q/89599/18238)  
  > **username_4**: I'm not sure I understood the question. "it seems like the algorithm wouldn't work correctly under certain circumstances" - do you mean that the algorithm is incorrect from a biologist's point of view, but was implemented correctly in the package, or that the general algorithm would compute everything properly, but its R implementation in the specific package does not meet the specification?  
  > **username_0**: @Llopis I wanted to create a new general tag, unfortunately I didn't have the reputation at the time. I'll have a think about it and retag if I can.  
  > **username_0**: @username_1 Yep, I probably should have been more clear - I've reported and also fixed the bug, but I was musing about the larger implications on public literature in a more general sense.  
  > **username_0**: @username_4 What I mean by that statement is that for some cases it works as expected, and for other cases it doesn't. I.e. for certain data sets the results would be erroneous, but for other data sets it works fine.  
  > **username_2**: @username_0 You seem to have created issues on the repository package but you haven't arised these concerns in the support site of Bioconductor. I would first report there your findings. You can always fork the most updated version and modify the code yourself.  
  > **username_0**: Thanks for the tip @Llopis, I'm still finding my feet and figuring out best practise in the field.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_5 on 2017-11-14. Score: 21
  
  I prefer to treat software tools and computers in a similar fashion to laboratory equipment, and in some sense biology in general. Biologists are used to unexpected things happening in their experiments, and it's not uncommon for a new discovery to change the way that people look at something. Things break down, cells die off quicker on a Wednesday afternoon, results are inconsistent, and that third reviewer keeps on about doing that thing that's been done a hundred times before without anything surprising happening (just not *this* time). It's a good idea to record as much as can be thought of that might influence an experiment, and for software that includes any input data or command line options, and especially software version numbers.
  
  In this sense, a discovered software bug can be treated as a new discovery of how the world works. If the discovery is made public, and other people consider that it's important enough, then some people might revisit old research to see if it changes things.
  
  Of course, the nice thing about software is that bugs can be reported back to the creators of programs, and possibly fixed, resulting in an improved version of the software at a later date. If the bug itself doesn't spark interest and the program gets fixed anyway, people unknowingly use newer versions, and there might be a bit more confusion and discussion about why results don't match similar studies carried out before the software change.
  
  If you want a bit of an idea of the biological equivalent of a major software bug, have a look at the barcode [index switching](https://www.biorxiv.org/content/early/2017/04/09/125724) issue, or the [cell line contamination](https://blogs.biomedcentral.com/on-biology/2016/03/22/mix-mayhem-cell-line-contamination-biological-research/) issue.
  
  > **Comments:**
  > **username_6**: Great answer (+1), except for introducing the IMHO unnecessary and confusing distinction between 'corner' and 'non-corner' case bugs. A code is not like a lab equipment in that you do not have a general understanding over the state-space of how the tool is used (e.g. you do not need to label all lab equipment with 'this tool might not work in outer space').  Another way to put this: all bugs are necessarily corner cases to the person that wrote the code.  
  > **username_5**: One person's bug could be another person's (feature)[https://xkcd.com/1172/]. A person doesn't need to know (and often can't know) everything about software in order to use it, and developers don't need to fix every known bug in order for a program to be useful.  
  > **username_0**: Thanks for those two links @username_5, definitely an interesting read.  
  > **username_7**: "people unknowingly use newer versions, and there might be a bit more confusion and discussion about why results don't match similar studies carried out before the software change." Which is why you should note the versions of the software you used in your publications. However small and however subtle you do it, make sure it's in there.  
  
  ---
  
  ### Answer by username_2 on 2017-11-14. Score: 9
  
  While I agree with [username_5's answer](https://bioinformatics.stackexchange.com/a/2836/48) I would first report the bug to the maintainers with a test case (if possible) in order to have the bug corrected.
  
  The bug might be an erroneous implementation in a corner case or in a more common use case of the software. But it is a good opportunity to collaborate to increase the quality of the software, either by introducing tests (or expanding them), by re-evaluating the state of the art with other tools, or to communicate with users of other software that depends on the tool.
  
  Depending on the bug and the time one is able to spend, do a collaboration to study the effect of such a bug in the published studies using the software.
  
  Sometimes the bug doesn't affect much, but sometimes it can have quite an impact which can be difficult to estimate. For instance, I [found](https://support.bioconductor.org/p/92979/) that the reactome.db package didn't have the same annotations that the available online database (in version 1.58.0), due to an incorrect mapping. However, it was impossible for me to evaluate the damage of such an error on the published literature, because gene set enrichment tools (usually) don't report the data they are based on and the versions of the databases used are often under reported in the literature.
  
  > **Comments:**
  > **username_8**: Don't worry about it! That's the beauty of a collaboratively edited site! And I cheat: I'm a native speaker :)  
  
  ---
  
  ### Answer by username_10 on 2017-11-14. Score: 7
  
  If you have a software development background you probably know about filing bug reports. Your question seems to be more about how to correct the scientific literature which has been affected by the bug.
  
  The traditional way that this has been done is by writing an paper that explains the bug and gives examples of how it may impact published results (see username_5's examples of index switching and HELA contamination). This way the community will be aware that older results using the implicated method need to be verified before building upon them. Unfortunately, awareness of these issues doesn't always diffuse through the community as quickly as it should, but it's probably still the best tool we have to ensure that science is self-correcting.
  
  > **Comments:**
  > **username_11**: I wonder if you could publish enough papers purely from bug-hunting to earn your way to a normal career in academia ;)  
  
  ---
  
  ### Answer by username_12 on 2017-11-15. Score: 7
  
  In extreme cases, it can destroy careers:
  
  > One of the most spectacular flameouts in science happened last year.
  > In a short letter (barely over 300 words long) published in Science in
  > the very last issue of 2006, Geoffrey Chang, a crystallographer,
  > retracted 3 Science articles, a Nature article, a PNAS article and a
  > JMB article. The sum of 5 years of work was destroyed, apparently,
  > over a single sign error in a data-processing program.
  > – [A Sign, a Flipped Structure, and a Scientific Flameout of Epic Proportions](http://boscoh.com/protein/a-sign-a-flipped-structure-and-a-scientific-flameout-of-epic-proportions.html)
  
  Most cases are not that extreme. In fact, realistically, I would guess that *most* papers have some data processing errors of some kinds, yet still we muddle on towards truth. And this isn't unique to software – the whole idea of independent replication in science is that we're always probably messing things up in ways we don't realize, so we need to re-do things and hope that the errors are different this time. But software is unusual in how tiny errors can cause very large effects, and often the effects are hard to detect. If you drop a test tube on the floor, you'll probably notice. If a piece of new software gives incorrect results... well, we didn't know what the results were supposed to look like, that's what makes it science! There are things you can do to [mitigate this somewhat](https://www.nature.com/news/blind-analysis-hide-results-to-seek-the-truth-1.18510), but to some extent it's just inherent to the scientific process.
  
  I still like to show that blog post to new grad students to scare them into writing tests. I actually think it's a good habit to, whenever you start using a new software package, start by writing some tests for it – it's a good way to check your assumptions and understanding about what it actually does, and even if it's a popular package by eminent researchers, you'll still sometimes be shocked by what you find.
  
  
  ---
  
  ### Answer by username_9 on 2017-11-14. Score: 4
  
  It depends also where the package is available from. I mean if it is from [bioconductor](https://bioconductor.org/), authors and maintainers are responsible to [respond to bug](https://www.bioconductor.org/developers/package-guidelines/#responsibilities) reports. It could be reported at [support sites](https://support.bioconductor.org/), so others are also aware of the bug. If the maintainer doesn't fix it, it might lose its spot in bioconductor. However, other sites, such as CRAN or Sourgeforge might be less stringent.
  
  > **Comments:**
  > **username_2**: I always wonder what the "Response to bug reports" means, but that would be for the bioconductor mailing list. I read it as they don't need to be corrected, but just have a (reasonable) response. And AFAIK it is not enforced, I reported several bugs and I didn't get any response from the maintainers :(  
  > **username_9**: In my opinion an adequate response would be a bug fix. Not just a "thanks, you'll hear from us" response on the support site. It's a pity to hear that you don't even got a response, I thought bioconductor was stringent is these matters.  
  
  ---
  
  