Answers.Body: <p>I think others have provided fine answers to your question already. I just thought it might be relevant to mention about a tool that allows using Spark to distribute computations that re-use existing commandline bioinformatics software ... so, avoiding the need to re-implement proven algorithms in a language that Spark supports, <a href="https://github.com/mcapuccini/EasyMapReduce" rel="nofollow noreferrer">EasyMapReduce</a>. </p>
  
  <p>This perceived need to re-implement algorithms, has otherwise been one of the biggest road blocks for wider adoption of Spark in the bioinformatics community in my understanding. </p>
  
  <p>EasyMapReduce was presented SparkSummit East this year (video link <a href="https://www.youtube.com/watch?v=4C4R9qptUQo" rel="nofollow noreferrer">here</a>, as shown on the repository home page).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-06T20:53:30.430Z
Answers.LastActivityDate: 2017-06-06T21:40:05.000Z
Answers.LastEditDate: 2017-06-06T21:40:05.000Z
Answers.UserDisplayName: Samuel Lampa
Answers.Body: <p><a href="https://github.com/bigdatagenomics/adam" rel="noreferrer">ADAM</a> and <a href="https://github.com/bigdatagenomics/avocado" rel="noreferrer">avocado</a> are Spark-based alignment and variant calling tools under active development by a collaboration (<a href="http://bdgenomics.org/" rel="noreferrer">http://bdgenomics.org</a>) that also includes the Broad, but I do not believe they have wide adoption.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-04T20:22:22.863Z
Answers.LastActivityDate: 2017-06-04T21:17:09.970Z
Answers.LastEditDate: 2017-06-04T21:17:09.970Z
Answers.UserDisplayName: leekaiinthesky
Answers.Body: <p>The only bioinformatics tool other than GATK4 that I am aware of that uses spark is <a href="https://github.com/hail-is/hail" rel="noreferrer">Hail</a> (a Spark based replacement for Plink). Hail is also supported by researchers at the Broad.</p>
  
  <p>Most places I have worked at have not switched over to Spark.  As such, I don't think it is widely used generally. Therefore, I don't think knowing Spark qualifies as a necessary skill to learn for bioinformatics at this time.</p>
  
  <p>However, I think Spark is a superior paradigm for handling the large amount of data that bioinformaticians routinely deal with, and I think we will soon see the field move towards using it more.  Any bioinformatician would do well to acquaint themselves with Spark and to play around with Hail.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-04T04:36:47.933Z
Answers.Text: From reading into that a bit more, it looks like calling Hail a Spark-based replacement for Plink is selling it short. It's mostly true, but a bit like calling a car a wheel-based replacement for a horse. The efficiency of variant storage, and ability to distribute the workload, seems like quite a disruptive change in how variant analysis, filtering and QC is currently done.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-04T11:26:29.313Z
Answers.Text: I agree with you.  On the other hand, I think there is value in thinking of a car as a wheel-based replacement for a horse O,o
Answers.UserDisplayName: winni2k
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-05T11:57:12.120Z
Answers.Text: This is my impression as well: few tools are using Spark as yet, but there is a lot of interest in Spark as an idiom for concurrency. It's more suited to biological problems than a many other choices and is less complicated. So we'll probably see a lot more use of it in the future.
Answers.UserDisplayName: agapow
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-04T01:33:49.913Z
Answers.LastActivityDate: 2017-06-04T01:33:49.913Z
Answers.UserDisplayName: winni2k
Body: <p>I learned that GATK 4 is using <a href="https://spark.apache.org/" rel="nofollow noreferrer">Spark</a> for parallelization. I googled around, though I am still not quite sure how spark really works and how to use it in practice. Besides GATK 4, are any other bioinformatics tool using spark? Generally, is spark widely used? Is it a necessary skill to learn? Thanks in advance.</p>
  
Tags: <gatk>
Title: Is spark widely used in bioinformatics?
ThreadText: # Is spark widely used in bioinformatics?
  
  (Asked by: username_0 on 2017-06-03)
  
  I learned that GATK 4 is using [Spark](https://spark.apache.org/) for parallelization. I googled around, though I am still not quite sure how spark really works and how to use it in practice. Besides GATK 4, are any other bioinformatics tool using spark? Generally, is spark widely used? Is it a necessary skill to learn? Thanks in advance.
  
  **Tags:** gatk
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-06-04. Score: 8 | Accepted answer
  
  The only bioinformatics tool other than GATK4 that I am aware of that uses spark is [Hail](https://github.com/hail-is/hail) (a Spark based replacement for Plink). Hail is also supported by researchers at the Broad.
  
  Most places I have worked at have not switched over to Spark. As such, I don't think it is widely used generally. Therefore, I don't think knowing Spark qualifies as a necessary skill to learn for bioinformatics at this time.
  
  However, I think Spark is a superior paradigm for handling the large amount of data that bioinformaticians routinely deal with, and I think we will soon see the field move towards using it more. Any bioinformatician would do well to acquaint themselves with Spark and to play around with Hail.
  
  > **Comments:**
  > **username_4**: From reading into that a bit more, it looks like calling Hail a Spark-based replacement for Plink is selling it short. It's mostly true, but a bit like calling a car a wheel-based replacement for a horse. The efficiency of variant storage, and ability to distribute the workload, seems like quite a disruptive change in how variant analysis, filtering and QC is currently done.  
  > **username_3**: I agree with you.  On the other hand, I think there is value in thinking of a car as a wheel-based replacement for a horse O,o  
  > **username_5**: This is my impression as well: few tools are using Spark as yet, but there is a lot of interest in Spark as an idiom for concurrency. It's more suited to biological problems than a many other choices and is less complicated. So we'll probably see a lot more use of it in the future.  
  
  ---
  
  ### Answer by username_1 on 2017-06-06. Score: 6
  
  I think others have provided fine answers to your question already. I just thought it might be relevant to mention about a tool that allows using Spark to distribute computations that re-use existing commandline bioinformatics software ... so, avoiding the need to re-implement proven algorithms in a language that Spark supports, [EasyMapReduce](https://github.com/mcapuccini/EasyMapReduce).
  
  This perceived need to re-implement algorithms, has otherwise been one of the biggest road blocks for wider adoption of Spark in the bioinformatics community in my understanding.
  
  EasyMapReduce was presented SparkSummit East this year (video link [here](https://www.youtube.com/watch?v=4C4R9qptUQo), as shown on the repository home page).
  
  
  ---
  
  ### Answer by username_2 on 2017-06-04. Score: 6
  
  [ADAM](https://github.com/bigdatagenomics/adam) and [avocado](https://github.com/bigdatagenomics/avocado) are Spark-based alignment and variant calling tools under active development by a collaboration ([http://bdgenomics.org](http://bdgenomics.org/)) that also includes the Broad, but I do not believe they have wide adoption.
  
  
  ---
  
  