Answers.Body: <p>As you suggested, sambamba is faster at marking duplicates than picard (it's also multithreaded). Recent versions of samtools have a rewritten duplicate marking algorithm, though I doubt it'll be as quick as sambamba. 46GB of RAM seems excessive for marking duplicates unless you're having it store the whole file in memory.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-12T21:50:50.313Z
Answers.Text: Does sambamba with default parameters equate to Picard MarkDuplicates?
Answers.UserDisplayName: 719016
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-12T23:55:43.347Z
Answers.Text: I'm not sure that any tool exactly equates (there are a lot of edge cases in duplicate marking and picard misses some things), but it's a sufficient replacement. Sambamba is also used for this purpose by a number of international consortia (I think either DEEP or IHEC, if I remember correctly).
Answers.UserDisplayName: Devon Ryan
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-11T20:05:44.793Z
Answers.LastActivityDate: 2017-12-11T20:05:44.793Z
Answers.UserDisplayName: Devon Ryan
Body: <p>I am running Picard MarkDuplicates with the following parameters below. On the file described, it takes about 41.6Gb of RAM memory and about 20-25 minutes to compute (only uses 1 core AFAICS).</p>
  
  <pre><code>java -Xmx54890m -jar picard.jar MarkDuplicates \
      I=/home/dnanexus/in/sorted_bam/TST73-86-2IC_S16_L00.bml.GRCh38.karyo.bam \
      O=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.deduplicated.bam \
      M=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.duplication_metrics \
      CREATE_INDEX=true VALIDATION_STRINGENCY=SILENT REMOVE_DUPLICATES=true
  </code></pre>
  
  <p>What equivalent free/open source software (samtools?, sambamba?) can I use to obtain the same deduplicated bam file but at a lower cost, considering the instance used here has 8 cores but only one of them is being used, and has about 68Gb of RAM out of which it's using 46Gb odd at peak.</p>
  
  <p>I think previous versions of samtools/sambamba didn't behave the same way as Picard MarkDuplicates with the parameters above, but I would like to know which recent versions would give me the same output.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-11T15:58:04.443Z
Comments.Text: I assume you only want free (in terms of price, at least) software, right? [Sentieon](http://sentieon.com/) have a tool for this that is orders of magnitude faster (and I would guess uses less RAM but I'd have to check) but you need to pay for it.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-11T16:05:59.533Z
Comments.Text: Correct, open source.
Comments.UserDisplayName: 719016
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-11T20:24:48.547Z
Comments.Text: I done no benchmark, but I guess [samblaster](https://github.com/GregoryFaust/samblaster) is bit faster (by my subjective perception of time).
Comments.UserDisplayName: Kamil S Jaron
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-12-12T15:21:54.107Z
Comments.Text: You should specify a smaller "-Xmx". MarkDuplicate doesn't need that much memory, but seeing a large "-Xmx", java will use extra.
Comments.UserDisplayName: user172818
Tags: <sam><samtools><sambamba><picard>
Title: low-memory high-speed replacement for Picard MarkDuplicates
ThreadText: # low-memory high-speed replacement for Picard MarkDuplicates
  
  (Asked by: username_0 on 2017-12-11)
  
  I am running Picard MarkDuplicates with the following parameters below. On the file described, it takes about 41.6Gb of RAM memory and about 20-25 minutes to compute (only uses 1 core AFAICS).
  
  ```
  java -Xmx54890m -jar picard.jar MarkDuplicates \
      I=/home/dnanexus/in/sorted_bam/TST73-86-2IC_S16_L00.bml.GRCh38.karyo.bam \
      O=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.deduplicated.bam \
      M=./TST73-86-2IC_S16_L00.bml.GRCh38.karyo.duplication_metrics \
      CREATE_INDEX=true VALIDATION_STRINGENCY=SILENT REMOVE_DUPLICATES=true
  
  ```
  
  What equivalent free/open source software (samtools?, sambamba?) can I use to obtain the same deduplicated bam file but at a lower cost, considering the instance used here has 8 cores but only one of them is being used, and has about 68Gb of RAM out of which it's using 46Gb odd at peak.
  
  I think previous versions of samtools/sambamba didn't behave the same way as Picard MarkDuplicates with the parameters above, but I would like to know which recent versions would give me the same output.
  
  **Tags:** sam, samtools, sambamba, picard
  
  > **Comments:**
  > **username_1**: I assume you only want free (in terms of price, at least) software, right? [Sentieon](http://sentieon.com/) have a tool for this that is orders of magnitude faster (and I would guess uses less RAM but I'd have to check) but you need to pay for it.  
  > **username_0**: Correct, open source.  
  > **username_2**: I done no benchmark, but I guess [samblaster](https://github.com/GregoryFaust/samblaster) is bit faster (by my subjective perception of time).  
  > **username_3**: You should specify a smaller "-Xmx". MarkDuplicate doesn't need that much memory, but seeing a large "-Xmx", java will use extra.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2017-12-11. Score: 2
  
  As you suggested, sambamba is faster at marking duplicates than picard (it's also multithreaded). Recent versions of samtools have a rewritten duplicate marking algorithm, though I doubt it'll be as quick as sambamba. 46GB of RAM seems excessive for marking duplicates unless you're having it store the whole file in memory.
  
  > **Comments:**
  > **username_0**: Does sambamba with default parameters equate to Picard MarkDuplicates?  
  > **username_4**: I'm not sure that any tool exactly equates (there are a lot of edge cases in duplicate marking and picard misses some things), but it's a sufficient replacement. Sambamba is also used for this purpose by a number of international consortia (I think either DEEP or IHEC, if I remember correctly).  
  
  ---
  
  