Answers.Body: <p>Another library you can use for this purpose is the <a href="https://github.com/samtools/htsjdk" rel="nofollow noreferrer">htsjdk</a>, which is written in java. Using htsjdk with java is analogous to using htslib with C; the BAM format is already handled by the library and you can manipulate fields in your own code. The same basic analysis applies to java as C; you do not need to convert the BAM file to text and then parse it. Generally, a program implemented in java will be slower than C but still significantly faster than scripting. You do get language features of java: portability, array bounds checking, garbage collection, etc. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-25T14:27:47.847Z
Answers.LastActivityDate: 2017-08-25T14:27:47.847Z
Answers.UserDisplayName: mattm
Answers.Body: <p>I modified your original question: as you are extracting 4 fields, you are not outputting BAM. The answer to the modified question is: yes, you can write a C program with htslib (or with bamtools, bioD, bioGo or rust-bio). Formatting an entire SAM is fairly expensive. You can see this by comparing <code>samtools view aln.bam &gt; /dev/null</code> and <code>samtools view -u aln.bam &gt; /dev/null</code>. With a C program, you can select fields to output. This will give you a noticeable performance boost, depending on the fields you extract.</p>
  
  <p>In general, if you really care about performance, don't use script. The time you save from optimizing <code>samtools view|cut -f</code> will be quickly neutralized by the inefficiency of most scripting languages.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-29T10:05:14.110Z
Answers.Text: > With a C program, you can select fields to output.
  
  Can you detail a bit more how this would be done? The BAM format is in binary, so one would have to convert to text first and then extract the fields desired?
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-29T15:36:51.683Z
Answers.Text: @ShanZhengYang The library parses the file record for you and places it in memory. From there, you print (or otherwise process) from the in-memory data only your desired fields in whatever format you want. There is no text parsing required.
Answers.UserDisplayName: mattm
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-25T12:17:12.407Z
Answers.LastActivityDate: 2017-08-25T12:17:12.407Z
Answers.UserDisplayName: user172818
Answers.Body: <p>The BAM file format is not a text-based format. It has a specific binary structure, specified in reasonable detail in the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="noreferrer">SAM file format specification</a>. Whenever this information is displayed on a screen as text, it needs to be converted from the binary format to a text format, which takes a bit of time and processing power.</p>
  
  <p>As this question suggests, if only a specific field from alignments is needed (or fields), then it will probably be better to extract just those fields and do any necessary conversion only on those fields. While this <em>can</em> be done by writing a BAM parser from scratch, many developers have already written software libraries to process BAM files in this way.</p>
  
  <p>Devon Ryan has suggested <a href="http://www.htslib.org/" rel="noreferrer">htslib</a>, which is a C library written by a group that includes the people who wrote the SAM/BAM/CRAM file format specifications. There's also <a href="https://github.com/pysam-developers/pysam" rel="noreferrer">pysam</a>, which is a python wrapper around htslib.</p>
  
  <p>The particular tool that is used will depend on your familiarity with programming and the specific thing that you want to do. If you just want to "output a BAM file", then cat is the quickest:</p>
  
  <pre><code>cat file1.bam
  </code></pre>
  
  <p>... but you probably don't want to do that, because it seems like you want to process a text-based representation with a script. Because you haven't specified which fields you are interested in, it's not possible to suggest the best thing to use. In the end, I expect that an "efficient" solution to your problem would involve <code>htslib</code> in some form. This is not really hacking <em>samtools</em>, it's using the backend of samtools to process BAM data.</p>
  
  <p>However, it's useful to also think about the time cost of coding. What application do you have which means that the text-based processing method is not sufficient? It takes time to write code, and a lot more time to debug that code to make sure it's doing the right thing in all situations. If this is for a one-off thing, then <code>samtools view</code> output fed into your script may be the quickest solution.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T04:02:43.557Z
Answers.Text: I'm a bit familiar with pysam. It seems like this would be very inefficient, as I would need to input the bam into a python data structure, e.g. ` pysam.AlignmentFile()`. As for the script, yes, I currently am parsing the text-format of the outputted bam file. I have done nothing with compression/binary yet.
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T21:49:41.357Z
Answers.Text: Whether or not it is inefficient depends on the way the library is implemented. It's possible for data structures to contain pointers to files (or places within files) and only retrieve information from the files when requested. There's no need to load the entire file into memory to explore just one aspect of it.
Answers.UserDisplayName: gringer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-23T03:14:30.797Z
Answers.LastActivityDate: 2017-08-23T03:14:30.797Z
Answers.UserDisplayName: gringer
Body: <p>I am constructing a bit of software which pipes the outputs of the bam file via <code>samtools view</code> into a script for parsing. My goal is to (somehow) make this process more efficient, and faster than <code>samtools view</code>. </p>
  
  <p>I am only using 3-4 fields in the bam. Naturally, if I only wanted those fields, I could do something like <code>samtools view file1.bam | cut -f ##</code>. </p>
  
  <p>However, I'm trying to be faster than this approach. More specifically: </p>
  
  <p>(1) Given the bam file format, is it theoretically possible to output several requested SAM fields faster than <code>samtools view|cut -f</code>? </p>
  
  <p>(2) Is there software available to achieve (1), or would one need to hack samtools? </p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-22T21:41:36.073Z
Comments.Text: Which fields do you need exactly? Using `htslib` is relatively easy if you know C, but if you can get by with something like [sambamba](http://lomereiter.github.io/sambamba/) then that'd be vastly easier still.
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-22T22:22:07.930Z
Comments.Text: @DevonRyan Outputting the bam could be faster with sambambda? These are "mandatory" fields, so they're present in each bam "row"
Comments.UserDisplayName: ShanZhengYang
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-22T22:39:07.903Z
Comments.Text: `samtools view`, especially in the way it is used here, produces a text representation of a BAM file. Using a library (or tool that taps into a library) allows faster access at the binary level by avoiding full text conversion of each alignment prior to filtering.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-22T22:52:41.050Z
Comments.Text: @gringer I'm not following this exactly. Could you provide more details/an explanation?
Comments.UserDisplayName: ShanZhengYang
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-23T04:53:07.897Z
Comments.Text: Sambamba is generally faster than samtools, or at least that used to be the case. Samtools is a reference implementation, it doesn't always have to be the fastest.
Comments.UserDisplayName: Devon Ryan
Tags: <sam><samtools>
Title: Is there a way to retrieve several SAM fields faster than `samtools view | cut -f`?
ThreadText: # Is there a way to retrieve several SAM fields faster than `samtools view | cut -f`?
  
  (Asked by: username_0 on 2017-08-22)
  
  I am constructing a bit of software which pipes the outputs of the bam file via `samtools view` into a script for parsing. My goal is to (somehow) make this process more efficient, and faster than `samtools view`.
  
  I am only using 3-4 fields in the bam. Naturally, if I only wanted those fields, I could do something like `samtools view file1.bam | cut -f ##`.
  
  However, I'm trying to be faster than this approach. More specifically:
  
  (1) Given the bam file format, is it theoretically possible to output several requested SAM fields faster than `samtools view|cut -f`?
  
  (2) Is there software available to achieve (1), or would one need to hack samtools?
  
  **Tags:** sam, samtools
  
  > **Comments:**
  > **username_1**: Which fields do you need exactly? Using `htslib` is relatively easy if you know C, but if you can get by with something like [sambamba](http://lomereiter.github.io/sambamba/) then that'd be vastly easier still.  
  > **username_0**: @username_1 Outputting the bam could be faster with sambambda? These are "mandatory" fields, so they're present in each bam "row"  
  > **username_2**: `samtools view`, especially in the way it is used here, produces a text representation of a BAM file. Using a library (or tool that taps into a library) allows faster access at the binary level by avoiding full text conversion of each alignment prior to filtering.  
  > **username_0**: @username_2 I'm not following this exactly. Could you provide more details/an explanation?  
  > **username_1**: Sambamba is generally faster than samtools, or at least that used to be the case. Samtools is a reference implementation, it doesn't always have to be the fastest.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2017-08-23. Score: 6 | Accepted answer
  
  The BAM file format is not a text-based format. It has a specific binary structure, specified in reasonable detail in the [SAM file format specification](https://samtools.github.io/hts-specs/SAMv1.pdf). Whenever this information is displayed on a screen as text, it needs to be converted from the binary format to a text format, which takes a bit of time and processing power.
  
  As this question suggests, if only a specific field from alignments is needed (or fields), then it will probably be better to extract just those fields and do any necessary conversion only on those fields. While this *can* be done by writing a BAM parser from scratch, many developers have already written software libraries to process BAM files in this way.
  
  username_1 username_1 has suggested [htslib](http://www.htslib.org/), which is a C library written by a group that includes the people who wrote the SAM/BAM/CRAM file format specifications. There's also [pysam](https://github.com/pysam-developers/pysam), which is a python wrapper around htslib.
  
  The particular tool that is used will depend on your familiarity with programming and the specific thing that you want to do. If you just want to "output a BAM file", then cat is the quickest:
  
  ```
  cat file1.bam
  
  ```
  
  ... but you probably don't want to do that, because it seems like you want to process a text-based representation with a script. Because you haven't specified which fields you are interested in, it's not possible to suggest the best thing to use. In the end, I expect that an "efficient" solution to your problem would involve `htslib` in some form. This is not really hacking *samtools*, it's using the backend of samtools to process BAM data.
  
  However, it's useful to also think about the time cost of coding. What application do you have which means that the text-based processing method is not sufficient? It takes time to write code, and a lot more time to debug that code to make sure it's doing the right thing in all situations. If this is for a one-off thing, then `samtools view` output fed into your script may be the quickest solution.
  
  > **Comments:**
  > **username_0**: I'm a bit familiar with pysam. It seems like this would be very inefficient, as I would need to input the bam into a python data structure, e.g. ` pysam.AlignmentFile()`. As for the script, yes, I currently am parsing the text-format of the outputted bam file. I have done nothing with compression/binary yet.  
  > **username_2**: Whether or not it is inefficient depends on the way the library is implemented. It's possible for data structures to contain pointers to files (or places within files) and only retrieve information from the files when requested. There's no need to load the entire file into memory to explore just one aspect of it.  
  
  ---
  
  ### Answer by username_4 on 2017-08-25. Score: 6
  
  I modified your original question: as you are extracting 4 fields, you are not outputting BAM. The answer to the modified question is: yes, you can write a C program with htslib (or with bamtools, bioD, bioGo or rust-bio). Formatting an entire SAM is fairly expensive. You can see this by comparing `samtools view aln.bam > /dev/null` and `samtools view -u aln.bam > /dev/null`. With a C program, you can select fields to output. This will give you a noticeable performance boost, depending on the fields you extract.
  
  In general, if you really care about performance, don't use script. The time you save from optimizing `samtools view|cut -f` will be quickly neutralized by the inefficiency of most scripting languages.
  
  > **Comments:**
  > **username_0**: > With a C program, you can select fields to output.  Can you detail a bit more how this would be done? The BAM format is in binary, so one would have to convert to text first and then extract the fields desired?  
  > **username_3**: @username_0 The library parses the file record for you and places it in memory. From there, you print (or otherwise process) from the in-memory data only your desired fields in whatever format you want. There is no text parsing required.  
  
  ---
  
  ### Answer by username_3 on 2017-08-25. Score: 3
  
  Another library you can use for this purpose is the [htsjdk](https://github.com/samtools/htsjdk), which is written in java. Using htsjdk with java is analogous to using htslib with C; the BAM format is already handled by the library and you can manipulate fields in your own code. The same basic analysis applies to java as C; you do not need to convert the BAM file to text and then parse it. Generally, a program implemented in java will be slower than C but still significantly faster than scripting. You do get language features of java: portability, array bounds checking, garbage collection, etc.
  
  
  ---
  
  