Answers.Body: <p>Assuming you are on an operating system that supports them, like Linux, you can try using <a href="https://en.wikipedia.org/wiki/Named_pipe" rel="nofollow noreferrer">named pipes</a> in a ramdisk or <a href="https://en.wikipedia.org/wiki/Tmpfs" rel="nofollow noreferrer">tmpfs</a>. Not the most elegant approach, but I don't know if what you are asking for is possible (it may very well be, I just don't know) and this can be a decent workaround.</p>
  <p>First, check that you do have a mounted tmpfs. On my system, I have a few:</p>
  <pre><code>$ mount | awk '$1==&quot;tmpfs&quot;'
  tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
  tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16278892k,nr_inodes=1048576,inode64)
  tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3255776k,nr_inodes=813944,mode=700,uid=1000,gid=1000,inode64)
  </code></pre>
  <p>These filesystems are actually stored in RAM and should be significantly faster. For instance, on my machine, which has an SSD as the main storage, I still get significant speedup when using <code>/dev/shm</code>:</p>
  <pre><code>## SSD
  $ time seq 1000000000 &gt; file
  real    0m13.687s
  user    0m5.623s
  sys     0m7.516s
  
  $ time wc -l &lt; file
  1000000000
  
  real    0m4.972s
  user    0m0.101s
  sys     0m2.248s
  
  ## tmpfs, RAM
  $ time seq 1000000000 &gt; /dev/shm/file
  
  real    0m6.755s
  user    0m4.982s
  sys     0m1.763s
  
  $ time wc -l &lt; /dev/shm/file
  1000000000
  
  real    0m1.024s
  user    0m0.120s
  sys     0m0.902s
  </code></pre>
  <p>As you can see, using <code>/dev/shm</code> is significantly faster. And using named pipes can give you a little boost too:</p>
  <pre><code>$ time ( seq 1000000000 &gt; /dev/shm/file;  wc -l &lt; /dev/shm/file)
  1000000000
  
  real    0m8.341s
  user    0m5.277s
  sys     0m3.044s
  
  <span class="math-container">$ rm /dev/shm/file &amp;&amp; mkfifo /dev/shm/file 
  $</span> time ( seq 1000000000 &gt; /dev/shm/file &amp;  wc -l &lt; /dev/shm/file)
  1000000000
  
  real    0m6.564s
  user    0m5.426s
  sys     0m3.965s
  </code></pre>
  <p>So, this is essentially a way of doing everything in RAM without needing to code it as doing everything in RAM. You can create these files (the FIFOs, the named pipes) and treat them as files, but since you're creating them in a <code>tmpfs</code> filesystem, they're actually in RAM and by using FIFOs you not only get a further speed jump but also minimise the actual data that need to be stored.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-16T00:35:57.857Z
Answers.Text: This seems like it gets at what I'm trying to accomplish. I will try it! Thank you.
Answers.UserDisplayName: ekofman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-14T13:33:52.703Z
Answers.LastActivityDate: 2023-10-16T09:20:47.210Z
Answers.LastEditDate: 2023-10-16T09:20:47.210Z
Answers.UserDisplayName: terdon
Body: <p>My current job is rate-limited chiefly by the IO overhead of writing reads to their corresponding bam files (they are split up by both cell barcode and contig). The only reason I need to write these mini-split bams is so that I can then use the pysam coverage function to assess read depth at certain locations. I already have all of these reads in memory as a list of AlignedSegments. Is it necessary to write them to a .bam file in order to be able to index and use the coverage function? I feel like there should be a way to have the AlignedSegments be treated as a bam object in memory, possibly even indexed in memory, and then treated as if they were just streamed from a bamfile with an index in order to be able to use the coverage method. Thanks in advance, if this is possible it would increase my processing speed x10. Or, given a list of AlignedSegments, is there a way to get coverage at specific positions across all those AlignedSegments in my list (that would get me what I need).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-14T02:05:31.577Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-14T13:21:20.633Z
Comments.Text: What operating system are you using?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-16T00:36:42.953Z
Comments.Text: My operating system is linux
Comments.UserDisplayName: ekofman
Tags: <coverage><pysam><indexing>
Title: Is it possible to, all in memory without writing a file, given a list of AlignedSegments and a bam header, build a bam, index and get coverage?
ThreadText: # Is it possible to, all in memory without writing a file, given a list of AlignedSegments and a bam header, build a bam, index and get coverage?
  
  (Asked by: username_0 on 2023-10-14)
  
  My current job is rate-limited chiefly by the IO overhead of writing reads to their corresponding bam files (they are split up by both cell barcode and contig). The only reason I need to write these mini-split bams is so that I can then use the pysam coverage function to assess read depth at certain locations. I already have all of these reads in memory as a list of AlignedSegments. Is it necessary to write them to a .bam file in order to be able to index and use the coverage function? I feel like there should be a way to have the AlignedSegments be treated as a bam object in memory, possibly even indexed in memory, and then treated as if they were just streamed from a bamfile with an index in order to be able to use the coverage method. Thanks in advance, if this is possible it would increase my processing speed x10. Or, given a list of AlignedSegments, is there a way to get coverage at specific positions across all those AlignedSegments in my list (that would get me what I need).
  
  **Tags:** coverage, pysam, indexing
  
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_2**: What operating system are you using?  
  > **username_0**: My operating system is linux  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-14. Score: 3 | Accepted answer
  
  Assuming you are on an operating system that supports them, like Linux, you can try using [named pipes](https://en.wikipedia.org/wiki/Named_pipe) in a ramdisk or [tmpfs](https://en.wikipedia.org/wiki/Tmpfs). Not the most elegant approach, but I don't know if what you are asking for is possible (it may very well be, I just don't know) and this can be a decent workaround.
  
  First, check that you do have a mounted tmpfs. On my system, I have a few:
  
  ```
  $ mount | awk '$1=="tmpfs"'
  tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
  tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16278892k,nr_inodes=1048576,inode64)
  tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3255776k,nr_inodes=813944,mode=700,uid=1000,gid=1000,inode64)
  
  ```
  
  These filesystems are actually stored in RAM and should be significantly faster. For instance, on my machine, which has an SSD as the main storage, I still get significant speedup when using `/dev/shm`:
  
  ```
  ## SSD
  $ time seq 1000000000 > file
  real    0m13.687s
  user    0m5.623s
  sys     0m7.516s
  
  $ time wc -l < file
  1000000000
  
  real    0m4.972s
  user    0m0.101s
  sys     0m2.248s
  
  ## tmpfs, RAM
  $ time seq 1000000000 > /dev/shm/file
  
  real    0m6.755s
  user    0m4.982s
  sys     0m1.763s
  
  $ time wc -l < /dev/shm/file
  1000000000
  
  real    0m1.024s
  user    0m0.120s
  sys     0m0.902s
  
  ```
  
  As you can see, using `/dev/shm` is significantly faster. And using named pipes can give you a little boost too:
  
  ```
  $ time ( seq 1000000000 > /dev/shm/file;  wc -l < /dev/shm/file)
  1000000000
  
  real    0m8.341s
  user    0m5.277s
  sys     0m3.044s
  
  $ rm /dev/shm/file && mkfifo /dev/shm/file 
  $ time ( seq 1000000000 > /dev/shm/file &  wc -l < /dev/shm/file)
  1000000000
  
  real    0m6.564s
  user    0m5.426s
  sys     0m3.965s
  
  ```
  
  So, this is essentially a way of doing everything in RAM without needing to code it as doing everything in RAM. You can create these files (the FIFOs, the named pipes) and treat them as files, but since you're creating them in a `tmpfs` filesystem, they're actually in RAM and by using FIFOs you not only get a further speed jump but also minimise the actual data that need to be stored.
  
  > **Comments:**
  > **username_0**: This seems like it gets at what I'm trying to accomplish. I will try it! Thank you.  
  
  ---
  
  