title: `mmap()` manual concurrent prefaulting / paging
body: <p>I'm trying to fine tune <code>mmap()</code> to perform fast writes or reads (generally not both) of a potentially very large file. The writes and reads will be <em>mostly</em> sequential on one pass and then likely very sparse on future passes. No region of memory needs to be accessed more than once.</p>
  
  <p>In other words, think of it as a file transfer with some lossiness that gets fixed asynchronously.</p>
  
  <p>It appears, as expected, that the main limitation of <code>mmap()</code>'s performance seems to be the number of minor page faults it generates on large files. Furthermore, I suspect the laziness of the Linux kernel's page-to-disk is causing some performance issues. Namely, any test programs that end up performing huge writes to <code>mmap</code>ed memory seem to take a long time after performing all writes to terminate/<code>munmap</code> memory.</p>
  
  <p>I was hoping to offset the cost of these faults by concurrently prefaulting pages while performing the almost-sequential access and paging out pages that I won't need again. But I have three main questions regarding this approach and my understanding of the problem:</p>
  
  <ol>
  <li>Is there a straightforward (preferably POSIX [or at least OSX] compatible) way of performing a partial prefault? I am aware of the <code>MAP_POPULATE</code> flag, but this seems to attempt loading the entire file into memory, which is intolerable in many cases. Also, this seems to cause the <code>mmap()</code> call to block until prefaulting is complete, which is also intolerable.  My idea for a manual alternative was to spawn a thread simply to try reading the next  <code>N</code> pages in memory to force a prefetch. But it might be that <code>madvise</code> with <code>MADV_SEQUENTIAL</code> already does this, in effect. </li>
  <li><code>msync()</code> can be used to flush changes to the disk. However, is it actually useful to do this periodically? My idea is that it might be useful if the program is frequently in an "Idle" state of disk IO and can afford to squeeze in some disk writebacks. Then again, the kernel might very well be handling this itself better than the ever application could. </li>
  <li>Is my understanding of disk IO accurate? My assumption is that prefaulting and reading/writing pages can be done <em>concurrently</em> by different threads or processes; if I am wrong about this, then manual  prefaulting would not be useful at all. Similarly, if an <code>msync()</code> call blocks all disk IO, both to the filesystem cache and to the raw filesystem, then there also isn't as much of an incentive to use it over flushing the entire disk cache at the program's termination.</li>
  </ol>