I/O Access Methods for Linux
The diagram is a bit misleading. There is no system call (switching to the kernel) with read or write hits, assuming the memory is already mapped, has a page table entry and is resident. You only incur switches when performing the mapping and due to various page fault modes (lazy mapping, non-resident data, copy-on-write needed).<p><i>> This usually happens when the ratio of storage size to RAM size is significantly higher than 1:1. Every page that is brought into cache causes another page to be evicted.</i><p>While true one can optimize these things by unmapping larger ranges in bulk (to reduce cache pressure) or prefetching them (to reduce blocking) with madvise, allowing the kernel doing the loading asynchronously while you're accessing the previously prefetched ones.<p>If you know your read and write patterns very well you can effectively use this for nearly asynchronous IO without the pains of AIO and few to none extra threads.