Turtles on the Wire: Understanding How the OS Uses the Modern NIC (2016)
This is why I&#x27;m really hoping RDMA [1] will catch on soon. It would be great if there was a cloud provider that would enable this feature on some of their offerings. Amazon has done something similar by allowing kernel bypass via DPDK [2] with their ENA offering but Kernel bypass is inferior to RDMA in so many ways IMO.<p>At this point we have 200Gbit&#x2F;s NICs being provided by Mellanox [3]. CPUs aren&#x27;t getting any faster and the scale out approach is extremely difficult to get right without going across NUMA domains [4]. Based on the progression of CPUs lately there just isn&#x27;t going to be enough time to process all these packets AND have time left over to actually run your application. There&#x27;s a lot of work focusing on data locality at the moment but at this point it&#x27;s still not fool proof and the work that has been done is woefully under documented.<p>As the article mentioned we&#x27;ve already added a bunch of hardware offloads. RDMA is just a continuation of these offloads but unfortunately it requires some minor changes on the application side to take advantage of which is why it&#x27;s probably been slow to be adopted.<p>RDMA has so many great applications for data-transfer for backend services. Whether it&#x27;s queries between a web-server and a DB, replication&#x2F;clustering of DBs, or micro-service fabric with micro seconds latency. Overall there&#x27;s a lot of low hanging fruit that could be optimized with RDMA.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Remote_direct_memory_access" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Remote_direct_memory_access</a><p>[2] <a href="http:&#x2F;&#x2F;dpdk.org&#x2F;doc&#x2F;nics" rel="nofollow">http:&#x2F;&#x2F;dpdk.org&#x2F;doc&#x2F;nics</a><p>[3] <a href="http:&#x2F;&#x2F;www.mellanox.com&#x2F;page&#x2F;products_dyn?product_family=266&amp;mtag=connectx_6_en_card" rel="nofollow">http:&#x2F;&#x2F;www.mellanox.com&#x2F;page&#x2F;products_dyn?product_family=266...</a><p>[4] <a href="http:&#x2F;&#x2F;rhelblog.redhat.com&#x2F;2015&#x2F;09&#x2F;29&#x2F;pushing-the-limits-of-kernel-networking&#x2F;" rel="nofollow">http:&#x2F;&#x2F;rhelblog.redhat.com&#x2F;2015&#x2F;09&#x2F;29&#x2F;pushing-the-limits-of-...</a>