Ask HN: What should a systems/low-level software engineer know?
A systems programmer will specialize in subdomains much like any other programmer, but there are some characteristic skills and knowledge that are common across most of them. Systems programmer coding style is driven by robustness, correctness, and performance to a much greater degree than higher up the stack. Most systems programming jobs these days is C&#x2F;C++ on Linux and will be for the foreseeable future.<p>- Know how your hardware actually works, especially CPU, storage, and network. This provides the &quot;first principles&quot; whence all of the below are derived and will allow you to reason about and predict system behavior without writing a single line of code.<p>- Understand the design of the operating system well enough to reimplement and bypass key parts of it as needed. The requirements that would cause you to to build, for example, a custom storage engine also means you aren&#x27;t mmap()-ing files, instead doing direct I&#x2F;O with io_submit() on a file or raw block device into a cache and I&#x2F;O scheduler you designed and wrote. Study the internals of existing systems code for examples of how things like this are done, it is esoteric but not difficult to learn.<p>- Locality-driven software design. Locality maximization, both spatial and temporal, is the source of most performance in modern software. In systems programming, this means you are always aware of what is currently in your hardware caches and efficiently using that resource. As a corollary, <i>compactness</i> is consistently a key objective of your data structures to a much greater extent than people think about it higher up the stack. One way you can identify code written by systems programmer is the use of data structures packed into bitfields.<p>- Understand the difference between interrupt-driven and schedule-driven computing models, the appropriate use cases for both, and how to safely design code that necessarily mixes both e.g. multithreading and coroutines. This is central to I&#x2F;O handling: network is interrupt-driven and disk is schedule-driven. Being explicit about where the boundaries are between these modes in your software designs greatly simplifies reasoning about concurrency. Most common concurrency primitives make assumptions about which model you are using.<p>- All systems are distributed systems. Part of the systems programmers job is creating the illusion that this is not the case to the higher levels in the stack, but a systems programmer unavoidably lives in this world even within a single server. Knowing &quot;latencies every programmer should know&quot; is just a starting point, it is also helpful to understand how hardware topology interacts with the routing&#x2F;messaging patterns to change latency -- tail latencies are more important than median latencies.<p>The above is relatively abstract and conceptual but generalizes to all systems programming. Domains of systems programming have deep and unique knowledge that is specific to the specialty e.g. network protocol stacks, database engines, high performance graphics, etc. Because the physics of hardware never changes except in the details, the abstractions are relatively thin, and the tool chains are necessarily conservative, systems programming skills age very well. The emergence of modern C++ (e.g. C++17) has made systems programming quite enjoyable.<p>Also, the best way to learn idiomatic systems programming is to read many examples of high-quality systems code. You will see many excellent techniques in the code that you&#x27;ve never seen before that are not documented in book&#x2F;paper form. I&#x27;ve been doing systems work for two decades and I <i>still</i> run across interesting new idioms and techniques.