Extreme Debugging
Some of this debugging is, per the title, "extreme" (trying to "rewind" the realities of source behavior through time until things start to click, dealing with ancient binaries, using disassembly/binary analysis tools to hack up branching logic).However, a lot of this is using tools that I'd expect everyone (even browser JS programmers, Java programmers, folks that write code in higher level languages) to have at least a passing familiarity with: strace/ltrace, nm/ldd, gdb...no matter who you are or what you do, there's immense benefit in trying to debug your programs like they're closed-source[1].I'm perennially disappointed to find that extremely skilled and intelligent colleagues (far better at writing and designing code than me) are routinely stymied when asked to figure out why their code is acting a certain way when it runs--why a behavior of their code is occurring in an environment where their typical tools (language-specific debuggers, IDEs, etc.) aren't available.I think this is a skill-gap that our iteration-focused, "just try shit until it works" industry has instilled in a generation of programmers to their detriment.This is super important stuff--not just for debugging ability, but because understanding at least the "shallow end" of runtime-analysis tools (and with eBPF and core analysis tools, to say nothing of actual embedded/kernel programming, the pool gets a lot deeper, if you want) makes you become a way better programmer!This isn't only for folks who write code in runtime-less languages (and don't come at me about crt0); even people who write Python for a living (like me!), get bloody superpowers from understanding things like kernel memory management, syscall semantics below the standard library, and garbage collection behavior. The folks I find that don't have those skills aren't stupid or talentless, but rather victims of a series of roles and managers that have instilled a kind of Stockholm Syndrome of "when these things fail, there's nothing you can do about it, just consider it spooky and move on with your life", when getting at what's really going on is not just easy to do, but super beneficial for your career development!Put another way, we're all systems programmers in our day jobs--some of us are just in denial./rant1. https://jvns.ca/blog/2014/04/20/debug-your-programs-like-the...