I write code that is executed in rad environments.<p>The fist thing I learned is that any plan that I come up with to help mitigate risks always ends up being proven less effective than I initially thought. The idea that the hardware can flip bits or latch up behind your back is rarely a consideration for most software design work, so it really requires an entire new way of thinking. In the end, it&#x27;s all about reducing probabilities, but sometimes your mitigation techniques can be borderline useless. This could be because of my relative lack of experience though. I&#x27;m sure there are people who have been doing this a while and have converged on a design process that works well.<p>Couple this with the fact that code in the failure path very rarely gets actually tested with _real_ failures during design, and you end up with situations where your entire test&#x2F;precaution&#x2F;recovery technique is rendered useless by a single oversight.<p>It kind of reminds me of how some bootloaders perform a DRAM test during power up. The only problem is that these tests are usually executing after the bootloader is already running from RAM itself. Sure, it&#x27;s better than nothing and may help to find some obscure errors like a bad address line pin or something, but the issue here should be obvious - if there was a real problem with the DRAM, it&#x27;s likely you wouldn&#x27;t even get to that test, so now what?<p>Another example: suppose you have a PI bus connecting to an FPGA, and you are relying on the PI_done line to pace transfers. Now, a common test to verify the bus is working would be to read some magic word from the FPGA or maybe use an XOR test register, but what happens when there&#x27;s an actual failure like a bad connection to the bus or bad power to the FPGA? Well, the CPU throws an exception and resets because the timeout for the PI_done expires. This all happens before your test has a change to even print anything to the console.<p>I run into these kinds of issues all the time when designing software routines to self-test the hardware on our devices.<p>Everything is all good when the hardware works, but as soon as hardware starts actually failing, all of those diagnostic tests are suddenly useless.<p>I&#x27;ve gotten kind of off topic here, but the thought process is what I&#x27;m getting at.<p>It usually goes a little something like this:<p>Possible solution: Always have a backup OS image.
Caveat: There&#x27;s only one copy of the bootloader, so if that area of the flash gets screwed, we&#x27;re done.<p>Possible solution: Scrub the contents of RAM to check for errors.
Caveat: The scrub routine is running from RAM.<p>Possible solution: Use a watchdog timer to force a reset if the CPU runs away.
Caveat: What if the watchdog timer latches up and the CPU hard locks? (Unlikely, I know).<p>In the end, you can do a lot in SW to make a system more rubust, but the ultimate solution is probably on the mechanical side. You&#x27;re better off spending your efforts modifying the enclosure and submerging the thing under water than you are spending hours designing in software techniques to try to recover from HW errors.<p>TLDR: Most people agree that software techniques for mitigating risks in a rad environment are all about reducing probabilities, but I feel that such techniques have a good chance of being less effective in practice than people may think. I don&#x27;t know what the point of this comment really was though, other than being negative. :)