Ask HN: How do you make sure your servers are up as a single founder?
Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn&#x27;t actually solve the issue I hit last night...<p>My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn&#x27;t able to handle, because I didn&#x27;t expect it so never thought to test for it -- the issue was that I was trying to insert IDs into a uuid database field, but this new data had non-uuid IDs. Because the original IDs were always generated by me, I was able to guarantee that the data was correct, but this new data was not generated by me. Of course, sufficient defensive programming would have avoided this as this database error shouldn&#x27;t have prevented other stuff from working, but my point is that mistakes get made (we&#x27;re humans after all) and things do get overlooked.<p>The problem is, restarting my service doesn&#x27;t prevent this external data from getting received again, so it would simply break again as soon as more is sent and the system would be in this endless reboot loop until a human fixes the root cause.<p>That&#x27;s a problem that I worry about, no matter how hard I try to make my system auto-healing and resilient (I don&#x27;t know of any way to fix it other than putting great care into programming defensively), but again, we&#x27;re human, so something will always slip through eventually...<p>Some people are suggesting to out-source an on-call person. That seems to me like the only way around this particular case. (The other suggestions can still be used to reduce the amount of times this person gets paged, though)