What is proper workflow for insuring "transactional procedures" in case of exceptions
We use microservices in our company and at least once a month, we have one of our microservices down for a while. We have Transaction model for the payment process and statuses for every step that go before we send a product to the user. If something goes wrong or one of the connected microservices is down, we mark it like status=error and save to the database. Then we use cron job to find and finish those processes. You need to try something for the beginning and if does not fit your needs, try something else.
In programming web applications, Django in particular, sometimes we have a set of actions that must all succeed or all fail (in order to insure a predictable state of some sort). Now obviously, when we are working with the database, we can use transactions.
But in some circumstances, these (all or nothing) constraints are needed outside of a database context
(e.g. If payment is a success, we must send the product activation code or else risk customer complaints, etc)
But lets say on some fateful day, the send_code() function just failed time and again due to some temporary network error (that lasted for 1+ hours)
Should I log the error, and manually fix the problem, e.g. send the mail manually  
Should I set up some kind of work queue, where when things fail, they just go back onto the end of the queue for future retry?
What if the logging/queueing systems also fail? (am I worrying too much at this point?)