Delivering Billions of Messages Exactly Once
I don&#x27;t want to ever see the phrase &quot;Exactly Once&quot; without several asterisks behind it. It might be exactly once from an &quot;overall&quot; point of view, but the client effectively needs infinitely durable infinite memory to perform the &quot;distributed transaction&quot; of acting on the message and responding to the server.<p>Imagine:<p>- Server delivers message M<p>- Client process event E entailed by message M<p>- Client tries to ack (A) message on server, but &quot;packet loss&quot;<p>- To make matters worse, let&#x27;s say the client also immediately dies after this<p>How do you handle this situation?
The client must transactionally&#x2F;simultaneously commit both E and A&#x2F;intent-to-A. Since the server never received an acknowledgment of M, it will either redeliver the message, in which case some record of E must be kept to deduplicate on, or it will wait for client to resend A, or some mixture of both. Note: if you say &quot;just make E idempotent&quot;, then you don&#x27;t need exactly-once delivery in the first place...<p>I suppose you could go back to some kind of lock-step processing of messages to avoid needing to record all (E,A) that are in flight, but that would obviously kill throughput of the message queue.<p>Exactly Once can only ever be At Least Once with some out-of-the-box idempotency that may not be as cheap as the natural idempotency of your system.<p>EDIT: Recommended reading: &quot;Life Beyond Distributed Transactions&quot;, Pat Helland - <a href="http:&#x2F;&#x2F;queue.acm.org&#x2F;detail.cfm?id=3025012" rel="nofollow">http:&#x2F;&#x2F;queue.acm.org&#x2F;detail.cfm?id=3025012</a>