This disclosure relates generally to distributed stream computing, and more particularly, to exactly-once semantic processing in non-idempotent output operations.
In distributed stream computing, a pure function is a transformation of input data to output data in which the compute logic is side effect free. Side effects may include writing out a message to another distributed bus or writing a value to a key value store during a computation. Side effect free code allows a cluster manager or driver to run this code again on the same compute node or perhaps another compute node. However, practical solutions, in distributed stream computing, typically contain side effects and, therefore cannot always be based on pure functions.
A typical operation in a distributed stream computing system takes a period of time (e.g. a minute, a year, etc.) to complete. During this period, several faults or losses may occur within the distributed computing environment. In a first example, a fault may occur when there is a power failure in one or more compute nodes. In a second example, a fault may occur where a batch of compute nodes are considerably faster than another batch. A system may determine the faster batch of compute nodes completed running the operation while the slower batch of compute nodes are still running or waiting to run the operation, in which the system might perform a speculative execution on the faster compute nodes.
Generally, distributed data transformation frameworks provide fault-tolerance mechanisms to recover from compute node faults. These mechanisms can provide exactly-once semantics for the idempotent transformations, but some output operations performed after the transformations have at-least once semantics.
In a typical fault-tolerance mechanism, when a compute node fails within a cluster of compute nodes, the system may automatically reapply transformations, using a distributed key value data store or a distributed messaging system, to replicated data sets and compute the input partition again on either the same compute node or another compute node. However, this recovery mechanism results in replaying the input data, thereby duplicating the output data and corrupting the state of the application. As a result of the duplicated coordination messages, the system performs duplicate message processing and produces erroneous results.
For idempotent operations, at-least once semantics is sufficient, but there are scenarios in which the output operation might involve message sending or some other non-idempotent action. In such scenarios, an extension to the original fault-tolerance mechanism that avoids duplication of output operations is necessary to achieve exactly-once semantics.