text:
\section{Introduction\label{sec:intro}}

Event-driven architectures are broadly used to build systems that react to events in the real world.  They include smart home systems, GUIs, mobile applications, and servers.
For example, in the context of smart home systems, event-driven systems include Samsung SmartThings~\cite{smartthings}, Android Things~\cite{androidthings}, Openhab~\cite{openhab}, and If This Then That (IFTTT)~\cite{ifttt}.

Event-driven architectures can have analogs of the concurrency bugs that are known to be problematic in multithreaded programming.  Subtle programming errors involving the ordering of events can easily cause event-driven programs to fail.  These failures can be challenging to find during testing as exposing these failures may require a specific set of events to occur in a specific order.
Model-checking tools can be helpful for finding subtle concurrency bugs or understanding complex interactions between different applications~\cite{trimanafse20}.  In recent years, significant work has been expended on developing model checkers for multithreaded concurrency~\cite{odpor,statelessmcr,jensen2015stateless,dporlocks,pldipsotso,inspect1,inspect2,inspect4}, but event-driven systems have received much less attention~\cite{jensen2015stateless,maiya2016por}.

Event-driven systems pose several challenges for existing \emph{stateless} and \emph{stateful} model-checking tools.  Stateless model checking of concurrent applications explores all execution schedules without checking whether these schedules visit the same program states.  Stateless model checking often uses dynamic partial order reduction (DPOR) to eliminate equivalent schedules. \mycomment{,\hbox{\emph{i.e.},}\xspace schedules that rearrange independent events}
While there has been much work on DPOR for stateless model checking of multithreaded programs~\cite{flanagan2005dynamic,odpor,dporlocks,pldipsotso,statelessmcr}, stateless model checking requires that the program under test terminates for fair schedules.  Event-driven systems are often intended to run continuously and may not terminate.  To handle non-termination, stateless model checkers require hacks such as bounding the length of executions to verify event-driven systems.

Stateful model checking keeps track of an application's states and avoids revisiting the same application states.  It is less common for stateful model checkers to use dynamic partial order reduction to eliminate equivalent executions.
Researchers have done much work on stateful model checking~\cite{jpf,spin,musuvathi2002cmc,gueta2007cartesian}.  While stateful model checking can handle non-terminating programs, they  miss an opportunity to efficiently reason about conflicting transitions to scale to large programs.  In particular, typical event-driven programs such as smart home applications have several event handlers that are completely independent of each other.  Stateful model checking enumerates different orderings of these event handlers, overlooking the fact that these handlers are independent of each other and hence the orderings are equivalent.

Stateful model checking and dynamic partial order reduction discover
different types of redundancy, and therefore it is beneficial to combine them to further improve model-checking scalability and efficiency.  For example, we have observed that some smart home systems have
several independent event handlers \mycomment{(\hbox{\emph{e.g.},}\xspace monitoring applications or
just independent systems)}in our experiments, and stateful model checkers can waste
an enormous amount of time exploring different orderings of these
independent transitions.  DPOR can substantially reduce the number of
states and transitions that must be explored.  Although work has been done to
combine DPOR algorithms with stateful model
checking~\cite{yang2008efficient,yi2006stateful} in the context of
multithreaded programs, this line of work requires that the application has
an \emph{acyclic state space}, \hbox{\emph{i.e.},}\xspace it terminates under all schedules.  In particular, the
approach of Yang \hbox{\emph{et al.}}\xspace\cite{yang2008efficient} is designed explicitly for
programs with acyclic state space and thus
cannot check programs that do not terminate.
Yi \hbox{\emph{et al.}}\xspace\cite{yi2006stateful} presents a DPOR algorithm for stateful model
checking, which is, however, incorrect for cyclic state spaces. For
instance, their algorithm fails to produce the asserting execution in
the example we will discuss shortly in
Figure~\ref{fig:example-non-terminating}.  As a result, prior DPOR
techniques all fall short for checking event-driven programs such as
smart home apps, that, in general, do not terminate.

\myparagraph{Our Contributions}
In this work, we present a stateful model checking technique for
event-driven programs that may not terminate. Such programs have cyclic state spaces, and existing algorithms can prematurely terminate an execution and thus fail to set
the necessary backtracking points to fully explore a program's
state space. Our \textbf{first} technical contribution is the \emph{formulation of a
sufficient condition to complete an execution of the application
that ensures that our algorithm fully explores the application's
state space}.

In addition to the early termination issue, for
programs with cyclic state spaces, a model checker can discover
multiple paths to a state \State{} before it explores the entire state
space that is reachable from state \State{}.  In this case, the
backtracking algorithms used by traditional DPOR techniques including
Yang \hbox{\emph{et al.}}\xspace\cite{yang2008efficient} can fail to set the necessary
backtracking points.  Our \textbf{second} technical contribution is \emph{a graph-traversal-based algorithm to appropriately set backtracking points on all paths that can reach the current state}.

Prior work on stateful DPOR only considers the multithreaded case
and assumes algorithms know the effects of the next transitions
of all threads before setting backtracking points.   For multithreaded programs, this assumption is \emph{not} a serious limitation as transitions model low-level memory operations (\hbox{\emph{i.e.},}\xspace reads, writes, and RMW operations), and each transition involves a \emph{single} memory operation.
However, in the context of event-driven programs,  events can involve
many memory operations that access multiple memory locations,
and knowing the effects of a transition requires actually
executing the event.  While it is conceptually possible to execute events and then rollback to discover their effects, this approach is likely to incur large overheads as model checkers need to know the effects of enabled events at each program state. As our \textbf{third} contribution, \emph{our algorithm avoids this extra rollback overhead by waiting until an event is actually
executed to set backtracking points and incorporates a modified backtracking
algorithm to appropriately handle events}.

\rahmadi{We have implemented the proposed algorithm in the Java Pathfinder model checker~\cite{jpf} and evaluated it on hundreds of real-world smart home apps. We have made our DPOR implementation publicly available~\cite{iotcheck-dpor-software}.}

\myparagraph{Paper Structure}
The remainder of this paper is structured as follows:
Section~\ref{sec:concurrency} presents the event-driven concurrency model that we use in this work.
Section~\ref{sec:definition} presents the definitions we use to describe our stateful DPOR algorithm.
Section~\ref{sec:basicideas} presents problems when using the classic DPOR algorithm to model check event-driven programs and the basic ideas behind how our algorithm solves these problems.
Section~\ref{sec:algorithm} presents our stateful DPOR algorithm for event-driven programs.
Section~\ref{sec:impeval} presents the evaluation of our algorithm
implementation on hundreds of smart home apps.
Section~\ref{sec:related} presents the related work; we conclude in
Section~\ref{sec:conclusion}.

\section{Related Work\label{sec:related}}

There has been much work on model checking.  Stateless model checking
techniques do not explicitly track which program states have been
visited and instead focus on enumerating
schedules~\cite{godefroidbook,godefroid1997model,godefroid2005software,musuvathi2007chess}.

To make model checking more efficient, researchers have
also looked into various partial order reduction techniques.
The original partial order reduction techniques (\hbox{\emph{e.g.},}\xspace persistent/stubborn sets~\cite{godefroidbook,stubbornsets} and sleep sets~\cite{godefroidbook}) can also be used in the context of cyclic state spaces when combined with a proviso that ensures that executions are not prematurely terminated~\cite{godefroidbook}, and ample sets~\cite{clarke1999model,clarke1999state} that are basically persistent sets with additional conditions. However, the persistent/stubborn set techniques ``suffer from severe fundamental limitation''\cite{flanagan2005dynamic}: the operations and their communication objects in future process executions are difficult or impossible to compute precisely through static analysis, while sleep sets alone only reduce the number of transitions (not states).  Work on \emph{collapses} by Katz and Peled also suffers from the same requirement for a statically known independence relation~\cite{collapses}.

The first DPOR technique was proposed by Flanagan and Godefroid~\cite{flanagan2005dynamic} to address those issues.
The authors introduced a
technique that combats the state space explosion by detecting
\emph{read-write} and \emph{write-write} conflicts on shared variable
on the fly.
Since then, a significant effort has been made to further improve
dynamic partial order reduction~\cite{sen2006automated,sen2006race,lauterburg2010evaluating,saarikivi2012improving,tasharofi2012transdpor}.
Unfortunately, a lot of DPOR algorithms assume the context of shared-memory concurrency in that each transition consists of a single memory operation.
In the context of event-driven applications, each transition is an event that can consist of different memory operations. Thus, we have to execute the event to know its effects and analyze it dynamically on the fly in our DPOR algorithm
(see Section~\ref{sec:events-as-transitions}).

\mycomment{
There has been much work on model checking.  Stateless model checking
techniques do not explicitly track which program states have been
visited and instead focus on enumerating
schedules~\cite{godefroidbook,godefroid1997model,godefroid2005software,musuvathi2007chess}.
To make model checking more efficient, researchers have
also looked into different DPOR techniques.
The first DPOR technique was proposed by Flanagan and Godefroid~\cite{flanagan2005dynamic}.
The authors introduced a
technique that combats the state space explosion by detecting
\emph{read-write} and \emph{write-write} conflicts on shared variable
on the fly.
This technique is based on the notion of independence between commutative transitions that borrows a similar concept from \emph{collapses} introduced in~\cite{katz1992defining}.
Since then, a significant effort has been made to further improve
dynamic partial order reduction~\cite{sen2006automated,sen2006race,lauterburg2010evaluating,saarikivi2012improving,tasharofi2012transdpor}.
}

Optimal DPOR~\cite{odpor} seeks to
make stateless model checking more efficient by skipping equivalent
executions.  Maximal causality reduction~\cite{statelessmcr} further
refines the technique with the insight that it is only necessary to
explore executions in which threads read different values.
Value-centric DPOR~\cite{chatterjee2019valuecentricdpor} has the insight that executions are equivalent if all of their loads read from the same writes.  Unfolding~\cite{unfoldings} is an alternative approach to POR for reducing the number of executions to be explored.  The unfolding algorithm involves solving an NP-complete problem to add events to the unfolding.

Recent work has extended these algorithms to handle the TSO and PSO
memory models~\cite{nidhugg,pldipsotso,mcrpsotso} and the release
acquire fragment of C/C++~\cite{statelessrelacq}.  The RCMC tool
implements a DPOR tool that operates on execution graphs for the RC11
memory model~\cite{rcmc}.  SAT solvers have been used to avoid
explicitly enumerating all executions.  SATCheck extends
partial order reduction with the insight that it is only necessary to
explore executions that exhibit new behaviors~\cite{satcheck}.
CheckFence checks code by translating it into
SAT~\cite{checkfence}.
\rahmadi{Other work has also presented techniques orthogonal to DPOR, either in a more general context~\cite{desai2015systematic} or platform specific (\hbox{\emph{e.g.},}\xspace Android~\cite{ozkan2015systematic} and Node.js~\cite{loring2017semantics}).
}

Recent work on dynamic partial order reduction for event-driven programs has
developed dynamic partial order reduction algorithms for stateless
model checking of event-driven
applications~\cite{jensen2015stateless,maiya2016por}.
Jensen \hbox{\emph{et al.}}\xspace\cite{jensen2015stateless} consider a model similar to ours in which
an event is treated as a single transition, while Maiya \hbox{\emph{et al.}}\xspace\cite{maiya2016por}
consider a model in which event execution interleaves concurrently
with threads.  Neither of these approaches handle cyclic state spaces
nor consider challenges that arise from stateful model checking.

Recent work on DPOR algorithms reduces the number of executions for programs with critical sections by considering
whether critical sections contain conflicting
operations~\cite{dporlocks}.  This work considers stateless
model checking of multithreaded programs, but like our work it must
consider code blocks that perform multiple memory
operations.

CHESS~\cite{musuvathi2007chess} is designed to find and reproduce
concurrency bugs in C, C++, and C\#.  It systematically explores
thread interleavings using a preemption bounded strategy.  The Inspect
tool combines stateless model checking and stateful model checking to
model check C and C++
code~\cite{inspect1,inspect2,inspect4}.

In stateful model checking, there has also been substantial work such as
SPIN~\cite{spin}, Bogor~\cite{dwyer2003bogor}, and JPF~\cite{jpf}.
In addition to these model checkers, other researchers have
proposed different techniques to capture
program states~\cite{musuvathi2002cmc,gueta2007cartesian}.

Versions of JPF include a partial order reduction algorithm.  The
design of this algorithm is not well documented, but some publications
have reverse engineered the pseudocode~\cite{porjpf}.  The algorithm
is naive compared to modern DPOR algorithms---this algorithm simply
identifies accesses to shared variables and adds backtracking points
for all threads at any shared variable access.

\subsection{The Algorithm}

We first present the states that our algorithm maintains:
}

This section presents our algorithm, which extends DPOR to support stateful model checking of event-driven applications with cyclic state spaces. We first present the states that our algorithm maintains:

\begin{enumerate}
\item {\bf The transition sequence \Trace{}} contains the new transitions that the current execution explores.  Our algorithm explores a given transition in at most one execution.

\item {\bf The state history \hbox{$\mathcal{H}$}\xspace} is a set of program states that have been visited in completed executions.

\item {\bf The state transition graph \RGraph{}} records the states our algorithm has explored thus far.  Nodes in this graph correspond to program states and edges to transitions between program states.
\end{enumerate}

\begin{algorithm}
\DontPrintSemicolon
\SetKwBlock{Begin}{}{end}
\SetAlgoLined
\Begin($\textsc{ExploreAll}\xspace {(}{)}$){
$\hbox{$\mathcal{H}$}\xspace := \emptyset$\;
$\RGraph{} := \emptyset$\;
$\Trace{} := \emptyset$\;
$\textsc{Explore}\xspace {(} \State{0} {)}$\;\label{line:firstexplore}
\While{$\exists \State{}, \: \hbox{\textit{backtrack}}\xspace(\State{}) \neq \hbox{\textit{done}}\xspace(\State{})$}
      {
$\textsc{Explore}\xspace {(} \State{} {)}$\; \label{line:explore}
      }
}
\caption{Top-level exploration algorithm.\label{alg:sdpormain}}
\end{algorithm}

\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{Explore}\xspace {(} \State{} {)}$)
{
\If{$\hbox{\textit{backtrack}}\xspace(\State{}) = \hbox{\textit{done}}\xspace(\State{}) \label{line:ifback}$}{ \label{line:enable-event-start}
\uIf{$\hbox{\textit{done}}\xspace(\State{}) = \hbox{\textit{enabled}}\xspace(\State{})$}{
\uIf{$\hbox{\textit{enabled}}\xspace(\State{}) \text{ is not empty}$}{\label{line:checkenabled}
$\text{select} \: \Event{} \: \in \hbox{\textit{enabled}}\xspace(\State{})\label{line:evselect}$\;
$\text{remove} \: \Event{} \: \text{from} \: \hbox{\textit{done}}\xspace(\State{})$\;
} \Else {
$\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
$\Trace{} := \emptyset$\;
\Return{}
}
} \Else {
$\text{select} \: \Event{} \: \in \hbox{\textit{enabled}}\xspace(\State{}) \setminus \hbox{\textit{done}}\xspace(\State{})$\;\label{line:select-enabled-event}
$\text{add} \: \Event{} \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\State{})$\;
}
} \label{line:enable-event-end}
\While{$\exists \hbox{$b$}\xspace \in \hbox{\textit{backtrack}}\xspace(\State{}) \setminus \hbox{\textit{done}}\xspace(\State{})$\label{line:while}} {
  $\text{add} \: \hbox{$b$}\xspace \: \text{to} \: \State{}.done$\;
  $\Trans{} := \text{\hbox{\emph{next}}\xspace} \left(\State{},\hbox{$b$}\xspace\right)$\;\label{line:algnext}
  $\State{}' := \hbox{\emph{dst}}\xspace\left(\Trans{}\right)$\;\label{line:algdst}
  $\text{add transition} \: \Trans{} \:\text{to}\:\RGraph{}$\;\label{line:addtrans}
  \ForEach{$\Event{} \in \hbox{\textit{enabled}}\xspace(\State{}) \setminus \hbox{\textit{enabled}}\xspace(\State{}')$ \label{line:add-disabled} }{
    $\text{add} \: \Event \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\State{})$\;\label{line:adddisabled}
  }
  $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:call-update-backtrack-set}
  \uIf{$\State{}' \in \hbox{$\mathcal{H}$}\xspace \vee \textsc{IsFullCycle}\xspace\left(\Trans{}\right)$\label{line:termination-condition}} {
    $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\;\label{line:endbacktrack}
    $\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
    $\Trace{} := \emptyset$\;
  }
  \Else {
    \If{$\State{}' \in \hbox{\emph{states}}\xspace{}(\Trace{}) $\label{line:cycle-current-exec-1}} {
      $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\label{line:cycle-current-exec-2}
    }
    $\Trace{} := \Trace{}.\Trans{}$ \;
    $\textsc{Explore}\xspace {(} \hbox{$s$}\xspace' {)}$\;\label{line:call-explore}
  }
}
}
 \caption{Stateful DPOR algorithm for event-driven applications.\label{alg:sdpor}}
\end{algorithm}

Recall that for each reachable state $\State{} \in \hbox{$\textit{States}$}\xspace{}$, our algorithm maintains the $\hbox{\textit{backtrack}}\xspace(\State{})$ set that contains the events to be explored at $\State{}$, the $\hbox{\textit{done}}\xspace(\State{})$ set that contains the events that have already been explored at $\State{}$, and the $\hbox{\textit{enabled}}\xspace(\State{})$ set that contains all events that are enabled at $\State{}$.

Algorithm~\ref{alg:sdpormain} presents the top-level \textsc{ExploreAll}\xspace procedure.  This procedure first invokes the \textsc{Explore}\xspace procedure to start model checking from the initial state.  However,
the presence of cycles in the state space means that our backtracking-based search algorithm may occasionally set new backtracking points for states in completed executions.  The \textsc{ExploreAll}\xspace procedure thus loops over all states that have unexplored items in their backtrack sets and invokes the \textsc{Explore}\xspace procedure to explore those transitions.

Algorithm~\ref{alg:sdpor} describes the logic of the \textsc{Explore}\xspace{} procedure.
The \code{if} statement in line~\ref{line:ifback} checks if the
current state \State{}'s $\hbox{\textit{backtrack}}\xspace$ set is the same as the
current state \State{}'s \hbox{\textit{done}}\xspace set.  If so, the algorithm
selects an event to execute in the next transition.  If some
enabled events are not yet explored, it selects an unexplored event
to add to the current state's $\hbox{\textit{backtrack}}\xspace$ set.  Otherwise, if
the $\hbox{\textit{enabled}}\xspace$ set is not empty, it selects an enabled event to
remove from the \hbox{\textit{done}}\xspace set.  Note that this scenario occurs only if
the execution is continuing past a state match to satisfy the termination condition.

Then the \code{while} loop in line~\ref{line:while} selects an event
$\hbox{$b$}\xspace$ to execute on the current state \State{} and
executes the event $\hbox{$b$}\xspace$ to generate the
transition \Trans{} that leads to a new state $\State{}'$.
At this point, the algorithm knows the memory accesses performed by
the transition \Trans{} and thus can add the event $\hbox{$b$}\xspace$
to the backtrack sets of the previous states.  This is done via the procedure \textsc{UpdateBacktrackSet}\xspace{}.

Traditional DPOR algorithms continue an execution until it terminates.  Since our programs may have cyclic state spaces, this would cause the model checker to potentially not terminate.  Our algorithm instead checks the conditions in
line~\ref{line:termination-condition} to decide whether to terminate the execution.  These checks see whether the new state $\State{}'$ matches a state from a previous execution, or if the current execution revisits a state the current execution previously explored and meets other criteria that are checked in the \textsc{IsFullCycle}\xspace{} procedure.   If so, the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure to set backtracking points, from transitions reachable from $\Trans{}$, to states that can reach $\Trans{}$.  An execution will also terminate if it reaches a state in which no event is enabled (line~\ref{line:checkenabled}).
It then adds the states from the current transition sequence to the set of previously visited states $\hbox{$\mathcal{H}$}\xspace$, resets the current execution transition sequence $\Trace{}$, and backtracks to start a new execution.

If the algorithm has reached a state $\State{}'$ that was previously discovered in this execution, it sets backtracking points by calling the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  Finally, it updates the transition sequence $\Trace{}$ and calls \textsc{Explore}\xspace{}.

\vspace{-2em}
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSetsFromGraph}\xspace {(} \Trans{s}{)}$)
{
  $\RGraph{t} := \lbrace
    \Trans{}\in\RGraph{} \mid \:\: \Trans{s}\hbox{$\:\rightarrow_{r}\:$}\xspace\Trans{}
    \rbrace$\;
  \ForEach{$\Trans{} \in \RGraph{t}$\label{line:check-cycle-in-prev-1}}
  {
     $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:check-cycle-in-prev-2}
  }
}
 \caption{Procedure that updates the backtrack sets of states in previous executions.\label{alg:backtrack-update-graph}}
\end{algorithm}
\vspace{-2em}

Algorithm~\ref{alg:backtrack-update-graph} shows the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  This procedure takes a transition $\Trans{}$ that connects the current execution to a previously discovered state in the transition graph $\RGraph{}$.  Since our algorithm does \emph{not} explore all of the transitions reachable from the previously discovered state, we need to set the backtracking points that would have been set by these skipped transitions. This procedure therefore computes the set of transitions reachable from the destination state of $\Trans{}$ and invokes $\textsc{UpdateBacktrackSet}\xspace{}$ on each of those transitions to set backtracking points.

\vspace{-2em}
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{IsFullCycle}\xspace {(} \Trans{} {)}$)
{
\If{$\neg \textit{dst}(\Trans{}) \in \hbox{\emph{states}}\xspace{}(\Trace{})$}{\Return{$\textit{false}$}}
  $\Trace{}^{\textit{fc}} := \lbrace
    \Trans{j} \in \Trace{} \mid
        i=\hbox{\emph{first}}\xspace(\Trace{}, \hbox{\emph{dst}}\xspace(\Trans{})), \:\: \text{and} \:\:
        i < j \rbrace \cup \{ \Trans{} \}$\;
  $\Events{\textit{fc}} := \lbrace
    \hbox{\emph{event}}\xspace{}(\Trans{}') \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\; \label{line:fullcycle-fc}
  $\Events{\textit{enabled}} := \lbrace
    \hbox{\textit{enabled}}\xspace(\hbox{\emph{dst}}\xspace{}(\Trans{}')) \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\; \label{line:fullcycle-enabled}
  \Return{$\Events{\textit{fc}} = \Events{\textit{enabled}}\:$\label{line:check-full-cycle}}
}
 \caption{Procedure that checks the looping termination condition: a cycle that contains every event enabled in the cycle.
 \label{alg:complete-full-cycle}}
\end{algorithm}
\vspace{-2em}

Algorithm~\ref{alg:complete-full-cycle} presents
the \textsc{IsFullCycle}\xspace{} procedure.  This procedure first checks if
there is a cycle that contains the transition $\Trans{}$ in the state
space explored by the current execution.  The example from
Figure~\ref{fig:example-non-terminating} shows that such a state match
is not sufficient to terminate the execution as the execution may not
have set the necessary backtracking points.  Our algorithm stops the
exploration of an execution when there is a cycle that has explored
\emph{every event that is enabled in that cycle}.  This ensures that for
every transition \Trans{} in the execution, there is a future
transition \Trans{e} for each enabled event \Event{} in the cycle that
can set a backtracking point if \Trans{} and \Trans{e} conflict.

Algorithm~\ref{alg:backtrack-updatev2} presents the \textsc{UpdateBacktrackSet}\xspace{} procedure, which sets backtracking points.  There are two differences between our algorithm and traditional DPOR algorithms.  First, since our algorithm supports programs with cyclic state spaces, it is possible that the algorithm has discovered multiple paths from the start state $\State{0}$ to the current transition $\Trans{}$.  Thus, the algorithm must potentially set backtracking points on multiple different paths.  We address this issue using a backwards depth first search traversal of the $\RGraph{}$ graph.   Second, since our transitions correspond to events, they may potentially access multiple different memory locations and thus the backtracking algorithm potentially needs to set separate backtracking points for each of these memory locations.

The \textsc{UpdateBacktrackSetDFS}\xspace{} procedure implements a backwards
depth first traversal to set backtracking points.  The procedure takes
the following parameters: \Trans{\text{curr}} is the current transition in the
DFS, \Trans{\text{conf}} is the transition that we are currently setting
a backtracking point for, \Access{} is the set of accesses that the
algorithm searches for conflicts for, and \Transitions{\text{exp}} is
the set of transitions that the algorithm has explored down this
search path.  Recall that accesses consist of both an operation, \hbox{\emph{i.e.},}\xspace a read or write, and a memory location.  Conflicts are defined in the usual way---writes to a memory location conflict with reads or writes to the same location.

\vspace{-2em}
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSet}\xspace {(} \Trans{} {)}$)
{
 $\textsc{UpdateBacktrackSetDFS}\xspace\left( \Trans{}, \Trans{}, \textit{accesses}(\Trans{}), \{ \Trans{} \} \right)$
}

\Begin($\textsc{UpdateBacktrackSetDFS}\xspace {(} \Trans{\text{curr}}, \Trans{\text{conf}},  \Access{}, \Transitions{\text{exp}} {)}$)
{
  \ForEach{$\Trans{b} \in \textit{pred}_{\RGraph{}}(\Trans{\text{curr}}) \setminus \Transitions{\text{exp}} $\label{lin:loop}}{
  $\Access{b} := \textit{accesses}(\Trans{b})$\;
  $\Trans{\text{conf}}' := \Trans{\text{conf}}$\;
  \If{$\exists a \in \Access{}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$\label{lin:conflict}} {
   \uIf{$\hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \in \hbox{\textit{enabled}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b}))$} {
   $\text{add} \: \hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b}))$\;
   } \Else {
      $\text{add} \: \hbox{\textit{enabled}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b})) \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b}))$\;
   }
   $\Trans{\text{conf}}' := \Trans{b}$\;
  }
  $\Access{r} := \{ a \in \Access{} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;  \label{lin:newaccesses}
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{b}, \Trans{\text{conf}}', \Access{r}, \Transitions{\text{exp}} \: \cup \{ \Trans{b} \} \right)$
  }
}

 \caption{Procedure that updates the backtrack sets of states for previously executed transitions that conflict with the current transition in the search stack.
  \label{alg:backtrack-updatev2}}
\end{algorithm}
\vspace{-2em}

Line~\ref{lin:loop} loops over each transition \Trans{b} that
immediately precedes transition \Trans{\text{curr}} in the state transition
graph and has not been explored.  Line~\ref{lin:conflict} checks for
conflicts between the accesses of \Trans{b} and the access
set \Access{} for the DFS.  If a conflict is detected, the
algorithm adds the event for transition \Trans{\text{conf}} to the backtrack set.
Line~\ref{lin:newaccesses} removes the accesses that conflicted
with transition \Trans{b}.  The search procedure then recursively
calls itself.  If the current transition \Trans{b} conflicts with the
transition \Trans{\text{conf}} for which we are setting a backtracking point, then
it is possible that the behavior we are interested in for \Trans{\text{conf}}
requires that \Trans{b} be executed first.  Thus, if there is a
conflict between \Trans{b} and \Trans{\text{conf}}, we pass \Trans{b} as the
conflict transition parameter to the recursive calls
to \textsc{UpdateBacktrackSetDFS}\xspace{}.

\rahmadi{
\inarxivversion{\appdx{Appendix~\ref{sec:proof}} proves correctness properties for our DPOR algorithm.}
\invmcaiversion{\appdx{Appendix B} proves correctness properties for our DPOR algorithm.}
\inarxivversion{\appdx{Appendix~\ref{sec:example}} revisits the example shown in}
\invmcaiversion{\appdx{Appendix C} revisits the example shown in}
Figure~\ref{fig:example-missing-execution}. It describes how our DPOR algorithm explores all executions in Figure~\ref{fig:example-missing-execution}, including Figure~\ref{fig:example-missing-execution}-i.
}

\mycomment{
\mysubsection{Discussion}

Although the algorithm we present is designed to explore executions of event-driven systems, the key ideas are also applicable to model checking multithreaded programs.  To the best of our knowledge, there are no correct algorithms for stateful dynamic partial order reduction for multithreaded applications that potentially do not terminate.  The proposed technique bridges the gap between stateful model checking and DPOR, allowing easy adaptation of many existing DPOR algorithms proposed for stateless model checking~\cite{tasharofi2012transdpor,odpor,statelessmcr,chatterjee2019valuecentricdpor,jensen2015stateless} to the stateful context.
}
\subsection{Event-Driven Concurrency Model}
\rahmadi{
We next present our event-driven concurrency model (see an example of event-driven
\invmcaiversion{systems in \appdx{Appendix A}).}
\inarxivversion{systems in \appdx{Appendix~\ref{sec:example-event-driven-system}}).}
}
We assume that
the event-driven system has a finite set $\Events{}$ of different event types.  Each event type $\Event{} \in \Events{}$ has a
corresponding event handler that is executed when an instance of the
event occurs.  We assume that there is a potentially shared state and
that event handlers have arbitrary access to read and write from this
shared state.

An event handler can be an arbitrarily long finite sequence of
instructions and can include an arbitrary number of accesses to shared
state.  We assume event-handlers are executed atomically by the
event-driven runtime system.
Events can be enabled by both external sources (\hbox{\emph{e.g.},}\xspace events in the
physical world) or event handlers.  Events can also be disabled by
the execution of an event handler.
We assume that the runtime system maintains an unordered set of
enabled events to execute.  It contains an event dispatch
loop that selects an arbitrary enabled event to execute next.

This work is inspired by smart-home systems that are widely deployed in the real world. However, the proposed techniques are general enough to handle other types of event-driven systems,
such as web applications, as long as the systems follow the concurrency model stated above.
\vspace{-.5em}

\subsection{Background on Stateless DPOR}

Partial order reduction is based on the observation that traces of concurrent systems are equivalent if they only reorder independent operations.  These equivalence classes are called Mazurkiewicz traces~\cite{tracetheory}.  The classical DPOR algorithm~\cite{flanagan2005dynamic} dynamically computes persistent sets for multithreaded programs and is guaranteed to explore at least one interleaving in each equivalence class.

The key idea behind the DPOR algorithm is to compute the next pending memory operation for each thread, and at each point in the execution to compute the most recent conflict for each thread's next operation.  These conflicts are used to set backtracking points so that future executions will reverse the order of conflicting operations and explore an execution in a different equivalence class. Due to space constraints, we refer the interested readers to \cite{flanagan2005dynamic} for a detailed description of the original DPOR algorithm.

\section*{Acknowledgment}
We would like to thank our anonymous reviewers for their thorough comments and feedback. This project was supported partly by the National Science Foundation under grants CCF-2006948, CCF-2102940, CNS-1703598, CNS-1763172, CNS-1907352, CNS-2006437, CNS-2007737, CNS-2106838, CNS-2128653, OAC-1740210 and by the Office of Naval Research under grant N00014-18-1-2037.
}

\section{#1}\vspace{-.27em}}
\newcommand{\mysubsection}[1]{\vspace{-.27em}\subsection{#1}\vspace{-.27em}}
\newcommand{\mysubsubsection}[1]{\subsubsection{#1}}
\newcommand{\myparagraph}[1]{\noindent\textit{\textbf{{#1}.}}}
\newcommand{\mysubparagraph}[1]{\subparagraph{#1}}
\newcommand{\mycomment}[1]{}
\newcommand{\tuple}[1]{\ensuremath \langle #1 \rangle}
\newcommand{\Naive}[0]{Na\"{i}ve\xspace}
\newcommand{\naively}[0]{na\"{i}vely\xspace}
\newcommand{\Naively}[0]{Na\"{i}vely\xspace}
\newcommand{\naive}[0]{na\"{i}ve\xspace}
\newcommand{\interacts}[0]{\emph{interacts-with}\xspace}
\newcommand{\conflict}[0]{\emph{conflict}\xspace}
\newcommand{\Interacts}[0]{\emph{Interacts-with}\xspace}
\newcommand{\Conflict}[0]{\emph{Conflict}\xspace}
\newcommand{\Devicestate}[0]{\emph{Device}\xspace}
\newcommand{\Physical}[0]{\emph{Physical-Medium}\xspace}
\newcommand{\Globalstate}[0]{\emph{Global-Variable}\xspace}
\newcommand{\devicestate}[0]{\emph{device}\xspace}
\newcommand{\globalstate}[0]{\emph{global-variable}\xspace}
\newcommand{\physical}[0]{\emph{physical-medium}\xspace}
\newcommand{\conflictinreader}[0]{\emph{conflict in reader}\xspace}
\newcommand{\readop}[0]{\emph{read-from}\xspace}
\newcommand{\writeop}[0]{\emph{write-to}\xspace}
\newcommand{\readsop}[0]{\emph{reads-from}\xspace}
\newcommand{\writesop}[0]{\emph{writes-to}\xspace}
\newcommand{\doorlock}[0]{door-lock\xspace}
\newcommand{\doorunlock}[0]{door-unlock\xspace}
\newcommand{\doorlocks}[0]{door-locks\xspace}
\newcommand{\doorunlocks}[0]{door-unlocks\xspace}
\newcommand{\locationmode}[0]{\code{location.mode}\xspace}
\newcommand{\locationspacemode}[0]{\code{location.} \code{mode}\xspace}

\newcommand{\tool}[0]{\textsc{IoTCheck}\xspace}
\newcommand{\initialstateeventstreamer}[0]{\code{Initial}-\code{State}-\code{Event}-\code{Streamer}\xspace}
\newcommand{\bigturnon}[0]{\code{Big}-\code{Turn}-\code{ON}\xspace}
\newcommand{\firecoalarm}[0]{\code{FireCO2Alarm}\xspace}
\newcommand{\devicetamperalarm}[0]{\code{Device}-\code{Tamper}-\code{Alarm}\xspace}
\newcommand{\windowordooropen}[0]{\code{WindowOrDoorOpen}\xspace}
\newcommand{\lockitwhenileave}[0]{\code{Lock}-\code{It}-\code{When}-\code{I}-\code{Leave}\xspace}
\newcommand{\lockitataspecifictime}[0]{\code{Lock}-\code{It}-\code{at}-\code{a}-\code{Specific}-\code{Time}\xspace}
\newcommand{\autolockdoor}[0]{\code{Auto}-\code{Lock}-\code{Door}\xspace}
\newcommand{\thermostats}[0]{\code{Thermostats}\xspace}
\newcommand{\thermostatautooff}[0]{\code{Thermostat}-\code{Auto}-\code{Off}\xspace}
\newcommand{\neato}[0]{\code{Neato}-\code{(Connect)}\xspace}
\newcommand{\forgivingsecurity}[0]{\code{Forgiving}-\code{Security}\xspace}
\newcommand{\turnonatsunset}[0]{\code{Turn}-\code{On}-\code{at}-\code{Sunset}\xspace}
\newcommand{\lightupthenight}[0]{\code{Light}-\code{Up}-\code{the}-\code{Night}\xspace}
\newcommand{\bosesound}[0]{\code{Bose}-\code{SoundTouch}-\code{Control}\xspace}
\newcommand{\sprayercontroller}[0]{\code{Sprayer}-\code{Controller}-\code{2}\xspace}
\newcommand{\closethevalve}[0]{\code{Close}-\code{The}-\code{Valve}\xspace}
\newcommand{\influxdblogger}[0]{\code{InfluxDB}-\code{Logger}\xspace}
\newcommand{\buttoncontroller}[0]{\code{Button}-\code{Controller}\xspace}
\newcommand{\smartsecurity}[0]{\code{Smart}-\code{Security}\xspace}
\newcommand{\vacationlightingdirector}[0]{\code{Vacation}-\code{Lighting}-\code{Director}\xspace}
\newcommand{\lightingdirector}[0]{\code{Lighting}-\code{Director}\xspace}
\newcommand{\bigturnoff}[0]{\code{Big}-\code{Turn}-\code{OFF}\xspace}
\newcommand{\monitoronsense}[0]{\code{Monitor}-\code{on}-\code{Sense}\xspace}
\newcommand{\huemoodlighting}[0]{\code{Hue}-\code{Mood}-\code{Lighting}\xspace}
\newcommand{\doorstatetocolorlight}[0]{\code{Door}-\code{State}-\code{to}-\code{Color}-\code{Light}\xspace}
\newcommand{\greetingsearthling}[0]{\code{Greetings}-\code{Earthling}\xspace}
\newcommand{\hellohomephrasedirector}[0]{\code{Hello,}-\code{Home}-\code{Phrase}-\code{Director}\xspace}
\newcommand{\bonvoyage}[0]{\code{Bon}-\code{Voyage}\xspace}
\newcommand{\goodnight}[0]{\code{Good}-\code{Night}\xspace}
\newcommand{\wholehousefan}[0]{\code{Whole}-\code{House}-\code{Fan}\xspace}
\newcommand{\autohumidityvent}[0]{\code{Auto}-\code{Humidity}-\code{Vent}\xspace}
\newcommand{\keepmecozy}[0]{\code{Keep}-\code{Me}-\code{Cozy}\xspace}

\newcommand{\harry}[1] {{\textcolor{blue}{Harry: {#1}}}}
\newcommand{\rahmadi}[1]{{#1}}
\newcommand{\weiyu}[1]{{\color{purple}#1}}
\newcommand{\camready}[1]{#1}

\newcounter{isvmcaiversion}
\setcounter{isvmcaiversion}{0}

\ifthenelse{\value{isvmcaiversion}>0}
{
    \SetWatermarkText{\hspace*{6in}\raisebox{8.8in}{\includegraphics[scale=0.15]{vmcai-badges/aec-badge-vmcai}}}
    \SetWatermarkAngle{0}
    \pagenumbering{gobble}
    \newcommand{\appdx}[1]{\text{\color{blue} #1 in~\todo{\cite{ArXiv}}}}
    \newcommand{\invmcaiversion}[1]{#1}
    \newcommand{\inarxivversion}[1]{}
}
{
    \SetWatermarkText{}
    \SetWatermarkAngle{0}

    \newcommand{\appdx}[1]{#1}
    \newcommand{\invmcaiversion}[1]{}
    \newcommand{\inarxivversion}[1]{#1}
}

\maketitle

\thispagestyle{plain}
\pagestyle{plain}

\input{abstract}
\input{intro}
\input{concurrency}
\input{definitions}
\input{basicideas}
\input{algorithm}
\input{evaluation-new}
\input{related-work}
\input{conclusion}
\input{acknowledgment}

\bibliographystyle{splncs04}

\subsection{The Algorithm}

We first present the state that our algorithm maintains:

\begin{enumerate}
\item {\bf The transition sequence \Trace{}} contains the new transitions that the current execution explores.  Our algorithm explores a given transition in at most one execution.

\item {\bf The state history \hbox{$\mathcal{H}$}\xspace} is a set of program states that have been visited in completed executions.

\item {\bf The state transition graph \RGraph{}} records the states our algorithm has explored thus far.  Nodes in this graph correspond to program states and edges to transitions between program states.

\rahmadi{
\item {\bf The state summary \Summary{}} records a summary of events (and their corresponding accesses) for which we have set backtracking points at every state our algorithm has explored. With \Summary{}, the algorithm can efficiently reexplore the previously explored transitions: (1) when a previously discovered state \State{} is reached, \Summary{} serves as a summary of all reachable transitions from \State{}, and (2) when performing backwards depth first search, our algorithm can stop exploring the execution at state \State{} whenever the algorithm finds that the currently executed event and its accesses are a subset of the summary for \State{} in \Summary{}.
}
\end{enumerate}

Recall that for each reachable state $\State{} \in \hbox{$\textit{States}$}\xspace{}$, our algorithm maintains the \State{}.\textit{backtrack} set that contains the events to be explored at that state, the \State{}.\textit{done} set that contains the events that have already been explored at the state, and the \State{}.\textit{enabled} set that contains all events that are enabled to be executed at that state.

\begin{algorithm}

\DontPrintSemicolon
\SetKwBlock{Begin}{}{end}
\SetAlgoLined
\Begin($\textsc{ExploreAll}\xspace {(}{)}$){
\rahmadi{
$\Trace{} := \emptyset$\;
$\hbox{$\mathcal{H}$}\xspace := \emptyset$\;
$\RGraph{} := \emptyset$\;
$\Summary{} := \emptyset$\;
}
$\textsc{Explore}\xspace {(} \State{0} {)}$\;\label{line:firstexplore}
\While{$\exists \State{}, \: \State{}.backtrack \neq \State{}.done$}
      {
$\textsc{Explore}\xspace {(} \State{} {)}$\; \label{line:explore}
      }
}
\caption{Top-level exploration algorithm.\label{alg:sdpormain}}
\end{algorithm}

Algorithm~\ref{alg:sdpormain} presents the top-level \textsc{ExploreAll}\xspace procedure.  This procedure first invokes the \textsc{Explore}\xspace procedure to start model checking from the initial state.  However,
the presence of cycles in the state space means that our backtracking-based search algorithm may occasionally set new backtracking points for states in completed executions.  The \textsc{ExploreAll}\xspace procedure thus loops over all states that have unexplored items in their backtrack sets and invokes the \textsc{Explore}\xspace procedure to explore those transitions.

\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{Explore}\xspace {(} \State{} {)}$)
{
\If{$\State{}.\textit{backtrack} = \State{}.\textit{done}\label{line:ifback}$}{ \label{line:enable-event-start}
\uIf{$\State{}.\textit{done} = \State{}.\textit{enabled}$}{
\uIf{$\State{}.\textit{enabled} \text{ is not empty}$}{\label{line:checkenabled}
$\text{select} \: \Event{} \: \in \State{}.\textit{enabled}\label{line:evselect}$\;
$\text{remove} \: \Event{} \: \text{from} \: \State{}.\textit{done}$\;
} \Else {
$\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
$\Trace{} := \emptyset$\;
\Return{}
}
} \Else {
$\text{select} \: \Event{} \: \in \State{}.\textit{enabled} \setminus \State{}.\textit{done}$\;
$\text{add} \: \Event{} \: \text{to} \: \State{}.\textit{backtrack}$\;
}
} \label{line:enable-event-end}
\While{$\exists \hbox{$b$}\xspace \in \State{}.\textit{backtrack} \setminus \State{}.\textit{done}$\label{line:while}} {
  $\text{add} \: \hbox{$b$}\xspace \: \text{to} \: \State{}.done$\;
  $\Trans{} := \text{\hbox{\emph{next}}\xspace} \left(\State{},\hbox{$b$}\xspace\right)$\;\label{line:algnext}
  $\State{}' := \hbox{\emph{dst}}\xspace\left(\Trans{}\right)$\;\label{line:algdst}
  $\text{add transition} \: \Trans{} \:\text{to}\:\RGraph{}$\;\label{line:addtrans}
  \ForEach{$\Event{} \in \State{}.\textit{enabled} \setminus    \label{line:add-disabled} \State{}'.\text{enabled}$}{
    $\text{add} \: \Event \: \text{to} \: \State{}.\textit{backtrack}$\;\label{line:adddisabled}
  }
  $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:call-update-backtrack-set}
  \uIf{$\State{}' \in \hbox{$\mathcal{H}$}\xspace \vee \textsc{IsFullCycle}\xspace\left(\Trans{}\right)$\label{line:termination-condition}} {
    $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\;\label{line:endbacktrack}
    $\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
    $\Trace{} := \emptyset$\;
  }
  \Else {
    \If{$\State{}' \in \hbox{\emph{states}}\xspace{}(\Trace{}) $\label{line:cycle-current-exec-1}} {
      $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\label{line:cycle-current-exec-2}
    }
    $\Trace{} := \Trace{}.\Trans{}$ \;
    $\textsc{Explore}\xspace {(} \hbox{$s$}\xspace' {)}$\;\label{line:call-explore}
  }
}
}
 \caption{Stateful DPOR algorithm for event-driven applications.\label{alg:sdpor}}
\end{algorithm}

Algorithm~\ref{alg:sdpor} describes the logic of the \textsc{Explore}\xspace{} procedure.
The \code{if} statement in line~\ref{line:ifback} checks if the
current state \State{}'s \textit{backtrack} set is the same as the
current state \State{}'s \textit{done} set.  If so, the algorithm
selects an event to execute in the next transition.  If there are
enabled events not yet explored, it selects an unexplored event
to add to the current state's \textit{backtrack} set.  Otherwise, if
the \textit{enabled} set is not empty, it selects an enabled event to
remove from the \textit{done} set.  Note that this scenario occurs only if
the execution is continuing past a state match to satisfy the termination condition.

Then the \code{while} loop in line~\ref{line:while} selects an event
$\hbox{$b$}\xspace$ to execute on the current state \State{} and
executes the event $\hbox{$b$}\xspace$ to generate the
transition \Trans{} that takes the execution to a new state \State{}'.
At this point, the algorithm knows the memory accesses performed by
the transition \Trans{} and thus can add the event $\hbox{$b$}\xspace$
to the backtrack sets of the previous states.  This is done via the procedure \textsc{UpdateBacktrackSet}\xspace{}.

Traditional DPOR algorithms continue an execution until it terminates.  Since our programs may have cyclic state spaces, this would cause the model checker to potentially not terminate.  Our algorithm instead checks the conditions in
line~\ref{line:termination-condition} to decide whether to terminate the execution.  These checks see whether the new state $\State{}'$ matches a state from a previous execution, or if the current execution revisits a state the current execution previously explored and meets other criteria that are checked in the \textsc{IsFullCycle}\xspace{} procedure.   If so, the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure to set backtracking points, from transitions reachable from $\Trans{}$, to states that can reach $\Trans{}$.  An execution will also terminate if it reaches a state in which no event is enabled (line~\ref{line:checkenabled}).
It then adds the states from the current transition sequence to the set of previously visited states $\hbox{$\mathcal{H}$}\xspace$, resets the current execution transition sequence $\Trace{}$, and backtracks to start a new execution.

If the algorithm has reached a state $\State{}'$ that was previously discovered in this execution, it sets backtracking points by calling the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  Finally, it updates the transition sequence $\Trace{}$ and calls \textsc{Explore}\xspace{}.

\mycomment{
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSetsFromGraph}\xspace {(} \Trans{s}{)}$)
{
  $\RGraph{t} := \lbrace
    \Trans{}\in\RGraph{} \mid \:\: \Trans{s}\hbox{$\:\rightarrow_{r}\:$}\xspace\Trans{}
    \rbrace$\;
  \ForEach{$\Trans{} \in \RGraph{t}$\label{line:check-cycle-in-prev-1}}
  {
     $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:check-cycle-in-prev-2}
  }
}
 \caption{Procedure that updates the backtrack sets of states in a previous execution.\label{alg:backtrack-update-graph}}
\end{algorithm}
}

\begin{algorithm}
\rahmadi{
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSetsFromGraph}\xspace {(} \Trans{}{)}$)
{
  $\Events{r} := \{ \Event{} \mid \forall \Event{}, \langle \Event{}, a \rangle \in \Summary{}(\hbox{\emph{dst}}\xspace(\Trans{}))\}$\;\label{line:compute-state-summary}
  \ForEach{$\Event{} \in \Events{r}$\label{line:check-cycle-in-prev-1}}
  {
     $\Access{r} := \{ a \mid \forall \langle \Event{}, a \rangle \in \Summary{}(\hbox{\emph{dst}}\xspace(\Trans{}))\}$\;
       $\Access{b} := \textit{accesses}(\Trans{})$\;
  $\Event{\text{conf}}' := \Event{}$\;
  \If{$\exists a \in \Access{r}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$} {
   \uIf{$\Event{} \in \hbox{\emph{src}}\xspace(\Trans{}).\textit{enabled}$} {
   $\text{add} \: \Event{} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{}).\textit{backtrack}$\;
   } \Else {
      $\text{add} \: \hbox{\emph{src}}\xspace(\Trans{}).\textit{enabled} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{}).\textit{backtrack}$\;
   }
   $\Event{\text{conf}}' := \hbox{\emph{event}}\xspace(\Trans{})$\;
  }
  $\Access{r}' := \{ a \in \Access{r} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{}, \Event{\text{conf}}', \Access{r}', \{ \Trans{} \} \right)$\;\label{line:check-cycle-in-prev-2}
  }
}
 \caption{Procedure that updates the backtrack sets of states in a previous execution.\label{alg:backtrack-update-graph}}
 }
\end{algorithm}

\mycomment{
Algorithm~\ref{alg:backtrack-update-graph} shows the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  This procedure takes a transition $\Trans{}$ that connects the current execution to a previously discovered state in the transition graph $\RGraph{}$.  Since our algorithm does \emph{not} explore all of the transitions reachable from the previously discovered state, we need to create the backtracking points that would have been created by these skipped transitions. This procedure therefore computes the set of transitions reachable from the destination state of $\Trans{}$ and invokes $\textsc{UpdateBacktrackSet}\xspace{}$ on each of those transitions to set backtracking points.
}
\rahmadi{
Algorithm~\ref{alg:backtrack-update-graph} shows the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  This procedure takes a transition $\Trans{}$ that connects the current execution to a previously discovered state in the transition graph $\RGraph{}$.  Since our algorithm does \emph{not} explore all of the transitions reachable from the previously discovered state, we need to create the backtracking points that would have been created by these skipped transitions. This procedure therefore computes the set of transitions reachable from the destination state of $\Trans{}$, namely $\hbox{\emph{dst}}\xspace{}(\Trans{})$. As the previously explored transitions are stored as a summary of events and their accesses in \Summary{} for every discovered state, the procedure first computes the summary of events and their accesses for $\hbox{\emph{dst}}\xspace{}(\Trans{})$ in line~\ref{line:compute-state-summary}.
It then invokes $\textsc{UpdateBacktrackSetDFS}\xspace{}$ on each of those events to set backtracking points.
}

\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{IsFullCycle}\xspace {(} \Trans{} {)}$)
{
\If{$\neg \textit{dst}(\Trans{}) \in \hbox{\emph{states}}\xspace{}(\Trace{})$}{\Return{$\textit{false}$}}
  $\Trace{}^{\textit{fc}} := \lbrace
    \Trans{j} \in \Trace{} \mid
        i=\hbox{\emph{first}}\xspace(\Trace{}, \hbox{\emph{dst}}\xspace(\Trans{})), \:\: \text{and} \:\:
        i < j \rbrace \cup \{ \Trans{} \}$\;
  $\Events{\textit{fc}} := \lbrace
    \hbox{\emph{event}}\xspace{}(\Trans{}') \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\;
  $\Events{\textit{enabled}} := \lbrace
    \hbox{\emph{dst}}\xspace{}(\Trans{}').\textit{enabled} \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\;
  \Return{$\Events{\textit{fc}} = \Events{\textit{enabled}}\:$\label{line:check-full-cycle}}cur
}
 \caption{Function that checks the looping termination condition: complete a cycle that contains all events.
 \label{alg:complete-full-cycle}}
\end{algorithm}

\mycomment{
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSet}\xspace {(} \Trans{} {)}$)
{
 $\textsc{UpdateBacktrackSetDFS}\xspace\left( \Trans{}, \Trans{}, \textit{accesses}(\Trans{}), \{ \Trans{} \} \right)$
}

\Begin($\textsc{UpdateBacktrackSetDFS}\xspace {(} \Trans{\text{curr}}, \Trans{\text{conf}},  \Access{}, \Transitions{\text{explored}} {)}$)
{
  \ForEach{$\Trans{b} \in \textit{pred}_{\RGraph{}}(\Trans{\text{curr}}) \setminus \Transitions{\text{explored}} $\label{lin:loop}}{
  $\Access{b} := \textit{accesses}(\Trans{b})$\;
  $\Trans{\text{conf}}' := \Trans{\text{conf}}$\;
  \If{$\exists a \in \Access{}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$\label{lin:conflict}} {
   \uIf{$\hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \in \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled}$} {
   $\text{add} \: \hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   } \Else {
      $\text{add} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   }
   $\Trans{\text{conf}}' := \Trans{b}$\;
  }
  $\Access{r} := \{ a \in \Access{} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;  \label{lin:newaccesses}
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{b}, \Trans{\text{conf}}', \Access{r}, \Transitions{\text{explored}} \: \cup \{ \Trans{b} \} \right)$\;
  }
}

 \caption{Procedure that updates the backtrack sets of the state of the event that conflict with the current event in the search stack.
  \label{alg:backtrack-updatev2}}
\end{algorithm}
}

\begin{algorithm}
\rahmadi{
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSet}\xspace {(} \Trans{} {)}$)
{
 $\textsc{UpdateBacktrackSetDFS}\xspace\left( \Trans{}, \hbox{\emph{event}}\xspace(\Trans{}), \textit{accesses}(\Trans{}), \{ \Trans{} \} \right)$
}

\Begin($\textsc{UpdateStateSummary}\xspace {(} \State{}, \Event{}, \Access{} {)}$)
{

  $\Access{}' := \Access{} \setminus \{ a \mid \forall \langle \Event{}, a \rangle \in \Summary{}(\State{})\}$\;
  $\Summary{} := \Summary{}[ \State{} := \Summary{}(\State{}) \cup (\{ \Event{} \} \times \Access{})]$\;
  \Return{$\Access{}'$}
}

\Begin($\textsc{UpdateBacktrackSetDFS}\xspace {(} \Trans{\text{curr}}, \Event{\text{conf}},  \Access{}, \Transitions{\text{explored}} {)}$)
{
  $\Access{} := \textsc{UpdateStateSummary}\xspace\left( \hbox{\emph{src}}\xspace(\Trans{curr}), \Event{conf}, \Access{} \right)$\;\label{lin:updatestatesummary}
  \If{$\Access{} \neq \emptyset$}{
  \ForEach{$\Trans{b} \in \textit{pred}_{\RGraph{}}(\Trans{\text{curr}}) \setminus \Transitions{\text{explored}} $\label{lin:loop}}{
  $\Access{b} := \textit{accesses}(\Trans{b})$\;
  $\Event{\text{conf}}' := \Event{\text{conf}}$\;
  \If{$\exists a \in \Access{}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$\label{lin:conflict}} {
   \uIf{$\Event{\text{conf}} \in \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled}$} {
   $\text{add} \: \Event{\text{conf}} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   } \Else {
      $\text{add} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   }
   $\Event{\text{conf}}' := \hbox{\emph{event}}\xspace(\Trans{b})$\;
  }
  $\Access{r} := \{ a \in \Access{} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;  \label{lin:newaccesses}
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{b}, \Event{\text{conf}}', \Access{r}, \Transitions{\text{explored}} \: \cup \{ \Trans{b} \} \right)$\;
  }
  }
}

 \caption{Procedure that updates the backtrack sets of the state of the event that conflict with the current event in the search stack.
  \label{alg:backtrack-updatev2}}}
\end{algorithm}

Algorithm~\ref{alg:complete-full-cycle} presents
the \textsc{IsFullCycle}\xspace{} procedure.  This procedure first checks that
there is a cycle that contains the transition $\Trans{}$ in the state
space of the current execution.  The example from
Figure~\ref{fig:example-non-terminating} shows that such a state match
is not sufficient to terminate the execution as the execution may not
have set the necessary backtracking points.  Our algorithm stops the
exploration of an execution when there is a cycle that has explored
\emph{every event that is enabled in that cycle}.  This ensures that for
every transition \Trans{} in the execution, there is a future
transition \Trans{e} for each enabled event \Event{} in the cycle that
can set a backtracking point if \Trans{} and \Trans{e} conflict.

\mycomment{
Algorithm~\ref{alg:backtrack-updatev2} presents the \textsc{UpdateBacktrackSet}\xspace{} procedure, which sets backtracking points.  There are two differences between our algorithm and traditional DPOR algorithms.  First, since our algorithm supports programs with cyclic state spaces, it is possible that the algorithm has discovered multiple paths from the start state $\State{0}$ to the current transition $\Trans{}$.  Thus, the algorithm must potentially set backtracking points on multiple different paths.  We address this issue using a backwards depth first search traversal of the state transition graph $\RGraph{}$.   Second, since our transitions correspond to events they may potentially access multiple different memory locations and thus the backtracking algorithm potentially needs to set separate backtracking points for each of these memory locations.

The \textsc{UpdateBacktrackSetDFS}\xspace{} procedure implements a backwards
depth first traversal to set backtracking points.  The procedure takes
the following parameters: \Trans{\text{curr}} is the current transition in the
DFS, \Trans{\text{conf}} is the transition that we are currently setting
a backtracking point for, \Access{} is the set of accesses that the
algorithm searches for conflicts for, and \Transitions{\text{explored}} is
the set of transitions that the algorithm has explored down this
search path.  Recall that accesses consist of both an operation, \hbox{\emph{i.e.},}\xspace a read or write, and a memory location.  Conflicts are defined in the usual way---writes to a memory location conflict with reads or writes to the same memory location.

Line~\ref{lin:loop} loops over each transition \Trans{b} that
immediately precedes transition \Trans{\text{curr}} in the state transition
graph and has not been explored.  Line~\ref{lin:conflict} checks for
conflicts between the accesses of \Trans{b} and the access
set \Access{} for the DFS.  If a conflict is detected, the
algorithm adds the event for transition \Trans{\text{conf}} to the backtracking
set.  Line~\ref{lin:newaccesses} removes the accesses that conflicted
with transition \Trans{b}.  The search procedure then recursively
calls itself.  If the current transition \Trans{b} conflicts with the
transition \Trans{\text{conf}} for which we are setting a backtracking point, then
it is possible that the behavior we are interested in for \Trans{\text{conf}}
requires that \Trans{b} be executed first.  Thus, if there is a
conflict between \Trans{b} and \Trans{\text{conf}}, we pass \Trans{b} as the
conflict transition parameter to the recursive calls
to \textsc{UpdateBacktrackSetDFS}\xspace{}.
}
\rahmadi{
Algorithm~\ref{alg:backtrack-updatev2} presents the \textsc{UpdateBacktrackSet}\xspace{} procedure, which sets backtracking points.  There are two differences between our algorithm and traditional DPOR algorithms.  First, since our algorithm supports programs with cyclic state spaces, it is possible that the algorithm has discovered multiple paths from the start state $\State{0}$ to the current transition $\Trans{}$.  Thus, the algorithm must potentially set backtracking points on multiple different paths.  We address this issue using a backwards depth first search traversal of the state transition graph $\RGraph{}$.   Second, since our transitions correspond to events they may potentially access multiple different memory locations and thus the backtracking algorithm potentially needs to set separate backtracking points for each of these memory locations.

The \textsc{UpdateBacktrackSetDFS}\xspace{} procedure implements a backwards
depth first traversal to set backtracking points.  The procedure takes
the following parameters: \Trans{\text{curr}} is the current transition in the
DFS, \Event{\text{conf}} is the event that we are currently setting
a backtracking point for, \Access{} is the set of accesses that the
algorithm searches for conflicts for, and \Transitions{\text{explored}} is
the set of transitions that the algorithm has explored down this
search path.  Recall that accesses consist of both an operation, \hbox{\emph{i.e.},}\xspace a read or write, and a memory location.  Conflicts are defined in the usual way---writes to a memory location conflict with reads or writes to the same memory location.

Line~\ref{lin:updatestatesummary} calls the \textsc{UpdateStateSummary}\xspace{} procedure to record \Event{\text{conf}} and the accesses set \Access{} in \Summary{}.
Line~\ref{lin:loop} loops over each transition \Trans{b} that
immediately precedes transition \Trans{\text{curr}} in the state transition
graph and has not been explored.  Line~\ref{lin:conflict} checks for
conflicts between the accesses of \Trans{b} and the accesses
set \Access{} for the DFS.  If a conflict is detected, the
algorithm adds the event for transition \Trans{\text{conf}} to the backtracking
set.  Line~\ref{lin:newaccesses} removes the accesses that conflicted
with transition \Trans{b}.  The search procedure then recursively
calls itself.  If the current transition \Trans{b} conflicts with the
transition \Trans{\text{conf}} for which we are setting a backtracking point, then
it is possible that the behavior we are interested in for \Trans{\text{conf}}
requires that \Trans{b} be executed first.  Thus, if there is a
conflict between \Trans{b} and \Trans{\text{conf}}, we pass \Trans{b} as the
conflict transition parameter to the recursive calls
to \textsc{UpdateBacktrackSetDFS}\xspace{}.
}

\mysubsection{Example\label{sec:example}}
Next, we illustrate how our new stateful DPOR algorithm works with the example from
Figure~\ref{fig:example-missing-execution}.

First, the algorithm starts by calling the \textsc{ExploreAll}\xspace procedure in
Algorithm~\ref{alg:sdpormain}. After the state is initialized, it calls the \textsc{Explore}\xspace procedure in line~\ref{line:firstexplore}.
Let us suppose that the algorithm first explores the execution shown in
Figure~\ref{fig:example-missing-execution}-a.
Line~\ref{line:evselect} in Algorithm~\ref{alg:sdpor} chooses
event \code{e$_1$} among the enabled events.
Next, event \code{e$_1$} is executed: it produces a new state $\State{1}$ and adds
the corresponding transition to \RGraph{}.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure,
but it does not update the backtrack set of any state since the algorithm has only
explored one event.
Line~\ref{line:termination-condition} evaluates to \emph{false} since the termination
condition is not met yet: $\State{1}$ is not found in \hbox{$\mathcal{H}$}\xspace and
it is not a full cycle.
Line~\ref{line:cycle-current-exec-1} also evaluates to \emph{false} since $\State{1}$
is not found in \Trace{}.
Finally, Algorithm~\ref{alg:sdpormain} calls the \textsc{Explore}\xspace procedure recursively to
explore the next transition.

In the subsequent transitions, line~\ref{line:evselect} chooses event \code{e$_2$} and then event \code{e$_3$}.
For each of these events, the \textsc{UpdateBacktrackSet}\xspace{} procedure invokes
\textsc{UpdateBacktrackSetDFS}\xspace{} in Algorithm~\ref{alg:backtrack-updatev2} to
perform the DFS, find conflicts, and update the corresponding backtrack
sets---it puts event \code{e$_2$} in the
backtrack set of $\State{0}$ and event \code{e$_3$} in the backtrack set of
$\State{1}$ as it finds conflicts between events \code{e$_1$}
and \code{e$_2$} that access the shared variable \code{z}, and events \code{e$_2$} and \code{e$_3$} that access the shared variable \code{y}.
Event \code{e$_2$} produces a new state $\State{2}$, while \code{e$_3$}
causes the execution to revisit state $\State{1}$.
At this point, line~\ref{line:cycle-current-exec-1} evaluates
to \emph{true} and the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{}
procedure.
This procedure (shown in Algorithm~\ref{alg:backtrack-update-graph}) finds that
the transition that corresponds to event \code{e$_2$} is reachable from
the transition that corresponds to event \code{e$_3$} in \RGraph{} and calls
the \textsc{UpdateBacktrackSet}\xspace{} procedure.
\textsc{UpdateBacktrackSet}\xspace{} puts event \code{e$_2$} in the backtrack
set of $\State{2}$ as it finds a conflict between the events \code{e$_2$} and \code{e$_3$}
that access the shared variable \code{y}.

In the final transition of the first execution, line~\ref{line:evselect} chooses event \code{e$_4$}.
The \textsc{UpdateBacktrackSet}\xspace{} procedure finds a conflict between \code{e$_4$} and \code{e$_1$} that both access the shared variable \code{x}:
it puts event \code{e$_4$} in the backtrack set of state $\State{0}$.
Event \code{e$_4$} directs the algorithm to revisit state $\State{0}$. This time
line~\ref{line:termination-condition} evaluates to \emph{true} as the termination
condition is satisfied.
The algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure---it finds
that the events \code{e$_1$}, \code{e$_2$}, and \code{e$_3$} are reachable from the transition for event \code{e$_4$}.
Next, it calls the \textsc{UpdateBacktrackSet}\xspace{} procedure for the reachable
transitions. Among all the conflicts found by the procedure, it
puts event \code{e$_1$} in the backtrack set of state $\State{1}$ as a new backtracking point
for the conflict between the events \code{e$_1$} and \code{e$_4$}.
Then, the algorithm saves the visited states in this execution into \hbox{$\mathcal{H}$}\xspace
and terminates the execution.

Now the algorithm explores the executions in the backtrack sets.
This time, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses
the events \code{e$_1$} and \code{e$_3$} in the backtrack set of $\State{1}$, and
event \code{e$_2$} in the backtrack set of $\State{2}$.
These executions terminate quickly as they immediately revisit states
$\State{1}$ and $\State{2}$ that have been stored in \hbox{$\mathcal{H}$}\xspace
at the end of the first execution---line~\ref{line:termination-condition} evaluates to \emph{true}.
These executions are shown in
Figures~\ref{fig:example-missing-execution}-b, c, and d.

Next, let us suppose that the algorithm explores the execution from event \code{e$_2$}
in the backtrack set of $\State{0}$ shown in Figure~\ref{fig:example-missing-execution}-e.
At first, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses event \code{e$_2$} in the backtrack set of $\State{0}$ that produces a new state $\State{3}$.
When line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{}
procedure, this invocation allows the algorithm to
perform the DFS backward to find conflicts between event \code{e$_2$} and the transitions in
previous executions.
Thus, it finds a conflict between the current event \code{e$_2$} and event \code{e$_3$} from the previous execution
shown in Figures~\ref{fig:example-missing-execution}-b---both access the shared
variable \code{y}. It then puts event \code{e$_2$} into the backtrack set of state $\State{1}$.
Since lines~\ref{line:termination-condition} and~\ref{line:cycle-current-exec-1}
evaluate to \emph{false}, the algorithm calls the \textsc{Explore}\xspace{} procedure recursively.
In the second recursion, let us suppose that event \code{e$_3$} is chosen.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure:
it finds a conflict between events \code{e$_3$} and \code{e$_2$} and puts event \code{e$_3$}
in the backtrack set of $\State{0}$.
Since event \code{e$_3$} directs the execution to revisit $\State{0}$ that is already in \hbox{$\mathcal{H}$}\xspace,
line~\ref{line:termination-condition} evaluates to \emph{true} and
the algorithm invokes the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.
This procedure finds a conflict between event \code{$e_2$} of the current execution as a
reachable transition from $\State{0}$ and event \code{$e_3$} from $\State{3}$ that was just executed; it next puts event \code{e$_2$} into the backtrack set of
$\State{3}$ as a new backtracking point.  At this point, this execution terminates.

The algorithm explores executions from the remaining backtracking points,
namely, executions from event \code{e$_2$} in the backtrack set of $\State{3}$, and
\code{e$_3$} and \code{e$_4$} in the backtrack set of $\State{0}$.
These executions are shown in Figures~\ref{fig:example-missing-execution}-f, g, and h.  In the execution of event \code{e$_4$} from state $\State{0}$, the algorithm detects a conflict with the prior execution of event \code{e$_3$} from state $\State{3}$; it puts event \code{e$_4$} into the backtrack set of $\State{3}$.

The algorithm then explores the execution shown in Figure~\ref{fig:example-missing-execution}-i, which contains the assertion. The model checker stops its exploration and returns an error.

\mysubsection{Optimizing Traversals}

\todo{Are we keeping this section? Limit is 20 pages!}

The algorithm as presented performs graph traversals to identify conflicts.  In our implementation atop the Java Pathfinder model checker, the overhead of these graph traversals was small compared to the overhead of Java Pathfinder itself and thus we did not implement these optimizations.  In other settings, it is possible that graph traversals could introduce a sufficient overhead such that optimizations would be warranted.  We next discuss potential optimizations to reduce traversal overheads.

Procedure \textsc{UpdateBacktrackSet}\xspace{} performs a graph traversal to find conflicts with the  transition \Trans{}.  If the path from the start state to the current transition only contains states with one incoming edge, then the traversal is not necessary.  Instead, the model checker can use a common approach of augmenting each shared memory locations with shadow information that records the last read and write accesses~\cite{flanagan2005dynamic}.  The \textsc{UpdateBacktrackSet}\xspace{} procedure can use this information to immediately determine the last conflicting transitions for a memory location. Even if the path has states with more than one incoming edge, this approach can be used to detect conflicts that occur after the most recent such state.  If there may be conflicts before the most recent state with more than one incoming edge, then the procedure would have to fall back to the existing graph traversal.

The procedure \textsc{UpdateBacktrackSetsFromGraph}\xspace{} traverses the graph after a state match to discover potential conflicting transactions from previous executions.  Many of these graph traversals could be eliminated by caching the results of the graph search the first time \textsc{UpdateBacktrackSetsFromGraph}\xspace is called.  The results can be cached as a summary
for each state of the potentially conflicting transitions that are reachable from the given state.  This cached summary can be updated by \textsc{UpdateBacktrackSet}\xspace{} during its backwards graph traversal.  Note that after \textsc{UpdateBacktrackSetsFromGraph}\xspace is called on a state, that state has at least two incoming transitions. Hence, even the previous optimizations to the \textsc{UpdateBacktrackSet}\xspace{} procedure need to implement a graph traversal for that state to update the cached information.

\mysubsection{Discussion}

Although the algorithm we present is designed to explore executions of event-driven systems, the key ideas are also applicable to model checking multithreaded programs.  To the best of our knowledge, there are no correct algorithms for stateful dynamic partial order reduction for multithreaded applications that potentially do not terminate.  The proposed technique bridges the gap between stateful model checking and DPOR, allowing easy adaptation of many existing DPOR algorithms proposed for stateless model checking~\cite{tasharofi2012transdpor,odpor,statelessmcr,chatterjee2019valuecentricdpor,jensen2015stateless} to the stateful context.

\section{Correctness\label{sec:proof}}

This section proves the correctness of our stateful DPOR algorithm with the following proof strategy: we first prove that the
stateless analog of our dynamic partial order reduction algorithm is correct in
Theorem~\ref{thm:stateless}.  We then prove that the stateful
algorithm explores all transitions that the stateless algorithm
explores.

Given a transition sequence $\Trace{}$, we use the following notations in our proof:
\begin{itemize}
 \item $\lvert \Trace{} \rvert$ returns the length of $\Trace{}$.
 \item $\Trace{i}$ denotes the $i$-th transition in $\Trace{}$, and $\Trace{i...j}$ denotes the subsequence of $\Trace{}$ from the $i$-th position to the $j$-th position.
\end{itemize}

We also define the notion of transitive dependence as follows:
\begin{definition}[Transitive Dependence]
Given a transition sequence $\tau$:
$\State{0} \xrightarrow{\Trans{1}} \State{1} \xrightarrow{\Trans{2}} \State{2} ... \xrightarrow{\Trans{n-1}} \State{n-1} \xrightarrow{\Trans{n}} \State{n}$,
the transitive dependence relation $\rightarrow_{\tau}$ is the smallest partial
order on $\tau$ such that if $i < j$ and $\Trans{i}$ is dependent on $\Trans{j}$, then
$\Trans{i} \rightarrow_{\tau} \Trans{j}$.
\end{definition}

The notion of transitive dependence enables partial order reduction.
The transition sequence $\tau$ is one linearization of the partial order
$\rightarrow_{\tau}$.  A search algorithm only needs to explore one linearization,
because other linearizations of $\rightarrow_{\tau}$ yield
``equivalent'' transition sequences of $\tau$, which can be obtained
by swapping adjacent independent transitions.

\mysubsection{Correctness of the Stateless Algorithm}
For the stateless analog of the algorithm, the \code{if} conditions
in lines~\ref{line:termination-condition} and
\ref{line:cycle-current-exec-1} of Algorithm~\ref{alg:sdpor} are always false, and the data structures
\Trace{} and \hbox{$\mathcal{H}$}\xspace in Algorithm~\ref{alg:sdpormain} and~\ref{alg:sdpor} are \emph{don't-care} terms.
When all events in $\hbox{\textit{backtrack}}\xspace(\State{})$ have been explored, the search from $\State{}$ is over, and we say that the state $\State{}$ is ``backtracked''.

\begin{theorem}
\label{thm:stateless}
Let $\Events{}$ be a finite set of events, $\tau$ be a transition sequence
from the initial state $\State{0}$ explored by the stateless analog of
the algorithm in an acyclic transition
system, and $\State{0} \xrightarrow{\tau} \State{}$.  Then, when $\State{}$ is backtracked,
\begin{enumerate}
  \item the set of events that have been explored from $\State{}$ is a persistent set in $\State{}$, and
  \item every trace $\tau \cdot \lambda$ in the state space of $A_G$ is a prefix
    of a linearization of $\rightarrow_{\tau \cdot \lambda'}$ for
    some explored trace $\tau \cdot \lambda'$.
\end{enumerate}
\end{theorem}

\begin{proof}
Acyclicity implies that the system
satisfies the descending chain condition, and we can apply well-founded induction.
We need to prove the statements for the trace
$\State{0} \xrightarrow{\tau} \State{}$,
assuming that the statements hold for all longer traces
$\State{0} \xrightarrow{\tau} \State{} \xrightarrow{\tau'} \State{}'$,
where $\State{}'$ is reachable from $\State{}$ in finite steps.

Let $T := \hbox{\textit{backtrack}}\xspace(\State{})$.  For the first statement, \hbox{\emph{i.e.},}\xspace
$T$ is a persistent set when $\State{}$ is backtracked, we will prove it by contradiction.
If $T$ is not a persistent set, then there must exist a transition sequence from $\State{}$,
\[ \State{} \xrightarrow{\Trans{1}} \State{1} \xrightarrow{\Trans{2}} \State{2} ... \xrightarrow{\Trans{n-1}} \State{n-1} \xrightarrow{\Trans{n}} \State{n}, \]
where $\hbox{\emph{event}}\xspace{(\Trans{1})}, ..., \hbox{\emph{event}}\xspace{(\Trans{n})} \notin T$,
events $\hbox{\emph{event}}\xspace{(\Trans{1})}, ..., \hbox{\emph{event}}\xspace{(\Trans{n-1})}$ are read-write
independent with all events in $T$,
and $\hbox{\emph{event}}\xspace{(\Trans{n})}$ is dependent with some $x \in T$ at state $\State{n-1}$.
Select the shortest such transition sequence.
Let $\sigma$ be the transition sequence $\Trans{1} ... \Trans{n-1}$.
For simplicity, we will label $\hbox{\emph{event}}\xspace(\Trans{i})$ as $\tildeEvent{i}$ for $1 \leq i \leq n$.
Note that $\tildeEvent{i}$ and $\tildeEvent{j}$ could be the same event even if $i \neq j$.

Since $x$ is read-write independent with $\tildeEvent{i}$ for all
$1 \leq i \leq n-1$, we have the following transition diagram:
\[
\xymatrix{ \State{}' \ar[r]^{\sigma} & \State{n-1}' \\
           \State{} \ar[r]^{\sigma} \ar[u]^{x} & \State{n-1} \ar[u]^{x} }
\]

There are two cases: (1) $\tildeEvent{n}$ is enabled in $\State{}$;
and (2) $\tildeEvent{n}$ is disabled in $\State{}$.

\vspace{.3em}
\textbf{Case 1:} Suppose that $\tildeEvent{n}$ is enabled
in $\State{}$ but disabled in $\State{}'$.
Since $x \in T$ and $\State{} \xrightarrow{x} \State{}'$ has been explored
when $\State{}$ is backtracked, the \code{for} loop at line~\ref{line:add-disabled}
of Algorithm~\ref{alg:sdpor} will add $\tildeEvent{n}$ to $T$, contradicting the
assumption that $\tildeEvent{n} \notin T$.

Suppose that $\tildeEvent{n}$ is enabled in $\State{}$ and $\State{}'$.
Then, $x$ does not enable or disable $\tildeEvent{n}$.  Since $x$
is read-write independent with all corresponding events in $\sigma$,
$\tildeEvent{n}$ is also enabled in $\State{n-1}'$.
Recall that $\tildeEvent{n}$ is dependent with $x$ at $\State{n-1}$.
Since $x$ does not enable or disable $\tildeEvent{n}$,
$x$ and $\tildeEvent{n}$ must have
conflicting memory accesses $\Access{x, n}$ in the transition
sequence $\tau \cdot \sigma \cdot \Trans{x} \cdot \Trans{n}$, which is equivalent to
the transition sequence $\tau \cdot \Trans{x} \cdot \sigma \cdot \Trans{n}$,
where $\hbox{\emph{event}}\xspace({\Trans{x}}) = x$.
Since $x \in T$ and by the inductive assumption,
we have explored some transition sequence $\tilde{\tau}$ where
some linearization of $\rightarrow_{\tilde{\tau}}$ has prefix
$\tau \cdot \Trans{x} \cdot \sigma \cdot \Trans{n}$.
Thus, line~\ref{line:call-update-backtrack-set} in Algorithm~\ref{alg:sdpor}
would have called $\textsc{UpdateBacktrackSet}\xspace{}$ and found the conflicting memory
accesses $\Access{x, n}$, which is not empty until some
$\tildeEvent{i}$ is added to $T$.  Thus, we have a contradiction.

\vspace{.3em}
\textbf{Case 2:} Suppose that $\tildeEvent{n}$ is disabled in $\State{}$.
Since $\tildeEvent{n}$ is enabled in $\State{n-1}$,
there exists some transition $\Trans{i}$ in $\sigma$ that writes to the
shadow location of $\tildeEvent{n}$.  Since $x$ is
read-write independent with $\tildeEvent{1}, ...,
\tildeEvent{n-1}$ by assumption,
$x$ does not write to the shadow location of $\tildeEvent{n}$,
\hbox{\emph{i.e.},}\xspace $x$ does not enable or disable $\tildeEvent{n}$.
Then, $\tildeEvent{n}$ is also enabled in $\State{n-1}'$.
Therefore, the same argument in the second paragraph of Case 1 applies,
and we have a contradiction. \\

We have shown that $T$ is a persistent set.
We now move to prove the second statement in this theorem, that every trace
$\State{0} \xrightarrow{\tau} \State{} \xrightarrow{\lambda} u$ is a prefix of a linearization
of $\rightarrow_{\tau \cdot \lambda'}$ for some explored trace $\tau \cdot \lambda'$.
If $\lambda$ is the null sequence, then we are done,
because $\State{0} \xrightarrow{\tau} \State{}$
is already explored.

Let $\lambda = \pi \cdot \eta$, where $\pi$ is the maximal transition
sequence such that no events executed in $\pi$ is in $T$.
If $\eta$ is not the null sequence, then the first transition in $\eta$,
denoted by $\eta_1$, is in $T$.
Since $T$ is a persistent set, $\eta_1$ is independent with all transitions in
$\pi$.  Thus, $\tau \cdot \pi \cdot \eta$ is equivalent to
$\tau \cdot \eta_1 \cdot \pi \cdot \eta_{2...\lvert \eta \rvert}$.
Note that $\tau \cdot \eta_1$ is a transition sequence
that is explored by the algorithm.  Thus, by the inductive assumption,
$\tau \cdot \eta_1 \cdot \pi \cdot \eta_{2...\lvert \eta \rvert}$
is a prefix of a linearization of $\rightarrow_{\tau \cdot \eta_1 \cdot \lambda'}$
for some explored trace $\tau \cdot \eta_1 \cdot \lambda'$.
Then, $\tau \cdot \lambda = \tau \cdot \pi \cdot \eta$ is a prefix of
another linearization of $\rightarrow_{\tau \cdot \eta_1 \cdot \lambda'}$.
The same argument holds if $\pi$ is the null sequence.

Let $\eta$ be the null sequence.  Pick $y \in T$.
Then, $\tau \cdot \Trans{y}$ is an explored transition sequence,
where $\hbox{\emph{event}}\xspace(\Trans{y}) = y$.
The inductive assumption implies that $\tau \cdot \Trans{y} \cdot \pi$
is a prefix of a linearization of $\rightarrow_{\tau \cdot t_y \cdot \lambda'}$
for some explored trace $\tau \cdot \Trans{y} \cdot \lambda'$.
Because $\Trans{y}$ is independent with all transitions in $\pi$, $\tau \cdot \pi \cdot \Trans{y}$
is also a prefix of some linearization of $\rightarrow_{\tau \cdot t_y \cdot \lambda'}$.
Hence, $\tau \cdot \pi$ is a prefix of some linearization of
$\rightarrow_{\tau \cdot t_y \cdot \lambda'}$ as well.
\end{proof}

\section*{Appendices}

\title{Appendices}
\author{}
\institute{}
\maketitle

\renewcommand\thefigure{A.\arabic{figure}}
\renewcommand\thetable{A.\arabic{table}}
\setcounter{figure}{0}
\setcounter{table}{0}

\mysection{Example Event-Driven System\label{sec:example-event-driven-system}}
Figure~\ref{fig:concurrency-model} depicts components of an example
event-driven system.  The application \code{SmartLightApp} in the
figure is developed using this concurrency model and may run in the
cloud.  It has a fixed set of events that are associated with it and
their respective event handlers. It also has logic to decide whether
to perform specific actions based on the triggering events.  For
example, \code{SmartLightApp} can turn on and off a light bulb based
on specific triggers.

\begin{figure}[t]
    \centering
	\includegraphics[width=0.8\linewidth]{figures/concurrency-model.pdf}

    \caption{Example Event-driven System.
      \label{fig:concurrency-model}}

\end{figure}

Our example event-driven system has the following 4 components:

\noindent \textbf{(I) Event handlers:} The application code is
composed of a set of event handlers.  Each event handler processes a
certain class of events.  When the event handler executes, it receives
the appropriate event object.  When the runtime system dequeues an event $\Event{}$, it executes the relevant event handler.

\noindent \textbf{(II) External events:}
An application can have a number of sources that deliver triggering events:
(1) a cloud service
(\hbox{\emph{e.g.},}\xspace a cloud service that reports the current outdoor temperature),
(2) a service that is running on a computing device
(\hbox{\emph{e.g.},}\xspace a messaging service that sends a trigger whenever there is an incoming message),
(3) a device
(\hbox{\emph{e.g.},}\xspace an illuminance sensor that sends a trigger whenever there is a change of
light intensity in its environment),
(4) a scheduled event within the application
(\hbox{\emph{e.g.},}\xspace turn on a light bulb at 6pm), or
(5) an event generated by the application that triggers another component of that app:
these events potentially change the app's state (\hbox{\emph{e.g.},}\xspace a scheduled mode change to \textit{night} may trigger a light bulb to turn on).

\noindent \textbf{(III) Internal events:}
An application can directly generate events, for example to:
(a) actuate a certain device
(\hbox{\emph{e.g.},}\xspace turn on a light bulb) or
(b) trigger another app running on a computing device
(\hbox{\emph{e.g.},}\xspace trigger an email to be sent from an email app).

\noindent \textbf{(IV) Event queue:}
All events are pushed into an event queue to be delivered by the runtime to the appropriate event handlers.

Examples of platforms with this event-driven model include the Samsung SmartThings platform~\cite{smartthings,smartthings-github}, a popular platform used to create
apps that control devices to automate tasks for a smart home.

\input{proof}

\input{proof2}

\mysection{Example for Stateful DPOR Algorithm\label{sec:example}}
Here, we illustrate how our new stateful DPOR algorithm works with the example from
Figure~\ref{fig:example-missing-execution}.

First, the algorithm starts by calling the \textsc{ExploreAll}\xspace procedure in
Algorithm~\ref{alg:sdpormain}. After the state is initialized, it calls the \textsc{Explore}\xspace procedure in line~\ref{line:firstexplore}.
Let us suppose that the algorithm first explores the execution shown in
Figure~\ref{fig:example-missing-execution}-a.
Line~\ref{line:evselect} in Algorithm~\ref{alg:sdpor} chooses
event \code{e$_1$} among the enabled events.
Next, event \code{e$_1$} is executed: it produces a new state $\State{1}$ and adds
the corresponding transition to \RGraph{}.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure,
but it does not update the backtrack set of any state since the algorithm has only
explored one event.
Line~\ref{line:termination-condition} evaluates to \emph{false} since the termination
condition is not met yet: $\State{1}$ is not found in \hbox{$\mathcal{H}$}\xspace and
it is not a full cycle.
Line~\ref{line:cycle-current-exec-1} also evaluates to \emph{false} since $\State{1}$
is not found in \Trace{}.
Finally, Algorithm~\ref{alg:sdpormain} calls the \textsc{Explore}\xspace procedure recursively to
explore the next transition.

In the subsequent transitions, line~\ref{line:evselect} chooses event \code{e$_2$} and then event \code{e$_3$}.
For each of these events, the \textsc{UpdateBacktrackSet}\xspace{} procedure invokes
\textsc{UpdateBacktrackSetDFS}\xspace{} in Algorithm~\ref{alg:backtrack-updatev2} to
perform the DFS, find conflicts, and update the corresponding backtrack
sets---it puts event \code{e$_2$} in the
backtrack set of $\State{0}$ and event \code{e$_3$} in the backtrack set of
$\State{1}$ as it finds conflicts between events \code{e$_1$}
and \code{e$_2$} that access the shared variable \code{z}, and events \code{e$_2$} and \code{e$_3$} that access the shared variable \code{y}.
Event \code{e$_2$} produces a new state $\State{2}$, while \code{e$_3$}
causes the execution to revisit state $\State{1}$.
At this point, line~\ref{line:cycle-current-exec-1} evaluates
to \emph{true} and the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{}
procedure.
This procedure (shown in Algorithm~\ref{alg:backtrack-update-graph}) finds that
the transition that corresponds to event \code{e$_2$} is reachable from
the transition that corresponds to event \code{e$_3$} in \RGraph{} and calls
the \textsc{UpdateBacktrackSet}\xspace{} procedure.
\textsc{UpdateBacktrackSet}\xspace{} puts event \code{e$_2$} in the backtrack
set of $\State{2}$ as it finds a conflict between the events \code{e$_2$} and \code{e$_3$}
that access the shared variable \code{y}.

In the final transition of the first execution, line~\ref{line:evselect} chooses event \code{e$_4$}.
The \textsc{UpdateBacktrackSet}\xspace{} procedure finds a conflict between \code{e$_4$} and \code{e$_1$} that both access the shared variable \code{x}:
it puts event \code{e$_4$} in the backtrack set of state $\State{0}$.
Event \code{e$_4$} directs the algorithm to revisit state $\State{0}$. This time
line~\ref{line:termination-condition} evaluates to \emph{true} as the termination
condition is satisfied.
The algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure---it finds
that the events \code{e$_1$}, \code{e$_2$}, and \code{e$_3$} are reachable from the transition for event \code{e$_4$}.
Next, it calls the \textsc{UpdateBacktrackSet}\xspace{} procedure for the reachable
transitions. Among all the conflicts found by the procedure, it
puts event \code{e$_1$} in the backtrack set of state $\State{1}$ as a new backtracking point
for the conflict between the events \code{e$_1$} and \code{e$_4$}.
Then, the algorithm saves the visited states in this execution into \hbox{$\mathcal{H}$}\xspace
and terminates the execution.

Now the algorithm explores the executions in the backtrack sets.
This time, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses
the events \code{e$_1$} and \code{e$_3$} in the backtrack set of $\State{1}$, and
event \code{e$_2$} in the backtrack set of $\State{2}$.
These executions terminate quickly as they immediately revisit states
$\State{1}$ and $\State{2}$ that have been stored in \hbox{$\mathcal{H}$}\xspace
at the end of the first execution---line~\ref{line:termination-condition} evaluates to \emph{true}.
These executions are shown in
Figures~\ref{fig:example-missing-execution}-b, c, and d.

Next, let us suppose that the algorithm explores the execution from event \code{e$_2$}
in the backtrack set of $\State{0}$ shown in Figure~\ref{fig:example-missing-execution}-e.
At first, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses event \code{e$_2$} in the backtrack set of $\State{0}$ that produces a new state $\State{3}$.
When line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{}
procedure, this invocation allows the algorithm to
perform the DFS backward to find conflicts between event \code{e$_2$} and the transitions in
previous executions.
Thus, it finds a conflict between the current event \code{e$_2$} and event \code{e$_3$} from the previous execution
shown in Figure~\ref{fig:example-missing-execution}-b---both access the shared
variable \code{y}. It then puts event \code{e$_2$} into the backtrack set of state $\State{1}$.
Since lines~\ref{line:termination-condition} and~\ref{line:cycle-current-exec-1}
evaluate to \emph{false}, the algorithm calls the \textsc{Explore}\xspace{} procedure recursively.
In the second recursion, let us suppose that event \code{e$_3$} is chosen.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure:
it finds a conflict between events \code{e$_3$} and \code{e$_2$} and puts event \code{e$_3$}
in the backtrack set of $\State{0}$.
Since event \code{e$_3$} directs the execution to revisit $\State{0}$ that is already in \hbox{$\mathcal{H}$}\xspace,
line~\ref{line:termination-condition} evaluates to \emph{true} and
the algorithm invokes the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.
This procedure finds a conflict between event \code{$e_2$} of the current execution as a
reachable transition from $\State{0}$ and event \code{$e_3$} from $\State{3}$ that was just executed; it next puts event \code{e$_2$} into the backtrack set of
$\State{3}$ as a new backtracking point.  At this point, this execution terminates.

The algorithm explores executions from the remaining backtracking points,
namely, executions from event \code{e$_2$} in the backtrack set of $\State{3}$, and
\code{e$_3$} and \code{e$_4$} in the backtrack set of $\State{0}$.
These executions are shown in Figures~\ref{fig:example-missing-execution}-f, g, and h.  In the execution of event \code{e$_4$} from state $\State{0}$, the algorithm detects a conflict with the prior execution of event \code{e$_3$} from state $\State{3}$; it puts event \code{e$_4$} into the backtrack set of $\State{3}$.

The algorithm then explores the execution shown in Figure~\ref{fig:example-missing-execution}-i, which contains the assertion. The model checker stops its exploration and returns an error.

\mysection{Optimizing Traversals\label{sec:optimizing-traversals}}

The algorithm as presented performs graph traversals to identify conflicts.  Our implementation implements several optimizations to reduce traversal overheads.

The procedure \textsc{UpdateBacktrackSetsFromGraph}\xspace{} traverses the graph after a state match to discover potential conflicting transactions from previous executions.  We eliminate many of these graph traversals by caching the results of the graph search the first time \textsc{UpdateBacktrackSetsFromGraph}\xspace is called.  The results can be cached for each state as a summary of the potentially conflicting transitions that are reachable from the given state.
A potentially conflicting transition, namely $\Trans{\text{conf}}$, is cached in the form of a tuple that contains the event \hbox{\emph{event}}\xspace{}(\Trans{\text{conf}}) and its corresponding accesses \Access{}.

This cached summary is updated by \textsc{UpdateBacktrackSet}\xspace{} during its backwards graph traversal.
With this summary, the algorithm can efficiently re-explore the previously explored transitions: (1) when a previously discovered state \State{} is reached, the summary contains all potentially conflicting events reachable from \State{}, and (2) when performing backwards depth first search traversal, the algorithm can stop the traversal at any state \State{b} whenever the algorithm finds that the current \hbox{\emph{event}}\xspace{}(\Trans{\text{conf}}) and \Access{} are already cached in the summary for \State{b}.

\begin{table*}[!htb]
  \centering
    \caption{Model-checked pairs that finished with DPOR but unfinished (\hbox{\emph{i.e.},}\xspace "Unf") without DPOR. \textbf{Evt.} is number of events and \textbf{Time} is in seconds.
    \label{tab:results-dnt-no-dpor-complete}}
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        1  & initial-state-event-streamer--thermostat-auto-off                 & 78 & Unf & Unf & Unf & 7,146 & 25,850 & 3,285 \\ \hline
        2  & unbuffered-event-sender--hvac-auto-off.smartapp                   & 78 & Unf & Unf & Unf & 7,123 & 26,016 & 3,432 \\ \hline
        3  & initial-state-event-sender--hvac-auto-off.smartapp                & 78 & Unf & Unf & Unf & 7,007 & 25,220 & 3,215 \\ \hline
        4  & initial-state-event-streamer--hvac-auto-off.smartapp              & 78 & Unf & Unf & Unf & 7,007 & 25,220 & 3,230 \\ \hline
        5  & initialstate-smart-app-v1.2.0--hvac-auto-off.smartapp             & 78 & Unf & Unf & Unf & 7,007 & 25,220 & 3,290 \\ \hline
        6  & lighting-director--circadian-daylight                             & 19 & Unf & Unf & Unf & 6,553 & 33,045 & 6,604 \\ \hline
        7  & initial-state-event-streamer--thermostat                          & 81 & Unf & Unf & Unf & 5,646 & 26,620 & 2,965 \\ \hline
        8  & forgiving-security--unbuffered-event-sender                       & 80 & Unf & Unf & Unf & 5,019 & 45,208 & 6,259 \\ \hline
        9  & forgiving-security--initial-state-event-streamer                  & 80 & Unf & Unf & Unf & 4,902 & 44,230 & 5,697 \\ \hline
        10 & forgiving-security--initialstate-smart-app-v1.2.0                 & 80 & Unf & Unf & Unf & 4,902 & 44,230 & 5,702 \\ \hline
        11 & forgiving-security--initial-state-event-sender                    & 80 & Unf & Unf & Unf & 4,902 & 44,230 & 5,716 \\ \hline
        12 & unbuffered-event-sender--thermostat-window-check                  & 79 & Unf & Unf & Unf & 4,546 & 17,411 & 2,069 \\ \hline
        13 & hue-mood-lighting--Hue-Party-Mode                                 & 18 & Unf & Unf & Unf & 3,457 & 49,138 & 6,132 \\ \hline
        14 & thermostat--initial-state-event-sender                            & 82 & Unf & Unf & Unf & 2,530 & 13,105 & 1,621 \\ \hline
        15 & thermostat--initialstate-smart-app-v1.2.0                         & 82 & Unf & Unf & Unf & 2,530 & 13,105 & 1,606 \\ \hline
        16 & thermostat--unbuffered-event-sender                               & 82 & Unf & Unf & Unf & 2,517 & 12,880 & 1,626 \\ \hline
        17 & initial-state-event-streamer--unlock-it-when-i-arrive             & 79 & Unf & Unf & Unf & 2,459 & 8,825  & 998   \\ \hline
        18 & lights-off-with-no-motion-and-presence--smart-security            & 11 & Unf & Unf & Unf & 2,299 & 11,261 & 1,475 \\ \hline
        19 & initial-state-event-streamer--lock-it-when-i-leave                & 78 & Unf & Unf & Unf & 1,750 & 5,387  & 595   \\ \hline
        20 & smart-nightlight--step-notifier                                   & 13 & Unf & Unf & Unf & 1,568 & 6,460  & 644   \\ \hline
        21 & initial-state-event-sender--NotifyIfLeftUnlocked                  & 78 & Unf & Unf & Unf & 1,482 & 3,830  & 428   \\ \hline
        22 & initial-state-event-streamer--NotifyIfLeftUnlocked                & 78 & Unf & Unf & Unf & 1,482 & 3,830  & 439   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[t]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        23 & initialstate-smart-app-v1.2.0--NotifyIfLeftUnlocked               & 78 & Unf & Unf & Unf & 1,482 & 3,830  & 437   \\ \hline
        24 & initial-state-event-streamer--auto-lock-door.smartapp             & 80 & Unf & Unf & Unf & 1,272 & 3,743  & 422   \\ \hline
        25 & lock-it-when-i-leave--unbuffered-event-sender                     & 78 & Unf & Unf & Unf & 1,234 & 3,482  & 438   \\ \hline
        26 & lock-it-when-i-leave--initial-state-event-sender                  & 78 & Unf & Unf & Unf & 1,192 & 3,369  & 424   \\ \hline
        27 & lock-it-when-i-leave--initialstate-smart-app-v1.2.0               & 78 & Unf & Unf & Unf & 1,192 & 3,369  & 428   \\ \hline
        28 & medicine-management-temp-motion--initial-state-event-sender       & 79 & Unf & Unf & Unf & 939   & 2,277  & 275   \\ \hline
        29 & medicine-management-temp-motion--initialstate-smart-app-v1.2.0    & 79 & Unf & Unf & Unf & 939   & 2,277  & 275   \\ \hline
        30 & medicine-management-temp-motion--unbuffered-event-sender          & 79 & Unf & Unf & Unf & 933   & 2,277  & 283   \\ \hline
        31 & initial-state-event-streamer--DeviceTamperAlarm                   & 80 & Unf & Unf & Unf & 860   & 2,439  & 277   \\ \hline
        32 & NotifyIfLeftUnlocked--unbuffered-event-sender                     & 78 & Unf & Unf & Unf & 811   & 2,181  & 270   \\ \hline
        33 & initial-state-event-streamer--smart-auto-lock-unlock              & 80 & Unf & Unf & Unf & 780   & 1,986  & 228   \\ \hline
        34 & initial-state-event-streamer--medicine-management-contact-sensor  & 78 & Unf & Unf & Unf & 738   & 1,657  & 197   \\ \hline
        35 & unlock-it-when-i-arrive--initial-state-event-sender               & 77 & Unf & Unf & Unf & 622   & 2,402  & 273   \\ \hline
        36 & unlock-it-when-i-arrive--initialstate-smart-app-v1.2.0            & 77 & Unf & Unf & Unf & 622   & 2,402  & 274   \\ \hline
        37 & initial-state-event-streamer--lock-it-at-a-specific-time          & 79 & Unf & Unf & Unf & 621   & 1,451  & 175   \\ \hline
        38 & unlock-it-when-i-arrive--unbuffered-event-sender                  & 77 & Unf & Unf & Unf & 618   & 2,405  & 277   \\ \hline
        39 & medicine-management-contact-sensor--initial-state-event-sender    & 78 & Unf & Unf & Unf & 605   & 1,241  & 156   \\ \hline
        40 & medicine-management-contact-sensor--initialstate-smart-app-v1.2.0 & 78 & Unf & Unf & Unf & 605   & 1,241  & 163   \\ \hline
        41 & DeviceTamperAlarm--initial-state-event-sender                     & 80 & Unf & Unf & Unf & 602   & 1,540  & 206   \\ \hline
        42 & DeviceTamperAlarm--initialstate-smart-app-v1.2.0                  & 80 & Unf & Unf & Unf & 602   & 1,540  & 209   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[t]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        43 & medicine-management-contact-sensor--unbuffered-event-sender       & 78 & Unf & Unf & Unf & 602   & 1,240  & 168   \\ \hline
        44 & DeviceTamperAlarm--unbuffered-event-sender                        & 80 & Unf & Unf & Unf & 600   & 1,534  & 217   \\ \hline
        45 & close-the-valve--initial-state-event-sender                       & 78 & Unf & Unf & Unf & 584   & 1,261  & 162   \\ \hline
        46 & close-the-valve--initial-state-event-streamer                     & 78 & Unf & Unf & Unf & 584   & 1,261  & 164   \\ \hline
        47 & close-the-valve--initialstate-smart-app-v1.2.0                    & 78 & Unf & Unf & Unf & 584   & 1,261  & 162   \\ \hline
        48 & close-the-valve--unbuffered-event-sender                          & 78 & Unf & Unf & Unf & 581   & 1,259  & 172   \\ \hline
        49 & initial-state-event-streamer--medicine-management-temp-motion     & 79 & Unf & Unf & Unf & 549   & 1,298  & 172   \\ \hline
        50 & lock-it-at-a-specific-time--initial-state-event-sender            & 79 & Unf & Unf & Unf & 502   & 1,080  & 141   \\ \hline
        51 & lock-it-at-a-specific-time--initialstate-smart-app-v1.2.0         & 79 & Unf & Unf & Unf & 502   & 1,080  & 140   \\ \hline
        52 & lock-it-at-a-specific-time--unbuffered-event-sender               & 79 & Unf & Unf & Unf & 500   & 1,079  & 146   \\ \hline
        53 & auto-lock-door.smartapp--initial-state-event-sender               & 80 & Unf & Unf & Unf & 498   & 1,617  & 196   \\ \hline
        54 & auto-lock-door.smartapp--initialstate-smart-app-v1.2.0            & 80 & Unf & Unf & Unf & 498   & 1,617  & 194   \\ \hline
        55 & auto-lock-door.smartapp--unbuffered-event-sender                  & 80 & Unf & Unf & Unf & 495   & 1,617  & 202   \\ \hline
        56 & initial-state-event-sender--initialstate-smart-app-v1.2.0         & 78 & Unf & Unf & Unf & 473   & 1,054  & 143   \\ \hline
        57 & initial-state-event-streamer--initial-state-event-sender          & 78 & Unf & Unf & Unf & 473   & 1,054  & 143   \\ \hline
        58 & initial-state-event-streamer--initialstate-smart-app-v1.2.0       & 78 & Unf & Unf & Unf & 473   & 1,054  & 143   \\ \hline
        59 & initial-state-event-sender--unbuffered-event-sender               & 78 & Unf & Unf & Unf & 471   & 1,053  & 147   \\ \hline
        60 & initial-state-event-streamer--unbuffered-event-sender             & 78 & Unf & Unf & Unf & 471   & 1,053  & 147   \\ \hline
        61 & initialstate-smart-app-v1.2.0--unbuffered-event-sender            & 78 & Unf & Unf & Unf & 471   & 1,053  & 145   \\ \hline
        62 & enhanced-auto-lock-door--initial-state-event-sender               & 80 & Unf & Unf & Unf & 309   & 1,139  & 158   \\ \hline
        63 & enhanced-auto-lock-door--initial-state-event-streamer             & 80 & Unf & Unf & Unf & 309   & 1,139  & 158   \\ \hline
        64 & enhanced-auto-lock-door--initialstate-smart-app-v1.2.0            & 80 & Unf & Unf & Unf & 309   & 1,139  & 162   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[t]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        65 & smart-auto-lock-unlock--initial-state-event-sender                & 80 & Unf & Unf & Unf & 309   & 1,140  & 159   \\ \hline
        66 & smart-auto-lock-unlock--initialstate-smart-app-v1.2.0             & 80 & Unf & Unf & Unf & 309   & 1,140  & 161   \\ \hline
        67 & enhanced-auto-lock-door--unbuffered-event-sender                  & 80 & Unf & Unf & Unf & 307   & 1,139  & 164   \\ \hline
        68 & smart-auto-lock-unlock--unbuffered-event-sender                   & 80 & Unf & Unf & Unf & 307   & 1,140  & 165   \\ \hline
        69 & lighting-director--turn-on-before-sunset                          & 11 & Unf & Unf & Unf & 257   & 960    & 77    \\
    \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
    \caption{Model-checked pairs that finished with or without DPOR. \textbf{Evt.} is number of events and \textbf{Time} is in seconds.
    \label{tab:results-both-terminated-complete}}
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        1  & smart-nightlight--ecobeeAwayFromHome                              & 14 & 16,441 & 76,720  & 5,059 & 11,743 & 46,196 & 5,498 \\ \hline
        2  & step-notifier--ecobeeAwayFromHome                                 & 11 & 14,401 & 52,800  & 4,885 & 11,490 & 35,142 & 5,079 \\ \hline
        3  & smart-security--ecobeeAwayFromHome                                & 11 & 14,301 & 47,608  & 4,385 & 8,187  & 21,269 & 2,980 \\ \hline
        4  & keep-me-cozy--whole-house-fan                                     & 17 & 8,793  & 149,464 & 4,736 & 8,776  & 95,084 & 6,043 \\ \hline
        5  & keep-me-cozy-ii--thermostat-window-check                          & 13 & 8,764  & 113,919 & 4,070 & 7,884  & 59,342 & 4,515 \\ \hline
        6  & step-notifier--mini-hue-controller                                & 6  & 7,967  & 47,796  & 2,063 & 7,907  & 40,045 & 3,582 \\ \hline
        7  & keep-me-cozy--thermostat-mode-director                            & 12 & 7,633  & 91,584  & 3,259 & 6,913  & 49,850 & 3,652 \\ \hline
        8  & lighting-director--step-notifier                                  & 14 & 7,611  & 106,540 & 5,278 & 2,723  & 25,295 & 2,552 \\ \hline
        9  & smart-alarm--DeviceTamperAlarm                                    & 15 & 5,665  & 84,960  & 3,559 & 3,437  & 40,906 & 4,441 \\ \hline
        10 & forgiving-security--smart-alarm                                   & 13 & 5,545  & 72,072  & 3,134 & 4,903  & 52,205 & 5,728 \\ \hline
        11 & smart-light-timer-x-minutes-unless-already-on--ecobeeAwayFromHome & 9  & 3,775  & 11,160  & 992   & 2,460  & 5,418  & 645   \\ \hline
        12 & smart-security--vacation-lighting-director                        & 12 & 3,759  & 33,264  & 1,641 & 2,849  & 14,108 & 1,604 \\ \hline
        13 & smart-security--turn-on-only-if-i-arrive-after-sunset             & 11 & 3,437  & 28,028  & 1,471 & 2,553  & 11,624 & 1,396 \\ \hline
        14 & smart-security--turn-it-on-when-im-here                           & 11 & 3,437  & 28,028  & 1,470 & 2,549  & 11,878 & 1,401 \\ \hline
        15 & vacation-lighting-director--ecobeeAwayFromHome                    & 10 & 3,313  & 11,040  & 856   & 2,158  & 5,779  & 652   \\ \hline
        16 & smart-light-timer-x-minutes-unless-already-on--step-notifier      & 10 & 3,213  & 32,120  & 2,402 & 2,343  & 11,149 & 1,122 \\ \hline
        17 & thermostat--thermostat-mode-director                              & 12 & 3,169  & 38,016  & 1,454 & 3,157  & 28,437 & 2,470 \\ \hline
        18 & ecobeeAwayFromHome--NotifyIfLeftUnlocked                          & 9  & 2,329  & 6,984   & 623   & 1,714  & 377    & 454   \\ \hline
        19 & keep-me-cozy--thermostat-window-check                             & 14 & 2,185  & 30,576  & 1,148 & 2,176  & 20,458 & 1,576 \\ \hline
        20 & smart-security--turn-on-before-sunset                             & 10 & 2,175  & 16,240  & 855   & 1,588  & 6,119  & 782   \\ \hline
        21 & smart-security--turn-on-at-sunset                                 & 10 & 2,175  & 16,240  & 909   & 1,542  & 5,599  & 783   \\ \hline
        22 & keep-me-cozy--thermostat                                          & 12 & 2,017  & 519,960 & 801   & 2,017  & 15,593 & 1,193 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        23 & smart-security--turn-it-on-when-it-opens                          & 10 & 1,763  & 12,760  & 700   & 1,340  & 5,913  & 737   \\ \hline
        24 & smart-security--undead-early-warning                              & 10 & 1,763  & 12,760  & 688   & 1,340  & 5,913  & 732   \\ \hline
        25 & photo-burst-when--ecobeeAwayFromHome                              & 13 & 1,409  & 4,576   & 599   & 1,109  & 2,695  & 627   \\ \hline
        26 & auto-lock-door.smartapp--ecobeeAwayFromHome                       & 11 & 1,381  & 4,048   & 373   & 927    & 2,357  & 294   \\ \hline
        27 & lighting-director--vacation-lighting-director                     & 13 & 1,373  & 17,836  & 623   & 782    & 4,943  & 395   \\ \hline
        28 & let-there-be-dark--smart-security                                 & 9  & 1,279  & 8,460   & 494   & 881    & 3,531  & 413   \\ \hline
        29 & thermostat--whole-house-fan                                       & 13 & 1,273  & 16,536  & 690   & 747    & 8,237  & 790   \\ \hline
        30 & let-there-be-dark--ecobeeAwayFromHome                             & 7  & 1,240  & 3,052   & 289   & 924    & 1,895  & 263   \\ \hline
        31 & forgiving-security--smart-security                                & 11 & 1,165  & 7,524   & 620   & 696    & 2,838  & 547   \\ \hline
        32 & lighting-director--smart-light-timer-x-minutes-unless-already-on  & 12 & 1,068  & 12,804  & 465   & 424    & 3,104  & 271   \\ \hline
        33 & smart-security--turn-off-with-motion                              & 9  & 989    & 6,732   & 371   & 732    & 3,050  & 394   \\ \hline
        34 & whole-house-fan--hvac-auto-off.smartapp                           & 9  & 958    & 8,613   & 296   & 935    & 6,852  & 722   \\ \hline
        35 & thermostat-auto-off--whole-house-fan                              & 9  & 958    & 8,613   & 286   & 660    & 5,377  & 424   \\ \hline
        36 & smart-security--turn-on-by-zip-code                               & 9  & 941    & 6,300   & 363   & 705    & 2,855  & 386   \\ \hline
        37 & laundry-monitor--step-notifier                                    & 6  & 873    & 3,954   & 263   & 170    & 327    & 54    \\ \hline
        38 & smart-security--DeviceTamperAlarm                                 & 11 & 872    & 7,546   & 371   & 593    & 2,939  & 376   \\ \hline
        39 & make-it-so--whole-house-fan                                       & 13 & 841    & 10,920  & 409   & 751    & 7,436  & 599   \\ \hline
        40 & step-notifier--BetterLaundryMonitor                               & 6  & 810    & 4,854   & 260   & 453    & 1,283  & 144   \\ \hline
        41 & thermostat-auto-off--thermostat-mode-director                     & 10 & 763    & 7,620   & 281   & 714    & 5,805  & 490   \\ \hline
        42 & medicine-management-temp-motion--circadian-daylight               & 14 & 756    & 10,500  & 431   & 350    & 3,133  & 293   \\ \hline
        43 & smart-nightlight--turn-it-on-when-im-here                         & 11 & 736    & 8,085   & 304   & 238    & 1,085  & 110   \\ \hline
        44 & smart-nightlight--turn-on-only-if-i-arrive-after-sunset           & 11 & 736    & 8,085   & 309   & 238    & 1,085  & 109   \\ \hline
        45 & lighting-director--turn-on-at-sunset                              & 11 & 734    & 8,063   & 294   & 241    & 927    & 80    \\ \hline
        46 & unlock-it-when-i-arrive--ecobeeAwayFromHome                       & 8  & 733    & 2,048   & 192   & 543    & 1,179  & 170   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        47 & hall-light-welcome-home--lighting-director                        & 12 & 675    & 8,088   & 335   & 225    & 1,857  & 172   \\ \hline
        48 & lights-off-with-no-motion-and-presence--ecobeeAwayFromHome        & 7  & 605    & 1,400   & 178   & 400    & 827    & 140   \\ \hline
        49 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 170   & 505    & 1,000  & 198   \\ \hline
        50 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 179   & 505    & 1,000  & 198   \\ \hline
        51 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 170   & 505    & 999    & 200   \\ \hline
        52 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 179   & 505    & 999    & 200   \\ \hline
        53 & keep-me-cozy--WindowOrDoorOpen                                    & 10 & 589    & 5,880   & 238   & 589    & 4,338  & 365   \\ \hline
        54 & keep-me-cozy--hvac-auto-off.smartapp                              & 10 & 589    & 5,880   & 190   & 589    & 4,066  & 290   \\ \hline
        55 & keep-me-cozy--thermostat-auto-off                                 & 10 & 589    & 5,880   & 188   & 589    & 4,126  & 283   \\ \hline
        56 & lock-it-at-a-specific-time--ecobeeAwayFromHome                    & 8  & 553    & 1,472   & 142   & 473    & 1,018  & 152   \\ \hline
        57 & darken-behind-me--ecobeeAwayFromHome                              & 8  & 553    & 1,472   & 154   & 437    & 1,019  & 155   \\ \hline
        58 & turn-off-with-motion--ecobeeAwayFromHome                          & 7  & 553    & 1,288   & 144   & 410    & 848    & 133   \\ \hline
        59 & lights-off-with-no-motion-and-presence--step-notifier             & 8  & 506    & 4,040   & 224   & 309    & 1,309  & 142   \\ \hline
        60 & medicine-management-contact-sensor--circadian-daylight            & 13 & 492    & 6,318   & 369   & 310    & 2,729  & 225   \\ \hline
        61  & make-it-so--single-button-controller                                 & 6  & 454 & 906   & 148 & 369 & 712   & 165 \\ \hline
        62  & lighting-director--turn-it-on-when-it-opens                          & 11 & 423 & 4,642 & 199 & 223 & 1,614 & 135 \\ \hline
        63  & lighting-director--undead-early-warning                              & 11 & 423 & 4,642 & 199 & 223 & 1,614 & 133 \\ \hline
        64  & thermostat-window-check--whole-house-fan                             & 10 & 388 & 3,870 & 178 & 388 & 3,409 & 318 \\ \hline
        65  & good-night--BetterLaundryMonitor                                     & 8  & 385 & 3,072 & 125 & 294 & 1,245 & 116 \\ \hline
        66  & hue-minimote--smart-light-timer-x-minutes-unless-already-on          & 8  & 373 & 2,976 & 183 & 312 & 2,201 & 475 \\ \hline
        67  & gentle-wake-up--BetterLaundryMonitor                                 & 7  & 361 & 2,520 & 114 & 361 & 2,309 & 226 \\ \hline
        68  & make-it-so--thermostat-mode-director                                 & 8  & 361 & 2,880 & 137 & 345 & 2,335 & 251 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        69  & smart-nightlight--turn-on-at-sunset                                  & 10 & 356 & 3,550 & 141 & 333 & 2,861 & 239 \\ \hline
        70  & smart-nightlight--turn-it-on-when-it-opens                           & 10 & 331 & 3,300 & 139 & 101 & 369   & 50  \\ \hline
        71  & smart-nightlight--undead-early-warning                               & 10 & 331 & 3,300 & 140 & 101 & 369   & 50  \\ \hline
        72  & gentle-wake-up--good-night                                           & 8  & 321 & 2,560 & 125 & 321 & 2,376 & 239 \\ \hline
        73  & thermostat--thermostat-window-check                                  & 10 & 313 & 3,120 & 147 & 193 & 1,383 & 159 \\ \hline
        74  & smart-nightlight--vacation-lighting-director                         & 12 & 294 & 3,516 & 133 & 168 & 1,649 & 122 \\ \hline
        75  & lighting-director--turn-on-by-zip-code                               & 10 & 278 & 2,770 & 124 & 115 & 791   & 77  \\ \hline
        76  & smart-nightlight--turn-on-before-sunset                              & 9  & 276 & 2,475 & 92  & 257 & 2,013 & 146 \\ \hline
        77  & good-night--humidity-alert                                           & 8  & 257 & 1,024 & 65  & 243 & 911   & 98  \\ \hline
        78  & thermostat-auto-off--thermostat-window-check                         & 8  & 257 & 2,048 & 86  & 153 & 1,025 & 97  \\ \hline
        79  & double-tap--gentle-wake-up                                           & 7  & 216 & 560   & 58  & 194 & 445   & 67  \\ \hline
        80  & good-night--nfc-tag-toggle                                           & 8  & 215 & 1,712 & 94  & 215 & 1,433 & 149 \\ \hline
        81  & step-notifier--Hue-Party-Mode                                        & 6  & 215 & 1,284 & 82  & 215 & 1,061 & 126 \\ \hline
        82  & step-notifier--turn-on-only-if-i-arrive-after-sunset                 & 6  & 214 & 1,278 & 81  & 104 & 305   & 46  \\ \hline
        83  & Hue-Party-Mode--mini-hue-controller                                  & 4  & 212 & 844   & 42  & 201 & 669   & 86  \\ \hline
        84  & brighten-dark-places--lighting-director                              & 11 & 206 & 2,255 & 111 & 114 & 918   & 89  \\ \hline
        85  & whole-house-fan--WindowOrDoorOpen                                    & 11 & 196 & 2,145 & 120 & 196 & 1,691 & 232 \\ \hline
        86  & smart-nightlight--turn-off-with-motion                               & 9  & 196 & 1,755 & 76  & 87  & 357   & 41  \\ \hline
        87  & smart-nightlight--turn-on-by-zip-code                                & 9  & 196 & 1,755 & 79  & 43  & 131   & 17  \\ \hline
        88  & good-night--the-big-switch                                           & 8  & 191 & 1,520 & 64  & 191 & 1,168 & 103 \\ \hline
        89  & hall-light-welcome-home--step-notifier                               & 6  & 178 & 1,062 & 83  & 40  & 125   & 26  \\ \hline
        90  & smart-light-timer-x-minutes-unless-already-on--turn-on-before-sunset & 7  & 173 & 1,204 & 46  & 163 & 798   & 66  \\ \hline
        91  & smart-light-timer-x-minutes-unless-already-on--turn-on-at-sunset     & 7  & 173 & 1,204 & 50  & 157 & 770   & 68  \\ \hline
        92  & double-tap--good-night                                               & 8  & 161 & 512   & 49  & 149 & 456   & 64  \\ \hline
        93  & make-it-so--thermostat-window-check                                  & 10 & 157 & 1,560 & 80  & 157 & 1,231 & 123 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        94  & make-it-so--BetterLaundryMonitor                                     & 6  & 145 & 864   & 43  & 145 & 735   & 79  \\ \hline
        95  & make-it-so--thermostat                                               & 8  & 145 & 1,152 & 56  & 145 & 905   & 97  \\ \hline
        96  & rise-and-shine--BetterLaundryMonitor                                 & 6  & 136 & 810   & 43  & 103 & 371   & 74  \\ \hline
        97  & nfc-tag-toggle--single-button-controller                             & 4  & 121 & 160   & 51  & 105 & 128   & 66  \\ \hline
        98  & step-notifier--turn-on-at-sunset                                     & 5  & 117 & 580   & 45  & 68  & 177   & 31  \\ \hline
        99  & step-notifier--turn-on-before-sunset                                 & 5  & 117 & 580   & 44  & 68  & 177   & 30  \\ \hline
        100 & energy-saver--good-night                                             & 8  & 113 & 896   & 45  & 68  & 271   & 29  \\ \hline
        101 & big-turn-on--BetterLaundryMonitor                                    & 6  & 112 & 666   & 35  & 112 & 580   & 60  \\ \hline
        102 & the-big-switch--BetterLaundryMonitor                                 & 5  & 109 & 540   & 32  & 109 & 486   & 63  \\ \hline
        103 & gentle-wake-up--rise-and-shine                                       & 6  & 103 & 612   & 37  & 103 & 511   & 62  \\ \hline
        104 & BetterLaundryMonitor--Hue-Party-Mode                                 & 4  & 103 & 408   & 27  & 73  & 219   & 36  \\ \hline
        105 & 01-control-lights-and-locks-with-contact-sensor--good-night          & 8  & 97  & 768   & 69  & 65  & 399   & 49  \\ \hline
        106 & control-switch-with-contact-sensor--good-night                       & 8  & 97  & 768   & 34  & 65  & 399   & 48  \\ \hline
        107 & big-turn-on--good-night                                              & 7  & 95  & 658   & 33  & 95  & 637   & 64  \\ \hline
        108 & forgiving-security--smart-light-timer-x-minutes-unless-already-on    & 8  & 95  & 752   & 40  & 40  & 212   & 42  \\ \hline
        109 & smart-light-timer-x-minutes-unless-already-on--turn-on-by-zip-code   & 6  & 87  & 516   & 24  & 74  & 349   & 35  \\ \hline
        110 & step-notifier--turn-it-on-when-it-opens                              & 5  & 87  & 430   & 37  & 43  & 111   & 25  \\ \hline
        111 & step-notifier--undead-early-warning                                  & 5  & 87  & 430   & 37  & 43  & 111   & 25  \\ \hline
        112 & brighten-my-path--step-notifier                                      & 5  & 87  & 430   & 37  & 42  & 111   & 26  \\ \hline
        113 & thermostat--WindowOrDoorOpen                                         & 6  & 85  & 504   & 37  & 22  & 44    & 14  \\ \hline
        114 & thermostat--hvac-auto-off.smartapp                                   & 6  & 85  & 504   & 32  & 20  & 47    & 14  \\ \hline
        115 & thermostat--thermostat-auto-off                                      & 6  & 85  & 504   & 30  & 20  & 47    & 12  \\ \hline
        116 & BetterLaundryMonitor--WindowOrDoorOpen                               & 4  & 82  & 324   & 28  & 62  & 192   & 37  \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        117 & good-night--make-it-so                                               & 7  & 81  & 560   & 33  & 81  & 448   & 55  \\ \hline
        118 & good-night--rise-and-shine                                           & 6  & 80  & 474   & 25  & 65  & 294   & 47  \\ \hline
        119 & thermostat-window-check--WindowOrDoorOpen                            & 7  & 76  & 525   & 37  & 75  & 455   & 62  \\ \hline
        120 & make-it-so--auto-lock-door.smartapp                                  & 8  & 73  & 576   & 29  & 73  & 509   & 56  \\ \hline
        121 & good-night--once-a-day                                       & 8 & 73 & 576 & 27 & 57 & 402 & 39 \\ \hline
        122 & brighten-dark-places--step-notifier                          & 5 & 71 & 350 & 35 & 26 & 72  & 17 \\ \hline
        123 & gentle-wake-up--monitor-on-sense                             & 6 & 69 & 408 & 27 & 69 & 348 & 44 \\ \hline
        124 & big-turn-off--good-night                                     & 7 & 67 & 462 & 49 & 59 & 290 & 42 \\ \hline
        125 & good-night--monitor-on-sense                                 & 7 & 65 & 448 & 25 & 40 & 132 & 23 \\ \hline
        126 & hue-minimote--mini-hue-controller                            & 2 & 60 & 118 & 15 & 60 & 118 & 32 \\ \hline
        127 & double-tap--nfc-tag-toggle                                   & 4 & 57 & 64  & 32 & 55 & 57  & 35 \\ \hline
        128 & make-it-so--WindowOrDoorOpen                                 & 6 & 55 & 324 & 25 & 54 & 288 & 44 \\ \hline
        129 & good-night--power-allowance                                  & 7 & 49 & 336 & 25 & 49 & 297 & 43 \\ \hline
        130 & nfc-tag-toggle--the-big-switch                               & 5 & 49 & 240 & 21 & 43 & 125 & 37 \\ \hline
        131 & good-night--turn-it-on-for-5-minutes                         & 7 & 47 & 322 & 24 & 41 & 193 & 31 \\ \hline
        132 & step-notifier--turn-on-by-zip-code                           & 4 & 46 & 180 & 21 & 27 & 62  & 17 \\ \hline
        133 & make-it-so--NotifyIfLeftUnlocked                             & 6 & 43 & 252 & 19 & 43 & 232 & 39 \\ \hline
        134 & make-it-so--hvac-auto-off.smartapp                           & 6 & 43 & 252 & 19 & 43 & 234 & 37 \\ \hline
        135 & make-it-so--thermostat-auto-off                              & 6 & 43 & 252 & 18 & 43 & 234 & 35 \\ \hline
        136 & gentle-wake-up--make-it-so                                   & 5 & 41 & 200 & 19 & 41 & 179 & 32 \\ \hline
        137 & big-turn-on--gentle-wake-up                                  & 5 & 41 & 200 & 17 & 33 & 121 & 24 \\ \hline
        138 & good-night--sunrise-sunset                                   & 8 & 41 & 320 & 36 & 33 & 205 & 43 \\ \hline
        139 & good-night-house--lights-off-with-no-motion-and-presence     & 5 & 40 & 195 & 19 & 40 & 178 & 44 \\ \hline
        140 & single-button-controller--NotifyIfLeftUnlocked               & 4 & 40 & 52  & 35 & 31 & 34  & 31 \\ \hline
        141 & big-turn-off--energy-saver                                   & 6 & 39 & 228 & 34 & 39 & 204 & 31 \\ \hline
        142 & good-night-house--single-button-controller                   & 3 & 37 & 36  & 26 & 37 & 36  & 32 \\ \hline
        143 & double-tap--humidity-alert                                   & 4 & 37 & 48  & 20 & 31 & 34  & 27 \\ \hline
        144 & make-it-so--rise-and-shine                                   & 5 & 36 & 175 & 16 & 36 & 149 & 29 \\ \hline
        145 & good-night-house--make-it-so                                 & 4 & 35 & 136 & 13 & 35 & 128 & 30 \\ \hline
        146 & big-turn-on--rise-and-shine                                  & 5 & 31 & 150 & 14 & 31 & 129 & 23 \\ \hline
        147 & hue-minimote--Hue-Party-Mode                                 & 4 & 31 & 120 & 19 & 31 & 93  & 38 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        148 & let-there-be-dark--vacation-lighting-director                & 5 & 31 & 150 & 14 & 31 & 150 & 23 \\ \hline
        149 & door-state-to-color-light-hue-bulb--Hue-Party-Mode           & 4 & 30 & 116 & 15 & 27 & 82  & 21 \\ \hline
        150 & monitor-on-sense--BetterLaundryMonitor                       & 3 & 28 & 81  & 12 & 28 & 72  & 20 \\ \hline
        151 & monitor-on-sense--rise-and-shine                             & 5 & 27 & 130 & 14 & 24 & 85  & 22 \\ \hline
        152 & turn-off-with-motion--vacation-lighting-director             & 5 & 27 & 130 & 13 & 24 & 101 & 19 \\ \hline
        153 & lights-off-with-no-motion-and-presence--turn-off-with-motion & 4 & 27 & 104 & 14 & 23 & 82  & 19 \\ \hline
        154 & good-night--smart-turn-it-on                                 & 6 & 25 & 144 & 13 & 25 & 119 & 18 \\ \hline
        155 & make-it-so--monitor-on-sense                                 & 5 & 25 & 120 & 14 & 25 & 102 & 23 \\ \hline
        156 & make-it-so--unlock-it-when-i-arrive                          & 5 & 25 & 120 & 14 & 25 & 102 & 22 \\ \hline
        157 & nfc-tag-toggle--sunrise-sunset                               & 4 & 25 & 96  & 17 & 25 & 86  & 37 \\ \hline
        158 & good-night-house--vacation-lighting-director                 & 4 & 23 & 88  & 12 & 23 & 88  & 23 \\ \hline
        159 & good-night-house--hue-minimote                               & 3 & 22 & 63  & 16 & 22 & 61  & 29 \\ \hline
        160 & darken-behind-me--lights-off-with-no-motion-and-presence     & 5 & 22 & 105 & 19 & 20 & 81  & 30 \\ \hline
        161 & big-turn-on--monitor-on-sense                                & 5 & 20 & 95  & 13 & 20 & 82  & 24 \\ \hline
        162 & auto-lock-door.smartapp--NotifyIfLeftUnlocked                & 4 & 19 & 72  & 16 & 19 & 68  & 20 \\ \hline
        163 & good-night-house--turn-off-with-motion                       & 3 & 18 & 51  & 10 & 18 & 49  & 21 \\ \hline
        164 & nfc-tag-toggle--NotifyIfLeftUnlocked                         & 4 & 18 & 68  & 15 & 16 & 43  & 26 \\ \hline
        165 & good-night-house--auto-lock-door.smartapp                    & 5 & 17 & 80  & 12 & 15 & 60  & 24 \\ \hline
        166 & big-turn-off--power-allowance                                & 5 & 14 & 65  & 11 & 14 & 60  & 19 \\ \hline
        167 & big-turn-on--make-it-so                                      & 4 & 14 & 52  & 10 & 14 & 52  & 17 \\ \hline
        168 & forgiving-security--turn-on-at-sunset                        & 3 & 14 & 39  & 10 & 14 & 28  & 15 \\ \hline
        169 & forgiving-security--turn-on-before-sunset                    & 3 & 14 & 39  & 9  & 14 & 28  & 11 \\ \hline
        170 & big-turn-off--smart-turn-it-on                               & 4 & 13 & 48  & 8  & 13 & 48  & 12 \\ \hline
        171 & its-too-hot--its-too-cold                                    & 2 & 13 & 12  & 10 & 13 & 12  & 17 \\ \hline
        172 & lock-it-at-a-specific-time--make-it-so                       & 5 & 13 & 60  & 10 & 13 & 54  & 15 \\ \hline
        173 & unlock-it-when-i-arrive--auto-lock-door.smartapp             & 5 & 13 & 60  & 12 & 13 & 56  & 21 \\ \hline
        174 & good-night-house--nfc-tag-toggle                             & 2 & 13 & 24  & 9  & 11 & 16  & 18 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        175 & hall-light-welcome-home--turn-on-at-sunset                   & 3 & 13 & 36  & 10 & 8  & 14  & 8  \\ \hline
        176 & hall-light-welcome-home--turn-on-before-sunset               & 3 & 13 & 36  & 8  & 8  & 14  & 8  \\ \hline
        177 & double-tap--smart-turn-it-on                                 & 2 & 12 & 8   & 11 & 12 & 8   & 10 \\ \hline
        178 & forgiving-security--hall-light-welcome-home                  & 3 & 12 & 33  & 10 & 12 & 24  & 15 \\ \hline
        179 & brighten-my-path--hall-light-welcome-home                    & 3 & 12 & 33  & 9  & 10 & 18  & 9  \\ \hline
        180 & good-night-house--NotifyIfLeftUnlocked                       & 3 & 12 & 33  & 10 & 9  & 20  & 14 \\ \hline
        181 & lock-it-when-i-leave--NotifyIfLeftUnlocked          & 4 & 11 & 40 & 11 & 9  & 26 & 17 \\ \hline
        182 & enhanced-auto-lock-door--NotifyIfLeftUnlocked       & 4 & 10 & 36 & 10 & 10 & 30 & 16 \\ \hline
        183 & forgiving-security--DeviceTamperAlarm               & 4 & 10 & 36 & 10 & 10 & 29 & 14 \\ \hline
        184 & hue-minimote--turn-on-by-zip-code                   & 2 & 10 & 18 & 10 & 10 & 18 & 17 \\ \hline
        185 & darken-behind-me--turn-off-with-motion              & 3 & 9  & 24 & 8  & 9  & 18 & 8  \\ \hline
        186 & enhanced-auto-lock-door--good-night-house           & 5 & 9  & 40 & 13 & 9  & 38 & 28 \\ \hline
        187 & hvac-auto-off.smartapp--WindowOrDoorOpen            & 4 & 9  & 32 & 12 & 9  & 2  & 13 \\ \hline
        188 & lock-it-at-a-specific-time--auto-lock-door.smartapp & 5 & 9  & 40 & 9  & 9  & 36 & 15 \\ \hline
        189 & nfc-tag-toggle--smart-turn-it-on                    & 2 & 9  & 16 & 8  & 9  & 16 & 14 \\ \hline
        190 & thermostat-auto-off--WindowOrDoorOpen               & 4 & 9  & 32 & 10 & 9  & 23 & 13 \\ \hline
        191 & energy-saver--power-allowance                       & 3 & 8  & 21 & 8  & 8  & 21 & 14 \\ \hline
        192 & thermostat-auto-off--hvac-auto-off.smartapp         & 2 & 8  & 14 & 8  & 8  & 14 & 10 \\ \hline
        193 & unlock-it-when-i-arrive--NotifyIfLeftUnlocked       & 3 & 8  & 21 & 9  & 8  & 17 & 13 \\ \hline
        194 & brighten-my-path--turn-on-at-sunset                 & 2 & 7  & 12 & 7  & 7  & 10 & 12 \\ \hline
        195 & brighten-my-path--turn-on-before-sunset             & 2 & 7  & 12 & 7  & 7  & 10 & 8  \\ \hline
        196 & darken-behind-me--good-night-house                  & 2 & 7  & 12 & 8  & 7  & 11 & 11 \\ \hline
        197 & energy-saver--smart-turn-it-on                      & 2 & 7  & 12 & 7  & 7  & 12 & 8  \\ \hline
        198 & turn-on-at-sunset--turn-it-on-when-it-opens         & 2 & 7  & 12 & 7  & 7  & 9  & 11 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        199 & turn-on-at-sunset--turn-on-before-sunset            & 2 & 7  & 12 & 8  & 7  & 12 & 10 \\ \hline
        200 & turn-on-at-sunset--undead-early-warning             & 2 & 7  & 12 & 7  & 7  & 9  & 11 \\ \hline
        201 & turn-on-before-sunset--turn-it-on-when-it-opens     & 2 & 7  & 12 & 7  & 7  & 9  & 8  \\ \hline
        202 & turn-on-before-sunset--undead-early-warning         & 2 & 7  & 12 & 7  & 7  & 9  & 8  \\ \hline
        203 & brighten-dark-places--turn-on-at-sunset             & 2 & 7  & 12 & 7  & 6  & 8  & 8  \\ \hline
        204 & brighten-dark-places--turn-on-before-sunset         & 2 & 7  & 12 & 7  & 6  & 8  & 8  \\ \hline
        205 & brighten-dark-places--hall-light-welcome-home       & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        206 & hall-light-welcome-home--turn-it-on-when-it-opens   & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        207 & hall-light-welcome-home--turn-on-by-zip-code        & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        208 & hall-light-welcome-home--undead-early-warning       & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        209 & brighten-dark-places--forgiving-security            & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        210 & brighten-my-path--forgiving-security                & 2 & 6  & 10 & 8  & 6  & 8  & 12 \\ \hline
        211 & brighten-my-path--turn-it-on-when-it-opens          & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        212 & brighten-my-path--undead-early-warning              & 2 & 6  & 10 & 7  & 6  & 8  & 10 \\ \hline
        213 & forgiving-security--turn-it-on-when-it-opens        & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        214 & forgiving-security--turn-on-by-zip-code             & 2 & 6  & 10 & 7  & 6  & 8  & 8  \\ \hline
        215 & forgiving-security--undead-early-warning            & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        216 & lock-it-at-a-specific-time--NotifyIfLeftUnlocked    & 3 & 6  & 15 & 8  & 6  & 14 & 11 \\ \hline
        217 & brighten-dark-places--brighten-my-path              & 2 & 6  & 10 & 8  & 5  & 6  & 9  \\ \hline
        218 & enhanced-auto-lock-door--auto-lock-door.smartapp    & 4 & 5  & 16 & 9  & 5  & 15 & 18 \\ \hline
        219 & good-night-house--lock-it-at-a-specific-time        & 2 & 5  & 8  & 7  & 5  & 6  & 8  \\ \hline
        220 & good-night-house--turn-on-by-zip-code               & 2 & 5  & 8  & 7  & 5  & 6  & 8  \\ \hline
        221 & turn-on-at-sunset--turn-on-by-zip-code              & 2 & 5  & 8  & 6  & 5  & 6  & 7  \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        222 & turn-on-before-sunset--turn-on-by-zip-code          & 2 & 5  & 8  & 6  & 5  & 6  & 7  \\ \hline
        223 & brighten-dark-places--turn-it-on-when-it-opens      & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        224 & brighten-dark-places--turn-on-by-zip-code           & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        225 & brighten-dark-places--undead-early-warning          & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        226 & brighten-my-path--turn-on-by-zip-code               & 2 & 4  & 6  & 6  & 4  & 5  & 6  \\ \hline
        227 & turn-it-on-when-it-opens--undead-early-warning      & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        228 & turn-on-by-zip-code--turn-it-on-when-it-opens       & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        229 & turn-on-by-zip-code--undead-early-warning           & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
  \end{tabular}
  }

\end{table*}

\section*{Acknowledgment}
We would like to thank our anonymous reviewers for their thorough comments and feedback. This project was supported partly by the National Science Foundation under grants CCF-2006948, CCF-2102940, CNS-1703598, CNS-1763172, CNS-1907352, CNS-2006437, CNS-2007737, CNS-2106838, CNS-2128653, OAC-1740210 and by the Office of Naval Research under grant N00014-18-1-2037.
}

\subsection{The Algorithm}

We first present the state that our algorithm maintains:

\begin{enumerate}
\item {\bf The transition sequence \Trace{}} contains the new transitions that the current execution explores.  Our algorithm explores a given transition in at most one execution.

\item {\bf The state history \hbox{$\mathcal{H}$}\xspace} is a set of program states that have been visited in completed executions.

\item {\bf The state transition graph \RGraph{}} records the states our algorithm has explored thus far.  Nodes in this graph correspond to program states and edges to transitions between program states.

\rahmadi{
\item {\bf The state summary \Summary{}} records a summary of events (and their corresponding accesses) for which we have set backtracking points at every state our algorithm has explored. With \Summary{}, the algorithm can efficiently reexplore the previously explored transitions: (1) when a previously discovered state \State{} is reached, \Summary{} serves as a summary of all reachable transitions from \State{}, and (2) when performing backwards depth first search, our algorithm can stop exploring the execution at state \State{} whenever the algorithm finds that the currently executed event and its accesses are a subset of the summary for \State{} in \Summary{}.
}
\end{enumerate}

Recall that for each reachable state $\State{} \in \hbox{$\textit{States}$}\xspace{}$, our algorithm maintains the \State{}.\textit{backtrack} set that contains the events to be explored at that state, the \State{}.\textit{done} set that contains the events that have already been explored at the state, and the \State{}.\textit{enabled} set that contains all events that are enabled to be executed at that state.

\begin{algorithm}

\DontPrintSemicolon
\SetKwBlock{Begin}{}{end}
\SetAlgoLined
\Begin($\textsc{ExploreAll}\xspace {(}{)}$){
\rahmadi{
$\Trace{} := \emptyset$\;
$\hbox{$\mathcal{H}$}\xspace := \emptyset$\;
$\RGraph{} := \emptyset$\;
$\Summary{} := \emptyset$\;
}
$\textsc{Explore}\xspace {(} \State{0} {)}$\;\label{line:firstexplore}
\While{$\exists \State{}, \: \State{}.backtrack \neq \State{}.done$}
      {
$\textsc{Explore}\xspace {(} \State{} {)}$\; \label{line:explore}
      }
}
\caption{Top-level exploration algorithm.\label{alg:sdpormain}}
\end{algorithm}

Algorithm~\ref{alg:sdpormain} presents the top-level \textsc{ExploreAll}\xspace procedure.  This procedure first invokes the \textsc{Explore}\xspace procedure to start model checking from the initial state.  However,
the presence of cycles in the state space means that our backtracking-based search algorithm may occasionally set new backtracking points for states in completed executions.  The \textsc{ExploreAll}\xspace procedure thus loops over all states that have unexplored items in their backtrack sets and invokes the \textsc{Explore}\xspace procedure to explore those transitions.

\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{Explore}\xspace {(} \State{} {)}$)
{
\If{$\State{}.\textit{backtrack} = \State{}.\textit{done}\label{line:ifback}$}{ \label{line:enable-event-start}
\uIf{$\State{}.\textit{done} = \State{}.\textit{enabled}$}{
\uIf{$\State{}.\textit{enabled} \text{ is not empty}$}{\label{line:checkenabled}
$\text{select} \: \Event{} \: \in \State{}.\textit{enabled}\label{line:evselect}$\;
$\text{remove} \: \Event{} \: \text{from} \: \State{}.\textit{done}$\;
} \Else {
$\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
$\Trace{} := \emptyset$\;
\Return{}
}
} \Else {
$\text{select} \: \Event{} \: \in \State{}.\textit{enabled} \setminus \State{}.\textit{done}$\;
$\text{add} \: \Event{} \: \text{to} \: \State{}.\textit{backtrack}$\;
}
} \label{line:enable-event-end}
\While{$\exists \hbox{$b$}\xspace \in \State{}.\textit{backtrack} \setminus \State{}.\textit{done}$\label{line:while}} {
  $\text{add} \: \hbox{$b$}\xspace \: \text{to} \: \State{}.done$\;
  $\Trans{} := \text{\hbox{\emph{next}}\xspace} \left(\State{},\hbox{$b$}\xspace\right)$\;\label{line:algnext}
  $\State{}' := \hbox{\emph{dst}}\xspace\left(\Trans{}\right)$\;\label{line:algdst}
  $\text{add transition} \: \Trans{} \:\text{to}\:\RGraph{}$\;\label{line:addtrans}
  \ForEach{$\Event{} \in \State{}.\textit{enabled} \setminus    \label{line:add-disabled} \State{}'.\text{enabled}$}{
    $\text{add} \: \Event \: \text{to} \: \State{}.\textit{backtrack}$\;\label{line:adddisabled}
  }
  $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:call-update-backtrack-set}
  \uIf{$\State{}' \in \hbox{$\mathcal{H}$}\xspace \vee \textsc{IsFullCycle}\xspace\left(\Trans{}\right)$\label{line:termination-condition}} {
    $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\;\label{line:endbacktrack}
    $\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
    $\Trace{} := \emptyset$\;
  }
  \Else {
    \If{$\State{}' \in \hbox{\emph{states}}\xspace{}(\Trace{}) $\label{line:cycle-current-exec-1}} {
      $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\label{line:cycle-current-exec-2}
    }
    $\Trace{} := \Trace{}.\Trans{}$ \;
    $\textsc{Explore}\xspace {(} \hbox{$s$}\xspace' {)}$\;\label{line:call-explore}
  }
}
}
 \caption{Stateful DPOR algorithm for event-driven applications.\label{alg:sdpor}}
\end{algorithm}

Algorithm~\ref{alg:sdpor} describes the logic of the \textsc{Explore}\xspace{} procedure.
The \code{if} statement in line~\ref{line:ifback} checks if the
current state \State{}'s \textit{backtrack} set is the same as the
current state \State{}'s \textit{done} set.  If so, the algorithm
selects an event to execute in the next transition.  If there are
enabled events not yet explored, it selects an unexplored event
to add to the current state's \textit{backtrack} set.  Otherwise, if
the \textit{enabled} set is not empty, it selects an enabled event to
remove from the \textit{done} set.  Note that this scenario occurs only if
the execution is continuing past a state match to satisfy the termination condition.

Then the \code{while} loop in line~\ref{line:while} selects an event
$\hbox{$b$}\xspace$ to execute on the current state \State{} and
executes the event $\hbox{$b$}\xspace$ to generate the
transition \Trans{} that takes the execution to a new state \State{}'.
At this point, the algorithm knows the memory accesses performed by
the transition \Trans{} and thus can add the event $\hbox{$b$}\xspace$
to the backtrack sets of the previous states.  This is done via the procedure \textsc{UpdateBacktrackSet}\xspace{}.

Traditional DPOR algorithms continue an execution until it terminates.  Since our programs may have cyclic state spaces, this would cause the model checker to potentially not terminate.  Our algorithm instead checks the conditions in
line~\ref{line:termination-condition} to decide whether to terminate the execution.  These checks see whether the new state $\State{}'$ matches a state from a previous execution, or if the current execution revisits a state the current execution previously explored and meets other criteria that are checked in the \textsc{IsFullCycle}\xspace{} procedure.   If so, the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure to set backtracking points, from transitions reachable from $\Trans{}$, to states that can reach $\Trans{}$.  An execution will also terminate if it reaches a state in which no event is enabled (line~\ref{line:checkenabled}).
It then adds the states from the current transition sequence to the set of previously visited states $\hbox{$\mathcal{H}$}\xspace$, resets the current execution transition sequence $\Trace{}$, and backtracks to start a new execution.

If the algorithm has reached a state $\State{}'$ that was previously discovered in this execution, it sets backtracking points by calling the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  Finally, it updates the transition sequence $\Trace{}$ and calls \textsc{Explore}\xspace{}.

\mycomment{
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSetsFromGraph}\xspace {(} \Trans{s}{)}$)
{
  $\RGraph{t} := \lbrace
    \Trans{}\in\RGraph{} \mid \:\: \Trans{s}\hbox{$\:\rightarrow_{r}\:$}\xspace\Trans{}
    \rbrace$\;
  \ForEach{$\Trans{} \in \RGraph{t}$\label{line:check-cycle-in-prev-1}}
  {
     $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:check-cycle-in-prev-2}
  }
}
 \caption{Procedure that updates the backtrack sets of states in a previous execution.\label{alg:backtrack-update-graph}}
\end{algorithm}
}

\begin{algorithm}
\rahmadi{
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSetsFromGraph}\xspace {(} \Trans{}{)}$)
{
  $\Events{r} := \{ \Event{} \mid \forall \Event{}, \langle \Event{}, a \rangle \in \Summary{}(\hbox{\emph{dst}}\xspace(\Trans{}))\}$\;\label{line:compute-state-summary}
  \ForEach{$\Event{} \in \Events{r}$\label{line:check-cycle-in-prev-1}}
  {
     $\Access{r} := \{ a \mid \forall \langle \Event{}, a \rangle \in \Summary{}(\hbox{\emph{dst}}\xspace(\Trans{}))\}$\;
       $\Access{b} := \textit{accesses}(\Trans{})$\;
  $\Event{\text{conf}}' := \Event{}$\;
  \If{$\exists a \in \Access{r}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$} {
   \uIf{$\Event{} \in \hbox{\emph{src}}\xspace(\Trans{}).\textit{enabled}$} {
   $\text{add} \: \Event{} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{}).\textit{backtrack}$\;
   } \Else {
      $\text{add} \: \hbox{\emph{src}}\xspace(\Trans{}).\textit{enabled} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{}).\textit{backtrack}$\;
   }
   $\Event{\text{conf}}' := \hbox{\emph{event}}\xspace(\Trans{})$\;
  }
  $\Access{r}' := \{ a \in \Access{r} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{}, \Event{\text{conf}}', \Access{r}', \{ \Trans{} \} \right)$\;\label{line:check-cycle-in-prev-2}
  }
}
 \caption{Procedure that updates the backtrack sets of states in a previous execution.\label{alg:backtrack-update-graph}}
 }
\end{algorithm}

\mycomment{
Algorithm~\ref{alg:backtrack-update-graph} shows the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  This procedure takes a transition $\Trans{}$ that connects the current execution to a previously discovered state in the transition graph $\RGraph{}$.  Since our algorithm does \emph{not} explore all of the transitions reachable from the previously discovered state, we need to create the backtracking points that would have been created by these skipped transitions. This procedure therefore computes the set of transitions reachable from the destination state of $\Trans{}$ and invokes $\textsc{UpdateBacktrackSet}\xspace{}$ on each of those transitions to set backtracking points.
}
\rahmadi{
Algorithm~\ref{alg:backtrack-update-graph} shows the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  This procedure takes a transition $\Trans{}$ that connects the current execution to a previously discovered state in the transition graph $\RGraph{}$.  Since our algorithm does \emph{not} explore all of the transitions reachable from the previously discovered state, we need to create the backtracking points that would have been created by these skipped transitions. This procedure therefore computes the set of transitions reachable from the destination state of $\Trans{}$, namely $\hbox{\emph{dst}}\xspace{}(\Trans{})$. As the previously explored transitions are stored as a summary of events and their accesses in \Summary{} for every discovered state, the procedure first computes the summary of events and their accesses for $\hbox{\emph{dst}}\xspace{}(\Trans{})$ in line~\ref{line:compute-state-summary}.
It then invokes $\textsc{UpdateBacktrackSetDFS}\xspace{}$ on each of those events to set backtracking points.
}

\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{IsFullCycle}\xspace {(} \Trans{} {)}$)
{
\If{$\neg \textit{dst}(\Trans{}) \in \hbox{\emph{states}}\xspace{}(\Trace{})$}{\Return{$\textit{false}$}}
  $\Trace{}^{\textit{fc}} := \lbrace
    \Trans{j} \in \Trace{} \mid
        i=\hbox{\emph{first}}\xspace(\Trace{}, \hbox{\emph{dst}}\xspace(\Trans{})), \:\: \text{and} \:\:
        i < j \rbrace \cup \{ \Trans{} \}$\;
  $\Events{\textit{fc}} := \lbrace
    \hbox{\emph{event}}\xspace{}(\Trans{}') \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\;
  $\Events{\textit{enabled}} := \lbrace
    \hbox{\emph{dst}}\xspace{}(\Trans{}').\textit{enabled} \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\;
  \Return{$\Events{\textit{fc}} = \Events{\textit{enabled}}\:$\label{line:check-full-cycle}}cur
}
 \caption{Function that checks the looping termination condition: complete a cycle that contains all events.
 \label{alg:complete-full-cycle}}
\end{algorithm}

\mycomment{
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSet}\xspace {(} \Trans{} {)}$)
{
 $\textsc{UpdateBacktrackSetDFS}\xspace\left( \Trans{}, \Trans{}, \textit{accesses}(\Trans{}), \{ \Trans{} \} \right)$
}

\Begin($\textsc{UpdateBacktrackSetDFS}\xspace {(} \Trans{\text{curr}}, \Trans{\text{conf}},  \Access{}, \Transitions{\text{explored}} {)}$)
{
  \ForEach{$\Trans{b} \in \textit{pred}_{\RGraph{}}(\Trans{\text{curr}}) \setminus \Transitions{\text{explored}} $\label{lin:loop}}{
  $\Access{b} := \textit{accesses}(\Trans{b})$\;
  $\Trans{\text{conf}}' := \Trans{\text{conf}}$\;
  \If{$\exists a \in \Access{}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$\label{lin:conflict}} {
   \uIf{$\hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \in \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled}$} {
   $\text{add} \: \hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   } \Else {
      $\text{add} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   }
   $\Trans{\text{conf}}' := \Trans{b}$\;
  }
  $\Access{r} := \{ a \in \Access{} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;  \label{lin:newaccesses}
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{b}, \Trans{\text{conf}}', \Access{r}, \Transitions{\text{explored}} \: \cup \{ \Trans{b} \} \right)$\;
  }
}

 \caption{Procedure that updates the backtrack sets of the state of the event that conflict with the current event in the search stack.
  \label{alg:backtrack-updatev2}}
\end{algorithm}
}

\begin{algorithm}
\rahmadi{
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSet}\xspace {(} \Trans{} {)}$)
{
 $\textsc{UpdateBacktrackSetDFS}\xspace\left( \Trans{}, \hbox{\emph{event}}\xspace(\Trans{}), \textit{accesses}(\Trans{}), \{ \Trans{} \} \right)$
}

\Begin($\textsc{UpdateStateSummary}\xspace {(} \State{}, \Event{}, \Access{} {)}$)
{

  $\Access{}' := \Access{} \setminus \{ a \mid \forall \langle \Event{}, a \rangle \in \Summary{}(\State{})\}$\;
  $\Summary{} := \Summary{}[ \State{} := \Summary{}(\State{}) \cup (\{ \Event{} \} \times \Access{})]$\;
  \Return{$\Access{}'$}
}

\Begin($\textsc{UpdateBacktrackSetDFS}\xspace {(} \Trans{\text{curr}}, \Event{\text{conf}},  \Access{}, \Transitions{\text{explored}} {)}$)
{
  $\Access{} := \textsc{UpdateStateSummary}\xspace\left( \hbox{\emph{src}}\xspace(\Trans{curr}), \Event{conf}, \Access{} \right)$\;\label{lin:updatestatesummary}
  \If{$\Access{} \neq \emptyset$}{
  \ForEach{$\Trans{b} \in \textit{pred}_{\RGraph{}}(\Trans{\text{curr}}) \setminus \Transitions{\text{explored}} $\label{lin:loop}}{
  $\Access{b} := \textit{accesses}(\Trans{b})$\;
  $\Event{\text{conf}}' := \Event{\text{conf}}$\;
  \If{$\exists a \in \Access{}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$\label{lin:conflict}} {
   \uIf{$\Event{\text{conf}} \in \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled}$} {
   $\text{add} \: \Event{\text{conf}} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   } \Else {
      $\text{add} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{enabled} \: \text{to} \: \hbox{\emph{src}}\xspace(\Trans{b}).\textit{backtrack}$\;
   }
   $\Event{\text{conf}}' := \hbox{\emph{event}}\xspace(\Trans{b})$\;
  }
  $\Access{r} := \{ a \in \Access{} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;  \label{lin:newaccesses}
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{b}, \Event{\text{conf}}', \Access{r}, \Transitions{\text{explored}} \: \cup \{ \Trans{b} \} \right)$\;
  }
  }
}

 \caption{Procedure that updates the backtrack sets of the state of the event that conflict with the current event in the search stack.
  \label{alg:backtrack-updatev2}}}
\end{algorithm}

Algorithm~\ref{alg:complete-full-cycle} presents
the \textsc{IsFullCycle}\xspace{} procedure.  This procedure first checks that
there is a cycle that contains the transition $\Trans{}$ in the state
space of the current execution.  The example from
Figure~\ref{fig:example-non-terminating} shows that such a state match
is not sufficient to terminate the execution as the execution may not
have set the necessary backtracking points.  Our algorithm stops the
exploration of an execution when there is a cycle that has explored
\emph{every event that is enabled in that cycle}.  This ensures that for
every transition \Trans{} in the execution, there is a future
transition \Trans{e} for each enabled event \Event{} in the cycle that
can set a backtracking point if \Trans{} and \Trans{e} conflict.

\mycomment{
Algorithm~\ref{alg:backtrack-updatev2} presents the \textsc{UpdateBacktrackSet}\xspace{} procedure, which sets backtracking points.  There are two differences between our algorithm and traditional DPOR algorithms.  First, since our algorithm supports programs with cyclic state spaces, it is possible that the algorithm has discovered multiple paths from the start state $\State{0}$ to the current transition $\Trans{}$.  Thus, the algorithm must potentially set backtracking points on multiple different paths.  We address this issue using a backwards depth first search traversal of the state transition graph $\RGraph{}$.   Second, since our transitions correspond to events they may potentially access multiple different memory locations and thus the backtracking algorithm potentially needs to set separate backtracking points for each of these memory locations.

The \textsc{UpdateBacktrackSetDFS}\xspace{} procedure implements a backwards
depth first traversal to set backtracking points.  The procedure takes
the following parameters: \Trans{\text{curr}} is the current transition in the
DFS, \Trans{\text{conf}} is the transition that we are currently setting
a backtracking point for, \Access{} is the set of accesses that the
algorithm searches for conflicts for, and \Transitions{\text{explored}} is
the set of transitions that the algorithm has explored down this
search path.  Recall that accesses consist of both an operation, \hbox{\emph{i.e.},}\xspace a read or write, and a memory location.  Conflicts are defined in the usual way---writes to a memory location conflict with reads or writes to the same memory location.

Line~\ref{lin:loop} loops over each transition \Trans{b} that
immediately precedes transition \Trans{\text{curr}} in the state transition
graph and has not been explored.  Line~\ref{lin:conflict} checks for
conflicts between the accesses of \Trans{b} and the access
set \Access{} for the DFS.  If a conflict is detected, the
algorithm adds the event for transition \Trans{\text{conf}} to the backtracking
set.  Line~\ref{lin:newaccesses} removes the accesses that conflicted
with transition \Trans{b}.  The search procedure then recursively
calls itself.  If the current transition \Trans{b} conflicts with the
transition \Trans{\text{conf}} for which we are setting a backtracking point, then
it is possible that the behavior we are interested in for \Trans{\text{conf}}
requires that \Trans{b} be executed first.  Thus, if there is a
conflict between \Trans{b} and \Trans{\text{conf}}, we pass \Trans{b} as the
conflict transition parameter to the recursive calls
to \textsc{UpdateBacktrackSetDFS}\xspace{}.
}
\rahmadi{
Algorithm~\ref{alg:backtrack-updatev2} presents the \textsc{UpdateBacktrackSet}\xspace{} procedure, which sets backtracking points.  There are two differences between our algorithm and traditional DPOR algorithms.  First, since our algorithm supports programs with cyclic state spaces, it is possible that the algorithm has discovered multiple paths from the start state $\State{0}$ to the current transition $\Trans{}$.  Thus, the algorithm must potentially set backtracking points on multiple different paths.  We address this issue using a backwards depth first search traversal of the state transition graph $\RGraph{}$.   Second, since our transitions correspond to events they may potentially access multiple different memory locations and thus the backtracking algorithm potentially needs to set separate backtracking points for each of these memory locations.

The \textsc{UpdateBacktrackSetDFS}\xspace{} procedure implements a backwards
depth first traversal to set backtracking points.  The procedure takes
the following parameters: \Trans{\text{curr}} is the current transition in the
DFS, \Event{\text{conf}} is the event that we are currently setting
a backtracking point for, \Access{} is the set of accesses that the
algorithm searches for conflicts for, and \Transitions{\text{explored}} is
the set of transitions that the algorithm has explored down this
search path.  Recall that accesses consist of both an operation, \hbox{\emph{i.e.},}\xspace a read or write, and a memory location.  Conflicts are defined in the usual way---writes to a memory location conflict with reads or writes to the same memory location.

Line~\ref{lin:updatestatesummary} calls the \textsc{UpdateStateSummary}\xspace{} procedure to record \Event{\text{conf}} and the accesses set \Access{} in \Summary{}.
Line~\ref{lin:loop} loops over each transition \Trans{b} that
immediately precedes transition \Trans{\text{curr}} in the state transition
graph and has not been explored.  Line~\ref{lin:conflict} checks for
conflicts between the accesses of \Trans{b} and the accesses
set \Access{} for the DFS.  If a conflict is detected, the
algorithm adds the event for transition \Trans{\text{conf}} to the backtracking
set.  Line~\ref{lin:newaccesses} removes the accesses that conflicted
with transition \Trans{b}.  The search procedure then recursively
calls itself.  If the current transition \Trans{b} conflicts with the
transition \Trans{\text{conf}} for which we are setting a backtracking point, then
it is possible that the behavior we are interested in for \Trans{\text{conf}}
requires that \Trans{b} be executed first.  Thus, if there is a
conflict between \Trans{b} and \Trans{\text{conf}}, we pass \Trans{b} as the
conflict transition parameter to the recursive calls
to \textsc{UpdateBacktrackSetDFS}\xspace{}.
}

\mysubsection{Example\label{sec:example}}
Next, we illustrate how our new stateful DPOR algorithm works with the example from
Figure~\ref{fig:example-missing-execution}.

First, the algorithm starts by calling the \textsc{ExploreAll}\xspace procedure in
Algorithm~\ref{alg:sdpormain}. After the state is initialized, it calls the \textsc{Explore}\xspace procedure in line~\ref{line:firstexplore}.
Let us suppose that the algorithm first explores the execution shown in
Figure~\ref{fig:example-missing-execution}-a.
Line~\ref{line:evselect} in Algorithm~\ref{alg:sdpor} chooses
event \code{e$_1$} among the enabled events.
Next, event \code{e$_1$} is executed: it produces a new state $\State{1}$ and adds
the corresponding transition to \RGraph{}.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure,
but it does not update the backtrack set of any state since the algorithm has only
explored one event.
Line~\ref{line:termination-condition} evaluates to \emph{false} since the termination
condition is not met yet: $\State{1}$ is not found in \hbox{$\mathcal{H}$}\xspace and
it is not a full cycle.
Line~\ref{line:cycle-current-exec-1} also evaluates to \emph{false} since $\State{1}$
is not found in \Trace{}.
Finally, Algorithm~\ref{alg:sdpormain} calls the \textsc{Explore}\xspace procedure recursively to
explore the next transition.

In the subsequent transitions, line~\ref{line:evselect} chooses event \code{e$_2$} and then event \code{e$_3$}.
For each of these events, the \textsc{UpdateBacktrackSet}\xspace{} procedure invokes
\textsc{UpdateBacktrackSetDFS}\xspace{} in Algorithm~\ref{alg:backtrack-updatev2} to
perform the DFS, find conflicts, and update the corresponding backtrack
sets---it puts event \code{e$_2$} in the
backtrack set of $\State{0}$ and event \code{e$_3$} in the backtrack set of
$\State{1}$ as it finds conflicts between events \code{e$_1$}
and \code{e$_2$} that access the shared variable \code{z}, and events \code{e$_2$} and \code{e$_3$} that access the shared variable \code{y}.
Event \code{e$_2$} produces a new state $\State{2}$, while \code{e$_3$}
causes the execution to revisit state $\State{1}$.
At this point, line~\ref{line:cycle-current-exec-1} evaluates
to \emph{true} and the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{}
procedure.
This procedure (shown in Algorithm~\ref{alg:backtrack-update-graph}) finds that
the transition that corresponds to event \code{e$_2$} is reachable from
the transition that corresponds to event \code{e$_3$} in \RGraph{} and calls
the \textsc{UpdateBacktrackSet}\xspace{} procedure.
\textsc{UpdateBacktrackSet}\xspace{} puts event \code{e$_2$} in the backtrack
set of $\State{2}$ as it finds a conflict between the events \code{e$_2$} and \code{e$_3$}
that access the shared variable \code{y}.

In the final transition of the first execution, line~\ref{line:evselect} chooses event \code{e$_4$}.
The \textsc{UpdateBacktrackSet}\xspace{} procedure finds a conflict between \code{e$_4$} and \code{e$_1$} that both access the shared variable \code{x}:
it puts event \code{e$_4$} in the backtrack set of state $\State{0}$.
Event \code{e$_4$} directs the algorithm to revisit state $\State{0}$. This time
line~\ref{line:termination-condition} evaluates to \emph{true} as the termination
condition is satisfied.
The algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure---it finds
that the events \code{e$_1$}, \code{e$_2$}, and \code{e$_3$} are reachable from the transition for event \code{e$_4$}.
Next, it calls the \textsc{UpdateBacktrackSet}\xspace{} procedure for the reachable
transitions. Among all the conflicts found by the procedure, it
puts event \code{e$_1$} in the backtrack set of state $\State{1}$ as a new backtracking point
for the conflict between the events \code{e$_1$} and \code{e$_4$}.
Then, the algorithm saves the visited states in this execution into \hbox{$\mathcal{H}$}\xspace
and terminates the execution.

Now the algorithm explores the executions in the backtrack sets.
This time, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses
the events \code{e$_1$} and \code{e$_3$} in the backtrack set of $\State{1}$, and
event \code{e$_2$} in the backtrack set of $\State{2}$.
These executions terminate quickly as they immediately revisit states
$\State{1}$ and $\State{2}$ that have been stored in \hbox{$\mathcal{H}$}\xspace
at the end of the first execution---line~\ref{line:termination-condition} evaluates to \emph{true}.
These executions are shown in
Figures~\ref{fig:example-missing-execution}-b, c, and d.

Next, let us suppose that the algorithm explores the execution from event \code{e$_2$}
in the backtrack set of $\State{0}$ shown in Figure~\ref{fig:example-missing-execution}-e.
At first, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses event \code{e$_2$} in the backtrack set of $\State{0}$ that produces a new state $\State{3}$.
When line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{}
procedure, this invocation allows the algorithm to
perform the DFS backward to find conflicts between event \code{e$_2$} and the transitions in
previous executions.
Thus, it finds a conflict between the current event \code{e$_2$} and event \code{e$_3$} from the previous execution
shown in Figures~\ref{fig:example-missing-execution}-b---both access the shared
variable \code{y}. It then puts event \code{e$_2$} into the backtrack set of state $\State{1}$.
Since lines~\ref{line:termination-condition} and~\ref{line:cycle-current-exec-1}
evaluate to \emph{false}, the algorithm calls the \textsc{Explore}\xspace{} procedure recursively.
In the second recursion, let us suppose that event \code{e$_3$} is chosen.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure:
it finds a conflict between events \code{e$_3$} and \code{e$_2$} and puts event \code{e$_3$}
in the backtrack set of $\State{0}$.
Since event \code{e$_3$} directs the execution to revisit $\State{0}$ that is already in \hbox{$\mathcal{H}$}\xspace,
line~\ref{line:termination-condition} evaluates to \emph{true} and
the algorithm invokes the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.
This procedure finds a conflict between event \code{$e_2$} of the current execution as a
reachable transition from $\State{0}$ and event \code{$e_3$} from $\State{3}$ that was just executed; it next puts event \code{e$_2$} into the backtrack set of
$\State{3}$ as a new backtracking point.  At this point, this execution terminates.

The algorithm explores executions from the remaining backtracking points,
namely, executions from event \code{e$_2$} in the backtrack set of $\State{3}$, and
\code{e$_3$} and \code{e$_4$} in the backtrack set of $\State{0}$.
These executions are shown in Figures~\ref{fig:example-missing-execution}-f, g, and h.  In the execution of event \code{e$_4$} from state $\State{0}$, the algorithm detects a conflict with the prior execution of event \code{e$_3$} from state $\State{3}$; it puts event \code{e$_4$} into the backtrack set of $\State{3}$.

The algorithm then explores the execution shown in Figure~\ref{fig:example-missing-execution}-i, which contains the assertion. The model checker stops its exploration and returns an error.

\mysubsection{Optimizing Traversals}

\todo{Are we keeping this section? Limit is 20 pages!}

The algorithm as presented performs graph traversals to identify conflicts.  In our implementation atop the Java Pathfinder model checker, the overhead of these graph traversals was small compared to the overhead of Java Pathfinder itself and thus we did not implement these optimizations.  In other settings, it is possible that graph traversals could introduce a sufficient overhead such that optimizations would be warranted.  We next discuss potential optimizations to reduce traversal overheads.

Procedure \textsc{UpdateBacktrackSet}\xspace{} performs a graph traversal to find conflicts with the  transition \Trans{}.  If the path from the start state to the current transition only contains states with one incoming edge, then the traversal is not necessary.  Instead, the model checker can use a common approach of augmenting each shared memory locations with shadow information that records the last read and write accesses~\cite{flanagan2005dynamic}.  The \textsc{UpdateBacktrackSet}\xspace{} procedure can use this information to immediately determine the last conflicting transitions for a memory location. Even if the path has states with more than one incoming edge, this approach can be used to detect conflicts that occur after the most recent such state.  If there may be conflicts before the most recent state with more than one incoming edge, then the procedure would have to fall back to the existing graph traversal.

The procedure \textsc{UpdateBacktrackSetsFromGraph}\xspace{} traverses the graph after a state match to discover potential conflicting transactions from previous executions.  Many of these graph traversals could be eliminated by caching the results of the graph search the first time \textsc{UpdateBacktrackSetsFromGraph}\xspace is called.  The results can be cached as a summary
for each state of the potentially conflicting transitions that are reachable from the given state.  This cached summary can be updated by \textsc{UpdateBacktrackSet}\xspace{} during its backwards graph traversal.  Note that after \textsc{UpdateBacktrackSetsFromGraph}\xspace is called on a state, that state has at least two incoming transitions. Hence, even the previous optimizations to the \textsc{UpdateBacktrackSet}\xspace{} procedure need to implement a graph traversal for that state to update the cached information.

\mysubsection{Discussion}

Although the algorithm we present is designed to explore executions of event-driven systems, the key ideas are also applicable to model checking multithreaded programs.  To the best of our knowledge, there are no correct algorithms for stateful dynamic partial order reduction for multithreaded applications that potentially do not terminate.  The proposed technique bridges the gap between stateful model checking and DPOR, allowing easy adaptation of many existing DPOR algorithms proposed for stateless model checking~\cite{tasharofi2012transdpor,odpor,statelessmcr,chatterjee2019valuecentricdpor,jensen2015stateless} to the stateful context.

\subsection{The Algorithm}

We first present the states that our algorithm maintains:
}

This section presents our algorithm, which extends DPOR to support stateful model checking of event-driven applications with cyclic state spaces. We first present the states that our algorithm maintains:

\begin{enumerate}
\item {\bf The transition sequence \Trace{}} contains the new transitions that the current execution explores.  Our algorithm explores a given transition in at most one execution.

\item {\bf The state history \hbox{$\mathcal{H}$}\xspace} is a set of program states that have been visited in completed executions.

\item {\bf The state transition graph \RGraph{}} records the states our algorithm has explored thus far.  Nodes in this graph correspond to program states and edges to transitions between program states.
\end{enumerate}

\begin{algorithm}
\DontPrintSemicolon
\SetKwBlock{Begin}{}{end}
\SetAlgoLined
\Begin($\textsc{ExploreAll}\xspace {(}{)}$){
$\hbox{$\mathcal{H}$}\xspace := \emptyset$\;
$\RGraph{} := \emptyset$\;
$\Trace{} := \emptyset$\;
$\textsc{Explore}\xspace {(} \State{0} {)}$\;\label{line:firstexplore}
\While{$\exists \State{}, \: \hbox{\textit{backtrack}}\xspace(\State{}) \neq \hbox{\textit{done}}\xspace(\State{})$}
      {
$\textsc{Explore}\xspace {(} \State{} {)}$\; \label{line:explore}
      }
}
\caption{Top-level exploration algorithm.\label{alg:sdpormain}}
\end{algorithm}

\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{Explore}\xspace {(} \State{} {)}$)
{
\If{$\hbox{\textit{backtrack}}\xspace(\State{}) = \hbox{\textit{done}}\xspace(\State{}) \label{line:ifback}$}{ \label{line:enable-event-start}
\uIf{$\hbox{\textit{done}}\xspace(\State{}) = \hbox{\textit{enabled}}\xspace(\State{})$}{
\uIf{$\hbox{\textit{enabled}}\xspace(\State{}) \text{ is not empty}$}{\label{line:checkenabled}
$\text{select} \: \Event{} \: \in \hbox{\textit{enabled}}\xspace(\State{})\label{line:evselect}$\;
$\text{remove} \: \Event{} \: \text{from} \: \hbox{\textit{done}}\xspace(\State{})$\;
} \Else {
$\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
$\Trace{} := \emptyset$\;
\Return{}
}
} \Else {
$\text{select} \: \Event{} \: \in \hbox{\textit{enabled}}\xspace(\State{}) \setminus \hbox{\textit{done}}\xspace(\State{})$\;\label{line:select-enabled-event}
$\text{add} \: \Event{} \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\State{})$\;
}
} \label{line:enable-event-end}
\While{$\exists \hbox{$b$}\xspace \in \hbox{\textit{backtrack}}\xspace(\State{}) \setminus \hbox{\textit{done}}\xspace(\State{})$\label{line:while}} {
  $\text{add} \: \hbox{$b$}\xspace \: \text{to} \: \State{}.done$\;
  $\Trans{} := \text{\hbox{\emph{next}}\xspace} \left(\State{},\hbox{$b$}\xspace\right)$\;\label{line:algnext}
  $\State{}' := \hbox{\emph{dst}}\xspace\left(\Trans{}\right)$\;\label{line:algdst}
  $\text{add transition} \: \Trans{} \:\text{to}\:\RGraph{}$\;\label{line:addtrans}
  \ForEach{$\Event{} \in \hbox{\textit{enabled}}\xspace(\State{}) \setminus \hbox{\textit{enabled}}\xspace(\State{}')$ \label{line:add-disabled} }{
    $\text{add} \: \Event \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\State{})$\;\label{line:adddisabled}
  }
  $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:call-update-backtrack-set}
  \uIf{$\State{}' \in \hbox{$\mathcal{H}$}\xspace \vee \textsc{IsFullCycle}\xspace\left(\Trans{}\right)$\label{line:termination-condition}} {
    $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\;\label{line:endbacktrack}
    $\text{add} \: \hbox{\emph{states}}\xspace{}(\Trace{}) \: \text{to} \: \hbox{$\mathcal{H}$}\xspace$\;
    $\Trace{} := \emptyset$\;
  }
  \Else {
    \If{$\State{}' \in \hbox{\emph{states}}\xspace{}(\Trace{}) $\label{line:cycle-current-exec-1}} {
      $\textsc{UpdateBacktrackSetsFromGraph}\xspace \left(\Trans{}\right)$\label{line:cycle-current-exec-2}
    }
    $\Trace{} := \Trace{}.\Trans{}$ \;
    $\textsc{Explore}\xspace {(} \hbox{$s$}\xspace' {)}$\;\label{line:call-explore}
  }
}
}
 \caption{Stateful DPOR algorithm for event-driven applications.\label{alg:sdpor}}
\end{algorithm}

Recall that for each reachable state $\State{} \in \hbox{$\textit{States}$}\xspace{}$, our algorithm maintains the $\hbox{\textit{backtrack}}\xspace(\State{})$ set that contains the events to be explored at $\State{}$, the $\hbox{\textit{done}}\xspace(\State{})$ set that contains the events that have already been explored at $\State{}$, and the $\hbox{\textit{enabled}}\xspace(\State{})$ set that contains all events that are enabled at $\State{}$.

Algorithm~\ref{alg:sdpormain} presents the top-level \textsc{ExploreAll}\xspace procedure.  This procedure first invokes the \textsc{Explore}\xspace procedure to start model checking from the initial state.  However,
the presence of cycles in the state space means that our backtracking-based search algorithm may occasionally set new backtracking points for states in completed executions.  The \textsc{ExploreAll}\xspace procedure thus loops over all states that have unexplored items in their backtrack sets and invokes the \textsc{Explore}\xspace procedure to explore those transitions.

Algorithm~\ref{alg:sdpor} describes the logic of the \textsc{Explore}\xspace{} procedure.
The \code{if} statement in line~\ref{line:ifback} checks if the
current state \State{}'s $\hbox{\textit{backtrack}}\xspace$ set is the same as the
current state \State{}'s \hbox{\textit{done}}\xspace set.  If so, the algorithm
selects an event to execute in the next transition.  If some
enabled events are not yet explored, it selects an unexplored event
to add to the current state's $\hbox{\textit{backtrack}}\xspace$ set.  Otherwise, if
the $\hbox{\textit{enabled}}\xspace$ set is not empty, it selects an enabled event to
remove from the \hbox{\textit{done}}\xspace set.  Note that this scenario occurs only if
the execution is continuing past a state match to satisfy the termination condition.

Then the \code{while} loop in line~\ref{line:while} selects an event
$\hbox{$b$}\xspace$ to execute on the current state \State{} and
executes the event $\hbox{$b$}\xspace$ to generate the
transition \Trans{} that leads to a new state $\State{}'$.
At this point, the algorithm knows the memory accesses performed by
the transition \Trans{} and thus can add the event $\hbox{$b$}\xspace$
to the backtrack sets of the previous states.  This is done via the procedure \textsc{UpdateBacktrackSet}\xspace{}.

Traditional DPOR algorithms continue an execution until it terminates.  Since our programs may have cyclic state spaces, this would cause the model checker to potentially not terminate.  Our algorithm instead checks the conditions in
line~\ref{line:termination-condition} to decide whether to terminate the execution.  These checks see whether the new state $\State{}'$ matches a state from a previous execution, or if the current execution revisits a state the current execution previously explored and meets other criteria that are checked in the \textsc{IsFullCycle}\xspace{} procedure.   If so, the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure to set backtracking points, from transitions reachable from $\Trans{}$, to states that can reach $\Trans{}$.  An execution will also terminate if it reaches a state in which no event is enabled (line~\ref{line:checkenabled}).
It then adds the states from the current transition sequence to the set of previously visited states $\hbox{$\mathcal{H}$}\xspace$, resets the current execution transition sequence $\Trace{}$, and backtracks to start a new execution.

If the algorithm has reached a state $\State{}'$ that was previously discovered in this execution, it sets backtracking points by calling the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  Finally, it updates the transition sequence $\Trace{}$ and calls \textsc{Explore}\xspace{}.

\vspace{-2em}
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSetsFromGraph}\xspace {(} \Trans{s}{)}$)
{
  $\RGraph{t} := \lbrace
    \Trans{}\in\RGraph{} \mid \:\: \Trans{s}\hbox{$\:\rightarrow_{r}\:$}\xspace\Trans{}
    \rbrace$\;
  \ForEach{$\Trans{} \in \RGraph{t}$\label{line:check-cycle-in-prev-1}}
  {
     $\textsc{UpdateBacktrackSet}\xspace \left(\Trans{}\right)$\;\label{line:check-cycle-in-prev-2}
  }
}
 \caption{Procedure that updates the backtrack sets of states in previous executions.\label{alg:backtrack-update-graph}}
\end{algorithm}
\vspace{-2em}

Algorithm~\ref{alg:backtrack-update-graph} shows the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.  This procedure takes a transition $\Trans{}$ that connects the current execution to a previously discovered state in the transition graph $\RGraph{}$.  Since our algorithm does \emph{not} explore all of the transitions reachable from the previously discovered state, we need to set the backtracking points that would have been set by these skipped transitions. This procedure therefore computes the set of transitions reachable from the destination state of $\Trans{}$ and invokes $\textsc{UpdateBacktrackSet}\xspace{}$ on each of those transitions to set backtracking points.

\vspace{-2em}
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{IsFullCycle}\xspace {(} \Trans{} {)}$)
{
\If{$\neg \textit{dst}(\Trans{}) \in \hbox{\emph{states}}\xspace{}(\Trace{})$}{\Return{$\textit{false}$}}
  $\Trace{}^{\textit{fc}} := \lbrace
    \Trans{j} \in \Trace{} \mid
        i=\hbox{\emph{first}}\xspace(\Trace{}, \hbox{\emph{dst}}\xspace(\Trans{})), \:\: \text{and} \:\:
        i < j \rbrace \cup \{ \Trans{} \}$\;
  $\Events{\textit{fc}} := \lbrace
    \hbox{\emph{event}}\xspace{}(\Trans{}') \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\; \label{line:fullcycle-fc}
  $\Events{\textit{enabled}} := \lbrace
    \hbox{\textit{enabled}}\xspace(\hbox{\emph{dst}}\xspace{}(\Trans{}')) \mid
        \forall \Trans{}' \in \Trace{}^{\textit{fc}} \: \rbrace$\; \label{line:fullcycle-enabled}
  \Return{$\Events{\textit{fc}} = \Events{\textit{enabled}}\:$\label{line:check-full-cycle}}
}
 \caption{Procedure that checks the looping termination condition: a cycle that contains every event enabled in the cycle.
 \label{alg:complete-full-cycle}}
\end{algorithm}
\vspace{-2em}

Algorithm~\ref{alg:complete-full-cycle} presents
the \textsc{IsFullCycle}\xspace{} procedure.  This procedure first checks if
there is a cycle that contains the transition $\Trans{}$ in the state
space explored by the current execution.  The example from
Figure~\ref{fig:example-non-terminating} shows that such a state match
is not sufficient to terminate the execution as the execution may not
have set the necessary backtracking points.  Our algorithm stops the
exploration of an execution when there is a cycle that has explored
\emph{every event that is enabled in that cycle}.  This ensures that for
every transition \Trans{} in the execution, there is a future
transition \Trans{e} for each enabled event \Event{} in the cycle that
can set a backtracking point if \Trans{} and \Trans{e} conflict.

Algorithm~\ref{alg:backtrack-updatev2} presents the \textsc{UpdateBacktrackSet}\xspace{} procedure, which sets backtracking points.  There are two differences between our algorithm and traditional DPOR algorithms.  First, since our algorithm supports programs with cyclic state spaces, it is possible that the algorithm has discovered multiple paths from the start state $\State{0}$ to the current transition $\Trans{}$.  Thus, the algorithm must potentially set backtracking points on multiple different paths.  We address this issue using a backwards depth first search traversal of the $\RGraph{}$ graph.   Second, since our transitions correspond to events, they may potentially access multiple different memory locations and thus the backtracking algorithm potentially needs to set separate backtracking points for each of these memory locations.

The \textsc{UpdateBacktrackSetDFS}\xspace{} procedure implements a backwards
depth first traversal to set backtracking points.  The procedure takes
the following parameters: \Trans{\text{curr}} is the current transition in the
DFS, \Trans{\text{conf}} is the transition that we are currently setting
a backtracking point for, \Access{} is the set of accesses that the
algorithm searches for conflicts for, and \Transitions{\text{exp}} is
the set of transitions that the algorithm has explored down this
search path.  Recall that accesses consist of both an operation, \hbox{\emph{i.e.},}\xspace a read or write, and a memory location.  Conflicts are defined in the usual way---writes to a memory location conflict with reads or writes to the same location.

\vspace{-2em}
\begin{algorithm}
\small
\DontPrintSemicolon
\SetAlgoLined
\SetKwBlock{Begin}{}{end}
\Begin($\textsc{UpdateBacktrackSet}\xspace {(} \Trans{} {)}$)
{
 $\textsc{UpdateBacktrackSetDFS}\xspace\left( \Trans{}, \Trans{}, \textit{accesses}(\Trans{}), \{ \Trans{} \} \right)$
}

\Begin($\textsc{UpdateBacktrackSetDFS}\xspace {(} \Trans{\text{curr}}, \Trans{\text{conf}},  \Access{}, \Transitions{\text{exp}} {)}$)
{
  \ForEach{$\Trans{b} \in \textit{pred}_{\RGraph{}}(\Trans{\text{curr}}) \setminus \Transitions{\text{exp}} $\label{lin:loop}}{
  $\Access{b} := \textit{accesses}(\Trans{b})$\;
  $\Trans{\text{conf}}' := \Trans{\text{conf}}$\;
  \If{$\exists a \in \Access{}, \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})$\label{lin:conflict}} {
   \uIf{$\hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \in \hbox{\textit{enabled}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b}))$} {
   $\text{add} \: \hbox{\emph{event}}\xspace(\Trans{\text{conf}}) \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b}))$\;
   } \Else {
      $\text{add} \: \hbox{\textit{enabled}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b})) \: \text{to} \: \hbox{\textit{backtrack}}\xspace(\hbox{\emph{src}}\xspace(\Trans{b}))$\;
   }
   $\Trans{\text{conf}}' := \Trans{b}$\;
  }
  $\Access{r} := \{ a \in \Access{} \mid \neg \exists a_{b} \in \Access{b}, \hbox{\textit{conflicts}}\xspace(a, a_{b})\}$\;  \label{lin:newaccesses}
  $\textsc{UpdateBacktrackSetDFS}\xspace \left( \Trans{b}, \Trans{\text{conf}}', \Access{r}, \Transitions{\text{exp}} \: \cup \{ \Trans{b} \} \right)$
  }
}

 \caption{Procedure that updates the backtrack sets of states for previously executed transitions that conflict with the current transition in the search stack.
  \label{alg:backtrack-updatev2}}
\end{algorithm}
\vspace{-2em}

Line~\ref{lin:loop} loops over each transition \Trans{b} that
immediately precedes transition \Trans{\text{curr}} in the state transition
graph and has not been explored.  Line~\ref{lin:conflict} checks for
conflicts between the accesses of \Trans{b} and the access
set \Access{} for the DFS.  If a conflict is detected, the
algorithm adds the event for transition \Trans{\text{conf}} to the backtrack set.
Line~\ref{lin:newaccesses} removes the accesses that conflicted
with transition \Trans{b}.  The search procedure then recursively
calls itself.  If the current transition \Trans{b} conflicts with the
transition \Trans{\text{conf}} for which we are setting a backtracking point, then
it is possible that the behavior we are interested in for \Trans{\text{conf}}
requires that \Trans{b} be executed first.  Thus, if there is a
conflict between \Trans{b} and \Trans{\text{conf}}, we pass \Trans{b} as the
conflict transition parameter to the recursive calls
to \textsc{UpdateBacktrackSetDFS}\xspace{}.

\rahmadi{
\inarxivversion{\appdx{Appendix~\ref{sec:proof}} proves correctness properties for our DPOR algorithm.}
\invmcaiversion{\appdx{Appendix B} proves correctness properties for our DPOR algorithm.}
\inarxivversion{\appdx{Appendix~\ref{sec:example}} revisits the example shown in}
\invmcaiversion{\appdx{Appendix C} revisits the example shown in}
Figure~\ref{fig:example-missing-execution}. It describes how our DPOR algorithm explores all executions in Figure~\ref{fig:example-missing-execution}, including Figure~\ref{fig:example-missing-execution}-i.
}

\mycomment{
\mysubsection{Discussion}

Although the algorithm we present is designed to explore executions of event-driven systems, the key ideas are also applicable to model checking multithreaded programs.  To the best of our knowledge, there are no correct algorithms for stateful dynamic partial order reduction for multithreaded applications that potentially do not terminate.  The proposed technique bridges the gap between stateful model checking and DPOR, allowing easy adaptation of many existing DPOR algorithms proposed for stateless model checking~\cite{tasharofi2012transdpor,odpor,statelessmcr,chatterjee2019valuecentricdpor,jensen2015stateless} to the stateful context.
}
\section*{Appendices}

\title{Appendices}
\author{}
\institute{}
\maketitle

\renewcommand\thefigure{A.\arabic{figure}}
\renewcommand\thetable{A.\arabic{table}}
\setcounter{figure}{0}
\setcounter{table}{0}

\mysection{Example Event-Driven System\label{sec:example-event-driven-system}}
Figure~\ref{fig:concurrency-model} depicts components of an example
event-driven system.  The application \code{SmartLightApp} in the
figure is developed using this concurrency model and may run in the
cloud.  It has a fixed set of events that are associated with it and
their respective event handlers. It also has logic to decide whether
to perform specific actions based on the triggering events.  For
example, \code{SmartLightApp} can turn on and off a light bulb based
on specific triggers.

\begin{figure}[t]
    \centering
	\includegraphics[width=0.8\linewidth]{figures/concurrency-model.pdf}

    \caption{Example Event-driven System.
      \label{fig:concurrency-model}}

\end{figure}

Our example event-driven system has the following 4 components:

\noindent \textbf{(I) Event handlers:} The application code is
composed of a set of event handlers.  Each event handler processes a
certain class of events.  When the event handler executes, it receives
the appropriate event object.  When the runtime system dequeues an event $\Event{}$, it executes the relevant event handler.

\noindent \textbf{(II) External events:}
An application can have a number of sources that deliver triggering events:
(1) a cloud service
(\hbox{\emph{e.g.},}\xspace a cloud service that reports the current outdoor temperature),
(2) a service that is running on a computing device
(\hbox{\emph{e.g.},}\xspace a messaging service that sends a trigger whenever there is an incoming message),
(3) a device
(\hbox{\emph{e.g.},}\xspace an illuminance sensor that sends a trigger whenever there is a change of
light intensity in its environment),
(4) a scheduled event within the application
(\hbox{\emph{e.g.},}\xspace turn on a light bulb at 6pm), or
(5) an event generated by the application that triggers another component of that app:
these events potentially change the app's state (\hbox{\emph{e.g.},}\xspace a scheduled mode change to \textit{night} may trigger a light bulb to turn on).

\noindent \textbf{(III) Internal events:}
An application can directly generate events, for example to:
(a) actuate a certain device
(\hbox{\emph{e.g.},}\xspace turn on a light bulb) or
(b) trigger another app running on a computing device
(\hbox{\emph{e.g.},}\xspace trigger an email to be sent from an email app).

\noindent \textbf{(IV) Event queue:}
All events are pushed into an event queue to be delivered by the runtime to the appropriate event handlers.

Examples of platforms with this event-driven model include the Samsung SmartThings platform~\cite{smartthings,smartthings-github}, a popular platform used to create
apps that control devices to automate tasks for a smart home.

\input{proof}

\input{proof2}

\mysection{Example for Stateful DPOR Algorithm\label{sec:example}}
Here, we illustrate how our new stateful DPOR algorithm works with the example from
Figure~\ref{fig:example-missing-execution}.

First, the algorithm starts by calling the \textsc{ExploreAll}\xspace procedure in
Algorithm~\ref{alg:sdpormain}. After the state is initialized, it calls the \textsc{Explore}\xspace procedure in line~\ref{line:firstexplore}.
Let us suppose that the algorithm first explores the execution shown in
Figure~\ref{fig:example-missing-execution}-a.
Line~\ref{line:evselect} in Algorithm~\ref{alg:sdpor} chooses
event \code{e$_1$} among the enabled events.
Next, event \code{e$_1$} is executed: it produces a new state $\State{1}$ and adds
the corresponding transition to \RGraph{}.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure,
but it does not update the backtrack set of any state since the algorithm has only
explored one event.
Line~\ref{line:termination-condition} evaluates to \emph{false} since the termination
condition is not met yet: $\State{1}$ is not found in \hbox{$\mathcal{H}$}\xspace and
it is not a full cycle.
Line~\ref{line:cycle-current-exec-1} also evaluates to \emph{false} since $\State{1}$
is not found in \Trace{}.
Finally, Algorithm~\ref{alg:sdpormain} calls the \textsc{Explore}\xspace procedure recursively to
explore the next transition.

In the subsequent transitions, line~\ref{line:evselect} chooses event \code{e$_2$} and then event \code{e$_3$}.
For each of these events, the \textsc{UpdateBacktrackSet}\xspace{} procedure invokes
\textsc{UpdateBacktrackSetDFS}\xspace{} in Algorithm~\ref{alg:backtrack-updatev2} to
perform the DFS, find conflicts, and update the corresponding backtrack
sets---it puts event \code{e$_2$} in the
backtrack set of $\State{0}$ and event \code{e$_3$} in the backtrack set of
$\State{1}$ as it finds conflicts between events \code{e$_1$}
and \code{e$_2$} that access the shared variable \code{z}, and events \code{e$_2$} and \code{e$_3$} that access the shared variable \code{y}.
Event \code{e$_2$} produces a new state $\State{2}$, while \code{e$_3$}
causes the execution to revisit state $\State{1}$.
At this point, line~\ref{line:cycle-current-exec-1} evaluates
to \emph{true} and the algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{}
procedure.
This procedure (shown in Algorithm~\ref{alg:backtrack-update-graph}) finds that
the transition that corresponds to event \code{e$_2$} is reachable from
the transition that corresponds to event \code{e$_3$} in \RGraph{} and calls
the \textsc{UpdateBacktrackSet}\xspace{} procedure.
\textsc{UpdateBacktrackSet}\xspace{} puts event \code{e$_2$} in the backtrack
set of $\State{2}$ as it finds a conflict between the events \code{e$_2$} and \code{e$_3$}
that access the shared variable \code{y}.

In the final transition of the first execution, line~\ref{line:evselect} chooses event \code{e$_4$}.
The \textsc{UpdateBacktrackSet}\xspace{} procedure finds a conflict between \code{e$_4$} and \code{e$_1$} that both access the shared variable \code{x}:
it puts event \code{e$_4$} in the backtrack set of state $\State{0}$.
Event \code{e$_4$} directs the algorithm to revisit state $\State{0}$. This time
line~\ref{line:termination-condition} evaluates to \emph{true} as the termination
condition is satisfied.
The algorithm calls the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure---it finds
that the events \code{e$_1$}, \code{e$_2$}, and \code{e$_3$} are reachable from the transition for event \code{e$_4$}.
Next, it calls the \textsc{UpdateBacktrackSet}\xspace{} procedure for the reachable
transitions. Among all the conflicts found by the procedure, it
puts event \code{e$_1$} in the backtrack set of state $\State{1}$ as a new backtracking point
for the conflict between the events \code{e$_1$} and \code{e$_4$}.
Then, the algorithm saves the visited states in this execution into \hbox{$\mathcal{H}$}\xspace
and terminates the execution.

Now the algorithm explores the executions in the backtrack sets.
This time, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses
the events \code{e$_1$} and \code{e$_3$} in the backtrack set of $\State{1}$, and
event \code{e$_2$} in the backtrack set of $\State{2}$.
These executions terminate quickly as they immediately revisit states
$\State{1}$ and $\State{2}$ that have been stored in \hbox{$\mathcal{H}$}\xspace
at the end of the first execution---line~\ref{line:termination-condition} evaluates to \emph{true}.
These executions are shown in
Figures~\ref{fig:example-missing-execution}-b, c, and d.

Next, let us suppose that the algorithm explores the execution from event \code{e$_2$}
in the backtrack set of $\State{0}$ shown in Figure~\ref{fig:example-missing-execution}-e.
At first, line~\ref{line:while} in Algorithm~\ref{alg:sdpor} chooses event \code{e$_2$} in the backtrack set of $\State{0}$ that produces a new state $\State{3}$.
When line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{}
procedure, this invocation allows the algorithm to
perform the DFS backward to find conflicts between event \code{e$_2$} and the transitions in
previous executions.
Thus, it finds a conflict between the current event \code{e$_2$} and event \code{e$_3$} from the previous execution
shown in Figure~\ref{fig:example-missing-execution}-b---both access the shared
variable \code{y}. It then puts event \code{e$_2$} into the backtrack set of state $\State{1}$.
Since lines~\ref{line:termination-condition} and~\ref{line:cycle-current-exec-1}
evaluate to \emph{false}, the algorithm calls the \textsc{Explore}\xspace{} procedure recursively.
In the second recursion, let us suppose that event \code{e$_3$} is chosen.
Line~\ref{line:call-update-backtrack-set} calls the \textsc{UpdateBacktrackSet}\xspace{} procedure:
it finds a conflict between events \code{e$_3$} and \code{e$_2$} and puts event \code{e$_3$}
in the backtrack set of $\State{0}$.
Since event \code{e$_3$} directs the execution to revisit $\State{0}$ that is already in \hbox{$\mathcal{H}$}\xspace,
line~\ref{line:termination-condition} evaluates to \emph{true} and
the algorithm invokes the \textsc{UpdateBacktrackSetsFromGraph}\xspace{} procedure.
This procedure finds a conflict between event \code{$e_2$} of the current execution as a
reachable transition from $\State{0}$ and event \code{$e_3$} from $\State{3}$ that was just executed; it next puts event \code{e$_2$} into the backtrack set of
$\State{3}$ as a new backtracking point.  At this point, this execution terminates.

The algorithm explores executions from the remaining backtracking points,
namely, executions from event \code{e$_2$} in the backtrack set of $\State{3}$, and
\code{e$_3$} and \code{e$_4$} in the backtrack set of $\State{0}$.
These executions are shown in Figures~\ref{fig:example-missing-execution}-f, g, and h.  In the execution of event \code{e$_4$} from state $\State{0}$, the algorithm detects a conflict with the prior execution of event \code{e$_3$} from state $\State{3}$; it puts event \code{e$_4$} into the backtrack set of $\State{3}$.

The algorithm then explores the execution shown in Figure~\ref{fig:example-missing-execution}-i, which contains the assertion. The model checker stops its exploration and returns an error.

\mysection{Optimizing Traversals\label{sec:optimizing-traversals}}

The algorithm as presented performs graph traversals to identify conflicts.  Our implementation implements several optimizations to reduce traversal overheads.

The procedure \textsc{UpdateBacktrackSetsFromGraph}\xspace{} traverses the graph after a state match to discover potential conflicting transactions from previous executions.  We eliminate many of these graph traversals by caching the results of the graph search the first time \textsc{UpdateBacktrackSetsFromGraph}\xspace is called.  The results can be cached for each state as a summary of the potentially conflicting transitions that are reachable from the given state.
A potentially conflicting transition, namely $\Trans{\text{conf}}$, is cached in the form of a tuple that contains the event \hbox{\emph{event}}\xspace{}(\Trans{\text{conf}}) and its corresponding accesses \Access{}.

This cached summary is updated by \textsc{UpdateBacktrackSet}\xspace{} during its backwards graph traversal.
With this summary, the algorithm can efficiently re-explore the previously explored transitions: (1) when a previously discovered state \State{} is reached, the summary contains all potentially conflicting events reachable from \State{}, and (2) when performing backwards depth first search traversal, the algorithm can stop the traversal at any state \State{b} whenever the algorithm finds that the current \hbox{\emph{event}}\xspace{}(\Trans{\text{conf}}) and \Access{} are already cached in the summary for \State{b}.

\begin{table*}[!htb]
  \centering
    \caption{Model-checked pairs that finished with DPOR but unfinished (\hbox{\emph{i.e.},}\xspace "Unf") without DPOR. \textbf{Evt.} is number of events and \textbf{Time} is in seconds.
    \label{tab:results-dnt-no-dpor-complete}}
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        1  & initial-state-event-streamer--thermostat-auto-off                 & 78 & Unf & Unf & Unf & 7,146 & 25,850 & 3,285 \\ \hline
        2  & unbuffered-event-sender--hvac-auto-off.smartapp                   & 78 & Unf & Unf & Unf & 7,123 & 26,016 & 3,432 \\ \hline
        3  & initial-state-event-sender--hvac-auto-off.smartapp                & 78 & Unf & Unf & Unf & 7,007 & 25,220 & 3,215 \\ \hline
        4  & initial-state-event-streamer--hvac-auto-off.smartapp              & 78 & Unf & Unf & Unf & 7,007 & 25,220 & 3,230 \\ \hline
        5  & initialstate-smart-app-v1.2.0--hvac-auto-off.smartapp             & 78 & Unf & Unf & Unf & 7,007 & 25,220 & 3,290 \\ \hline
        6  & lighting-director--circadian-daylight                             & 19 & Unf & Unf & Unf & 6,553 & 33,045 & 6,604 \\ \hline
        7  & initial-state-event-streamer--thermostat                          & 81 & Unf & Unf & Unf & 5,646 & 26,620 & 2,965 \\ \hline
        8  & forgiving-security--unbuffered-event-sender                       & 80 & Unf & Unf & Unf & 5,019 & 45,208 & 6,259 \\ \hline
        9  & forgiving-security--initial-state-event-streamer                  & 80 & Unf & Unf & Unf & 4,902 & 44,230 & 5,697 \\ \hline
        10 & forgiving-security--initialstate-smart-app-v1.2.0                 & 80 & Unf & Unf & Unf & 4,902 & 44,230 & 5,702 \\ \hline
        11 & forgiving-security--initial-state-event-sender                    & 80 & Unf & Unf & Unf & 4,902 & 44,230 & 5,716 \\ \hline
        12 & unbuffered-event-sender--thermostat-window-check                  & 79 & Unf & Unf & Unf & 4,546 & 17,411 & 2,069 \\ \hline
        13 & hue-mood-lighting--Hue-Party-Mode                                 & 18 & Unf & Unf & Unf & 3,457 & 49,138 & 6,132 \\ \hline
        14 & thermostat--initial-state-event-sender                            & 82 & Unf & Unf & Unf & 2,530 & 13,105 & 1,621 \\ \hline
        15 & thermostat--initialstate-smart-app-v1.2.0                         & 82 & Unf & Unf & Unf & 2,530 & 13,105 & 1,606 \\ \hline
        16 & thermostat--unbuffered-event-sender                               & 82 & Unf & Unf & Unf & 2,517 & 12,880 & 1,626 \\ \hline
        17 & initial-state-event-streamer--unlock-it-when-i-arrive             & 79 & Unf & Unf & Unf & 2,459 & 8,825  & 998   \\ \hline
        18 & lights-off-with-no-motion-and-presence--smart-security            & 11 & Unf & Unf & Unf & 2,299 & 11,261 & 1,475 \\ \hline
        19 & initial-state-event-streamer--lock-it-when-i-leave                & 78 & Unf & Unf & Unf & 1,750 & 5,387  & 595   \\ \hline
        20 & smart-nightlight--step-notifier                                   & 13 & Unf & Unf & Unf & 1,568 & 6,460  & 644   \\ \hline
        21 & initial-state-event-sender--NotifyIfLeftUnlocked                  & 78 & Unf & Unf & Unf & 1,482 & 3,830  & 428   \\ \hline
        22 & initial-state-event-streamer--NotifyIfLeftUnlocked                & 78 & Unf & Unf & Unf & 1,482 & 3,830  & 439   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[t]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        23 & initialstate-smart-app-v1.2.0--NotifyIfLeftUnlocked               & 78 & Unf & Unf & Unf & 1,482 & 3,830  & 437   \\ \hline
        24 & initial-state-event-streamer--auto-lock-door.smartapp             & 80 & Unf & Unf & Unf & 1,272 & 3,743  & 422   \\ \hline
        25 & lock-it-when-i-leave--unbuffered-event-sender                     & 78 & Unf & Unf & Unf & 1,234 & 3,482  & 438   \\ \hline
        26 & lock-it-when-i-leave--initial-state-event-sender                  & 78 & Unf & Unf & Unf & 1,192 & 3,369  & 424   \\ \hline
        27 & lock-it-when-i-leave--initialstate-smart-app-v1.2.0               & 78 & Unf & Unf & Unf & 1,192 & 3,369  & 428   \\ \hline
        28 & medicine-management-temp-motion--initial-state-event-sender       & 79 & Unf & Unf & Unf & 939   & 2,277  & 275   \\ \hline
        29 & medicine-management-temp-motion--initialstate-smart-app-v1.2.0    & 79 & Unf & Unf & Unf & 939   & 2,277  & 275   \\ \hline
        30 & medicine-management-temp-motion--unbuffered-event-sender          & 79 & Unf & Unf & Unf & 933   & 2,277  & 283   \\ \hline
        31 & initial-state-event-streamer--DeviceTamperAlarm                   & 80 & Unf & Unf & Unf & 860   & 2,439  & 277   \\ \hline
        32 & NotifyIfLeftUnlocked--unbuffered-event-sender                     & 78 & Unf & Unf & Unf & 811   & 2,181  & 270   \\ \hline
        33 & initial-state-event-streamer--smart-auto-lock-unlock              & 80 & Unf & Unf & Unf & 780   & 1,986  & 228   \\ \hline
        34 & initial-state-event-streamer--medicine-management-contact-sensor  & 78 & Unf & Unf & Unf & 738   & 1,657  & 197   \\ \hline
        35 & unlock-it-when-i-arrive--initial-state-event-sender               & 77 & Unf & Unf & Unf & 622   & 2,402  & 273   \\ \hline
        36 & unlock-it-when-i-arrive--initialstate-smart-app-v1.2.0            & 77 & Unf & Unf & Unf & 622   & 2,402  & 274   \\ \hline
        37 & initial-state-event-streamer--lock-it-at-a-specific-time          & 79 & Unf & Unf & Unf & 621   & 1,451  & 175   \\ \hline
        38 & unlock-it-when-i-arrive--unbuffered-event-sender                  & 77 & Unf & Unf & Unf & 618   & 2,405  & 277   \\ \hline
        39 & medicine-management-contact-sensor--initial-state-event-sender    & 78 & Unf & Unf & Unf & 605   & 1,241  & 156   \\ \hline
        40 & medicine-management-contact-sensor--initialstate-smart-app-v1.2.0 & 78 & Unf & Unf & Unf & 605   & 1,241  & 163   \\ \hline
        41 & DeviceTamperAlarm--initial-state-event-sender                     & 80 & Unf & Unf & Unf & 602   & 1,540  & 206   \\ \hline
        42 & DeviceTamperAlarm--initialstate-smart-app-v1.2.0                  & 80 & Unf & Unf & Unf & 602   & 1,540  & 209   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[t]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        43 & medicine-management-contact-sensor--unbuffered-event-sender       & 78 & Unf & Unf & Unf & 602   & 1,240  & 168   \\ \hline
        44 & DeviceTamperAlarm--unbuffered-event-sender                        & 80 & Unf & Unf & Unf & 600   & 1,534  & 217   \\ \hline
        45 & close-the-valve--initial-state-event-sender                       & 78 & Unf & Unf & Unf & 584   & 1,261  & 162   \\ \hline
        46 & close-the-valve--initial-state-event-streamer                     & 78 & Unf & Unf & Unf & 584   & 1,261  & 164   \\ \hline
        47 & close-the-valve--initialstate-smart-app-v1.2.0                    & 78 & Unf & Unf & Unf & 584   & 1,261  & 162   \\ \hline
        48 & close-the-valve--unbuffered-event-sender                          & 78 & Unf & Unf & Unf & 581   & 1,259  & 172   \\ \hline
        49 & initial-state-event-streamer--medicine-management-temp-motion     & 79 & Unf & Unf & Unf & 549   & 1,298  & 172   \\ \hline
        50 & lock-it-at-a-specific-time--initial-state-event-sender            & 79 & Unf & Unf & Unf & 502   & 1,080  & 141   \\ \hline
        51 & lock-it-at-a-specific-time--initialstate-smart-app-v1.2.0         & 79 & Unf & Unf & Unf & 502   & 1,080  & 140   \\ \hline
        52 & lock-it-at-a-specific-time--unbuffered-event-sender               & 79 & Unf & Unf & Unf & 500   & 1,079  & 146   \\ \hline
        53 & auto-lock-door.smartapp--initial-state-event-sender               & 80 & Unf & Unf & Unf & 498   & 1,617  & 196   \\ \hline
        54 & auto-lock-door.smartapp--initialstate-smart-app-v1.2.0            & 80 & Unf & Unf & Unf & 498   & 1,617  & 194   \\ \hline
        55 & auto-lock-door.smartapp--unbuffered-event-sender                  & 80 & Unf & Unf & Unf & 495   & 1,617  & 202   \\ \hline
        56 & initial-state-event-sender--initialstate-smart-app-v1.2.0         & 78 & Unf & Unf & Unf & 473   & 1,054  & 143   \\ \hline
        57 & initial-state-event-streamer--initial-state-event-sender          & 78 & Unf & Unf & Unf & 473   & 1,054  & 143   \\ \hline
        58 & initial-state-event-streamer--initialstate-smart-app-v1.2.0       & 78 & Unf & Unf & Unf & 473   & 1,054  & 143   \\ \hline
        59 & initial-state-event-sender--unbuffered-event-sender               & 78 & Unf & Unf & Unf & 471   & 1,053  & 147   \\ \hline
        60 & initial-state-event-streamer--unbuffered-event-sender             & 78 & Unf & Unf & Unf & 471   & 1,053  & 147   \\ \hline
        61 & initialstate-smart-app-v1.2.0--unbuffered-event-sender            & 78 & Unf & Unf & Unf & 471   & 1,053  & 145   \\ \hline
        62 & enhanced-auto-lock-door--initial-state-event-sender               & 80 & Unf & Unf & Unf & 309   & 1,139  & 158   \\ \hline
        63 & enhanced-auto-lock-door--initial-state-event-streamer             & 80 & Unf & Unf & Unf & 309   & 1,139  & 158   \\ \hline
        64 & enhanced-auto-lock-door--initialstate-smart-app-v1.2.0            & 80 & Unf & Unf & Unf & 309   & 1,139  & 162   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[t]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | c | c | c | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        65 & smart-auto-lock-unlock--initial-state-event-sender                & 80 & Unf & Unf & Unf & 309   & 1,140  & 159   \\ \hline
        66 & smart-auto-lock-unlock--initialstate-smart-app-v1.2.0             & 80 & Unf & Unf & Unf & 309   & 1,140  & 161   \\ \hline
        67 & enhanced-auto-lock-door--unbuffered-event-sender                  & 80 & Unf & Unf & Unf & 307   & 1,139  & 164   \\ \hline
        68 & smart-auto-lock-unlock--unbuffered-event-sender                   & 80 & Unf & Unf & Unf & 307   & 1,140  & 165   \\ \hline
        69 & lighting-director--turn-on-before-sunset                          & 11 & Unf & Unf & Unf & 257   & 960    & 77    \\
    \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
    \caption{Model-checked pairs that finished with or without DPOR. \textbf{Evt.} is number of events and \textbf{Time} is in seconds.
    \label{tab:results-both-terminated-complete}}
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        1  & smart-nightlight--ecobeeAwayFromHome                              & 14 & 16,441 & 76,720  & 5,059 & 11,743 & 46,196 & 5,498 \\ \hline
        2  & step-notifier--ecobeeAwayFromHome                                 & 11 & 14,401 & 52,800  & 4,885 & 11,490 & 35,142 & 5,079 \\ \hline
        3  & smart-security--ecobeeAwayFromHome                                & 11 & 14,301 & 47,608  & 4,385 & 8,187  & 21,269 & 2,980 \\ \hline
        4  & keep-me-cozy--whole-house-fan                                     & 17 & 8,793  & 149,464 & 4,736 & 8,776  & 95,084 & 6,043 \\ \hline
        5  & keep-me-cozy-ii--thermostat-window-check                          & 13 & 8,764  & 113,919 & 4,070 & 7,884  & 59,342 & 4,515 \\ \hline
        6  & step-notifier--mini-hue-controller                                & 6  & 7,967  & 47,796  & 2,063 & 7,907  & 40,045 & 3,582 \\ \hline
        7  & keep-me-cozy--thermostat-mode-director                            & 12 & 7,633  & 91,584  & 3,259 & 6,913  & 49,850 & 3,652 \\ \hline
        8  & lighting-director--step-notifier                                  & 14 & 7,611  & 106,540 & 5,278 & 2,723  & 25,295 & 2,552 \\ \hline
        9  & smart-alarm--DeviceTamperAlarm                                    & 15 & 5,665  & 84,960  & 3,559 & 3,437  & 40,906 & 4,441 \\ \hline
        10 & forgiving-security--smart-alarm                                   & 13 & 5,545  & 72,072  & 3,134 & 4,903  & 52,205 & 5,728 \\ \hline
        11 & smart-light-timer-x-minutes-unless-already-on--ecobeeAwayFromHome & 9  & 3,775  & 11,160  & 992   & 2,460  & 5,418  & 645   \\ \hline
        12 & smart-security--vacation-lighting-director                        & 12 & 3,759  & 33,264  & 1,641 & 2,849  & 14,108 & 1,604 \\ \hline
        13 & smart-security--turn-on-only-if-i-arrive-after-sunset             & 11 & 3,437  & 28,028  & 1,471 & 2,553  & 11,624 & 1,396 \\ \hline
        14 & smart-security--turn-it-on-when-im-here                           & 11 & 3,437  & 28,028  & 1,470 & 2,549  & 11,878 & 1,401 \\ \hline
        15 & vacation-lighting-director--ecobeeAwayFromHome                    & 10 & 3,313  & 11,040  & 856   & 2,158  & 5,779  & 652   \\ \hline
        16 & smart-light-timer-x-minutes-unless-already-on--step-notifier      & 10 & 3,213  & 32,120  & 2,402 & 2,343  & 11,149 & 1,122 \\ \hline
        17 & thermostat--thermostat-mode-director                              & 12 & 3,169  & 38,016  & 1,454 & 3,157  & 28,437 & 2,470 \\ \hline
        18 & ecobeeAwayFromHome--NotifyIfLeftUnlocked                          & 9  & 2,329  & 6,984   & 623   & 1,714  & 377    & 454   \\ \hline
        19 & keep-me-cozy--thermostat-window-check                             & 14 & 2,185  & 30,576  & 1,148 & 2,176  & 20,458 & 1,576 \\ \hline
        20 & smart-security--turn-on-before-sunset                             & 10 & 2,175  & 16,240  & 855   & 1,588  & 6,119  & 782   \\ \hline
        21 & smart-security--turn-on-at-sunset                                 & 10 & 2,175  & 16,240  & 909   & 1,542  & 5,599  & 783   \\ \hline
        22 & keep-me-cozy--thermostat                                          & 12 & 2,017  & 519,960 & 801   & 2,017  & 15,593 & 1,193 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        23 & smart-security--turn-it-on-when-it-opens                          & 10 & 1,763  & 12,760  & 700   & 1,340  & 5,913  & 737   \\ \hline
        24 & smart-security--undead-early-warning                              & 10 & 1,763  & 12,760  & 688   & 1,340  & 5,913  & 732   \\ \hline
        25 & photo-burst-when--ecobeeAwayFromHome                              & 13 & 1,409  & 4,576   & 599   & 1,109  & 2,695  & 627   \\ \hline
        26 & auto-lock-door.smartapp--ecobeeAwayFromHome                       & 11 & 1,381  & 4,048   & 373   & 927    & 2,357  & 294   \\ \hline
        27 & lighting-director--vacation-lighting-director                     & 13 & 1,373  & 17,836  & 623   & 782    & 4,943  & 395   \\ \hline
        28 & let-there-be-dark--smart-security                                 & 9  & 1,279  & 8,460   & 494   & 881    & 3,531  & 413   \\ \hline
        29 & thermostat--whole-house-fan                                       & 13 & 1,273  & 16,536  & 690   & 747    & 8,237  & 790   \\ \hline
        30 & let-there-be-dark--ecobeeAwayFromHome                             & 7  & 1,240  & 3,052   & 289   & 924    & 1,895  & 263   \\ \hline
        31 & forgiving-security--smart-security                                & 11 & 1,165  & 7,524   & 620   & 696    & 2,838  & 547   \\ \hline
        32 & lighting-director--smart-light-timer-x-minutes-unless-already-on  & 12 & 1,068  & 12,804  & 465   & 424    & 3,104  & 271   \\ \hline
        33 & smart-security--turn-off-with-motion                              & 9  & 989    & 6,732   & 371   & 732    & 3,050  & 394   \\ \hline
        34 & whole-house-fan--hvac-auto-off.smartapp                           & 9  & 958    & 8,613   & 296   & 935    & 6,852  & 722   \\ \hline
        35 & thermostat-auto-off--whole-house-fan                              & 9  & 958    & 8,613   & 286   & 660    & 5,377  & 424   \\ \hline
        36 & smart-security--turn-on-by-zip-code                               & 9  & 941    & 6,300   & 363   & 705    & 2,855  & 386   \\ \hline
        37 & laundry-monitor--step-notifier                                    & 6  & 873    & 3,954   & 263   & 170    & 327    & 54    \\ \hline
        38 & smart-security--DeviceTamperAlarm                                 & 11 & 872    & 7,546   & 371   & 593    & 2,939  & 376   \\ \hline
        39 & make-it-so--whole-house-fan                                       & 13 & 841    & 10,920  & 409   & 751    & 7,436  & 599   \\ \hline
        40 & step-notifier--BetterLaundryMonitor                               & 6  & 810    & 4,854   & 260   & 453    & 1,283  & 144   \\ \hline
        41 & thermostat-auto-off--thermostat-mode-director                     & 10 & 763    & 7,620   & 281   & 714    & 5,805  & 490   \\ \hline
        42 & medicine-management-temp-motion--circadian-daylight               & 14 & 756    & 10,500  & 431   & 350    & 3,133  & 293   \\ \hline
        43 & smart-nightlight--turn-it-on-when-im-here                         & 11 & 736    & 8,085   & 304   & 238    & 1,085  & 110   \\ \hline
        44 & smart-nightlight--turn-on-only-if-i-arrive-after-sunset           & 11 & 736    & 8,085   & 309   & 238    & 1,085  & 109   \\ \hline
        45 & lighting-director--turn-on-at-sunset                              & 11 & 734    & 8,063   & 294   & 241    & 927    & 80    \\ \hline
        46 & unlock-it-when-i-arrive--ecobeeAwayFromHome                       & 8  & 733    & 2,048   & 192   & 543    & 1,179  & 170   \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        47 & hall-light-welcome-home--lighting-director                        & 12 & 675    & 8,088   & 335   & 225    & 1,857  & 172   \\ \hline
        48 & lights-off-with-no-motion-and-presence--ecobeeAwayFromHome        & 7  & 605    & 1,400   & 178   & 400    & 827    & 140   \\ \hline
        49 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 170   & 505    & 1,000  & 198   \\ \hline
        50 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 179   & 505    & 1,000  & 198   \\ \hline
        51 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 170   & 505    & 999    & 200   \\ \hline
        52 & good-night-house--ecobeeAwayFromHome                              & 7  & 599    & 1,288   & 179   & 505    & 999    & 200   \\ \hline
        53 & keep-me-cozy--WindowOrDoorOpen                                    & 10 & 589    & 5,880   & 238   & 589    & 4,338  & 365   \\ \hline
        54 & keep-me-cozy--hvac-auto-off.smartapp                              & 10 & 589    & 5,880   & 190   & 589    & 4,066  & 290   \\ \hline
        55 & keep-me-cozy--thermostat-auto-off                                 & 10 & 589    & 5,880   & 188   & 589    & 4,126  & 283   \\ \hline
        56 & lock-it-at-a-specific-time--ecobeeAwayFromHome                    & 8  & 553    & 1,472   & 142   & 473    & 1,018  & 152   \\ \hline
        57 & darken-behind-me--ecobeeAwayFromHome                              & 8  & 553    & 1,472   & 154   & 437    & 1,019  & 155   \\ \hline
        58 & turn-off-with-motion--ecobeeAwayFromHome                          & 7  & 553    & 1,288   & 144   & 410    & 848    & 133   \\ \hline
        59 & lights-off-with-no-motion-and-presence--step-notifier             & 8  & 506    & 4,040   & 224   & 309    & 1,309  & 142   \\ \hline
        60 & medicine-management-contact-sensor--circadian-daylight            & 13 & 492    & 6,318   & 369   & 310    & 2,729  & 225   \\ \hline
        61  & make-it-so--single-button-controller                                 & 6  & 454 & 906   & 148 & 369 & 712   & 165 \\ \hline
        62  & lighting-director--turn-it-on-when-it-opens                          & 11 & 423 & 4,642 & 199 & 223 & 1,614 & 135 \\ \hline
        63  & lighting-director--undead-early-warning                              & 11 & 423 & 4,642 & 199 & 223 & 1,614 & 133 \\ \hline
        64  & thermostat-window-check--whole-house-fan                             & 10 & 388 & 3,870 & 178 & 388 & 3,409 & 318 \\ \hline
        65  & good-night--BetterLaundryMonitor                                     & 8  & 385 & 3,072 & 125 & 294 & 1,245 & 116 \\ \hline
        66  & hue-minimote--smart-light-timer-x-minutes-unless-already-on          & 8  & 373 & 2,976 & 183 & 312 & 2,201 & 475 \\ \hline
        67  & gentle-wake-up--BetterLaundryMonitor                                 & 7  & 361 & 2,520 & 114 & 361 & 2,309 & 226 \\ \hline
        68  & make-it-so--thermostat-mode-director                                 & 8  & 361 & 2,880 & 137 & 345 & 2,335 & 251 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        69  & smart-nightlight--turn-on-at-sunset                                  & 10 & 356 & 3,550 & 141 & 333 & 2,861 & 239 \\ \hline
        70  & smart-nightlight--turn-it-on-when-it-opens                           & 10 & 331 & 3,300 & 139 & 101 & 369   & 50  \\ \hline
        71  & smart-nightlight--undead-early-warning                               & 10 & 331 & 3,300 & 140 & 101 & 369   & 50  \\ \hline
        72  & gentle-wake-up--good-night                                           & 8  & 321 & 2,560 & 125 & 321 & 2,376 & 239 \\ \hline
        73  & thermostat--thermostat-window-check                                  & 10 & 313 & 3,120 & 147 & 193 & 1,383 & 159 \\ \hline
        74  & smart-nightlight--vacation-lighting-director                         & 12 & 294 & 3,516 & 133 & 168 & 1,649 & 122 \\ \hline
        75  & lighting-director--turn-on-by-zip-code                               & 10 & 278 & 2,770 & 124 & 115 & 791   & 77  \\ \hline
        76  & smart-nightlight--turn-on-before-sunset                              & 9  & 276 & 2,475 & 92  & 257 & 2,013 & 146 \\ \hline
        77  & good-night--humidity-alert                                           & 8  & 257 & 1,024 & 65  & 243 & 911   & 98  \\ \hline
        78  & thermostat-auto-off--thermostat-window-check                         & 8  & 257 & 2,048 & 86  & 153 & 1,025 & 97  \\ \hline
        79  & double-tap--gentle-wake-up                                           & 7  & 216 & 560   & 58  & 194 & 445   & 67  \\ \hline
        80  & good-night--nfc-tag-toggle                                           & 8  & 215 & 1,712 & 94  & 215 & 1,433 & 149 \\ \hline
        81  & step-notifier--Hue-Party-Mode                                        & 6  & 215 & 1,284 & 82  & 215 & 1,061 & 126 \\ \hline
        82  & step-notifier--turn-on-only-if-i-arrive-after-sunset                 & 6  & 214 & 1,278 & 81  & 104 & 305   & 46  \\ \hline
        83  & Hue-Party-Mode--mini-hue-controller                                  & 4  & 212 & 844   & 42  & 201 & 669   & 86  \\ \hline
        84  & brighten-dark-places--lighting-director                              & 11 & 206 & 2,255 & 111 & 114 & 918   & 89  \\ \hline
        85  & whole-house-fan--WindowOrDoorOpen                                    & 11 & 196 & 2,145 & 120 & 196 & 1,691 & 232 \\ \hline
        86  & smart-nightlight--turn-off-with-motion                               & 9  & 196 & 1,755 & 76  & 87  & 357   & 41  \\ \hline
        87  & smart-nightlight--turn-on-by-zip-code                                & 9  & 196 & 1,755 & 79  & 43  & 131   & 17  \\ \hline
        88  & good-night--the-big-switch                                           & 8  & 191 & 1,520 & 64  & 191 & 1,168 & 103 \\ \hline
        89  & hall-light-welcome-home--step-notifier                               & 6  & 178 & 1,062 & 83  & 40  & 125   & 26  \\ \hline
        90  & smart-light-timer-x-minutes-unless-already-on--turn-on-before-sunset & 7  & 173 & 1,204 & 46  & 163 & 798   & 66  \\ \hline
        91  & smart-light-timer-x-minutes-unless-already-on--turn-on-at-sunset     & 7  & 173 & 1,204 & 50  & 157 & 770   & 68  \\ \hline
        92  & double-tap--good-night                                               & 8  & 161 & 512   & 49  & 149 & 456   & 64  \\ \hline
        93  & make-it-so--thermostat-window-check                                  & 10 & 157 & 1,560 & 80  & 157 & 1,231 & 123 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        94  & make-it-so--BetterLaundryMonitor                                     & 6  & 145 & 864   & 43  & 145 & 735   & 79  \\ \hline
        95  & make-it-so--thermostat                                               & 8  & 145 & 1,152 & 56  & 145 & 905   & 97  \\ \hline
        96  & rise-and-shine--BetterLaundryMonitor                                 & 6  & 136 & 810   & 43  & 103 & 371   & 74  \\ \hline
        97  & nfc-tag-toggle--single-button-controller                             & 4  & 121 & 160   & 51  & 105 & 128   & 66  \\ \hline
        98  & step-notifier--turn-on-at-sunset                                     & 5  & 117 & 580   & 45  & 68  & 177   & 31  \\ \hline
        99  & step-notifier--turn-on-before-sunset                                 & 5  & 117 & 580   & 44  & 68  & 177   & 30  \\ \hline
        100 & energy-saver--good-night                                             & 8  & 113 & 896   & 45  & 68  & 271   & 29  \\ \hline
        101 & big-turn-on--BetterLaundryMonitor                                    & 6  & 112 & 666   & 35  & 112 & 580   & 60  \\ \hline
        102 & the-big-switch--BetterLaundryMonitor                                 & 5  & 109 & 540   & 32  & 109 & 486   & 63  \\ \hline
        103 & gentle-wake-up--rise-and-shine                                       & 6  & 103 & 612   & 37  & 103 & 511   & 62  \\ \hline
        104 & BetterLaundryMonitor--Hue-Party-Mode                                 & 4  & 103 & 408   & 27  & 73  & 219   & 36  \\ \hline
        105 & 01-control-lights-and-locks-with-contact-sensor--good-night          & 8  & 97  & 768   & 69  & 65  & 399   & 49  \\ \hline
        106 & control-switch-with-contact-sensor--good-night                       & 8  & 97  & 768   & 34  & 65  & 399   & 48  \\ \hline
        107 & big-turn-on--good-night                                              & 7  & 95  & 658   & 33  & 95  & 637   & 64  \\ \hline
        108 & forgiving-security--smart-light-timer-x-minutes-unless-already-on    & 8  & 95  & 752   & 40  & 40  & 212   & 42  \\ \hline
        109 & smart-light-timer-x-minutes-unless-already-on--turn-on-by-zip-code   & 6  & 87  & 516   & 24  & 74  & 349   & 35  \\ \hline
        110 & step-notifier--turn-it-on-when-it-opens                              & 5  & 87  & 430   & 37  & 43  & 111   & 25  \\ \hline
        111 & step-notifier--undead-early-warning                                  & 5  & 87  & 430   & 37  & 43  & 111   & 25  \\ \hline
        112 & brighten-my-path--step-notifier                                      & 5  & 87  & 430   & 37  & 42  & 111   & 26  \\ \hline
        113 & thermostat--WindowOrDoorOpen                                         & 6  & 85  & 504   & 37  & 22  & 44    & 14  \\ \hline
        114 & thermostat--hvac-auto-off.smartapp                                   & 6  & 85  & 504   & 32  & 20  & 47    & 14  \\ \hline
        115 & thermostat--thermostat-auto-off                                      & 6  & 85  & 504   & 30  & 20  & 47    & 12  \\ \hline
        116 & BetterLaundryMonitor--WindowOrDoorOpen                               & 4  & 82  & 324   & 28  & 62  & 192   & 37  \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        117 & good-night--make-it-so                                               & 7  & 81  & 560   & 33  & 81  & 448   & 55  \\ \hline
        118 & good-night--rise-and-shine                                           & 6  & 80  & 474   & 25  & 65  & 294   & 47  \\ \hline
        119 & thermostat-window-check--WindowOrDoorOpen                            & 7  & 76  & 525   & 37  & 75  & 455   & 62  \\ \hline
        120 & make-it-so--auto-lock-door.smartapp                                  & 8  & 73  & 576   & 29  & 73  & 509   & 56  \\ \hline
        121 & good-night--once-a-day                                       & 8 & 73 & 576 & 27 & 57 & 402 & 39 \\ \hline
        122 & brighten-dark-places--step-notifier                          & 5 & 71 & 350 & 35 & 26 & 72  & 17 \\ \hline
        123 & gentle-wake-up--monitor-on-sense                             & 6 & 69 & 408 & 27 & 69 & 348 & 44 \\ \hline
        124 & big-turn-off--good-night                                     & 7 & 67 & 462 & 49 & 59 & 290 & 42 \\ \hline
        125 & good-night--monitor-on-sense                                 & 7 & 65 & 448 & 25 & 40 & 132 & 23 \\ \hline
        126 & hue-minimote--mini-hue-controller                            & 2 & 60 & 118 & 15 & 60 & 118 & 32 \\ \hline
        127 & double-tap--nfc-tag-toggle                                   & 4 & 57 & 64  & 32 & 55 & 57  & 35 \\ \hline
        128 & make-it-so--WindowOrDoorOpen                                 & 6 & 55 & 324 & 25 & 54 & 288 & 44 \\ \hline
        129 & good-night--power-allowance                                  & 7 & 49 & 336 & 25 & 49 & 297 & 43 \\ \hline
        130 & nfc-tag-toggle--the-big-switch                               & 5 & 49 & 240 & 21 & 43 & 125 & 37 \\ \hline
        131 & good-night--turn-it-on-for-5-minutes                         & 7 & 47 & 322 & 24 & 41 & 193 & 31 \\ \hline
        132 & step-notifier--turn-on-by-zip-code                           & 4 & 46 & 180 & 21 & 27 & 62  & 17 \\ \hline
        133 & make-it-so--NotifyIfLeftUnlocked                             & 6 & 43 & 252 & 19 & 43 & 232 & 39 \\ \hline
        134 & make-it-so--hvac-auto-off.smartapp                           & 6 & 43 & 252 & 19 & 43 & 234 & 37 \\ \hline
        135 & make-it-so--thermostat-auto-off                              & 6 & 43 & 252 & 18 & 43 & 234 & 35 \\ \hline
        136 & gentle-wake-up--make-it-so                                   & 5 & 41 & 200 & 19 & 41 & 179 & 32 \\ \hline
        137 & big-turn-on--gentle-wake-up                                  & 5 & 41 & 200 & 17 & 33 & 121 & 24 \\ \hline
        138 & good-night--sunrise-sunset                                   & 8 & 41 & 320 & 36 & 33 & 205 & 43 \\ \hline
        139 & good-night-house--lights-off-with-no-motion-and-presence     & 5 & 40 & 195 & 19 & 40 & 178 & 44 \\ \hline
        140 & single-button-controller--NotifyIfLeftUnlocked               & 4 & 40 & 52  & 35 & 31 & 34  & 31 \\ \hline
        141 & big-turn-off--energy-saver                                   & 6 & 39 & 228 & 34 & 39 & 204 & 31 \\ \hline
        142 & good-night-house--single-button-controller                   & 3 & 37 & 36  & 26 & 37 & 36  & 32 \\ \hline
        143 & double-tap--humidity-alert                                   & 4 & 37 & 48  & 20 & 31 & 34  & 27 \\ \hline
        144 & make-it-so--rise-and-shine                                   & 5 & 36 & 175 & 16 & 36 & 149 & 29 \\ \hline
        145 & good-night-house--make-it-so                                 & 4 & 35 & 136 & 13 & 35 & 128 & 30 \\ \hline
        146 & big-turn-on--rise-and-shine                                  & 5 & 31 & 150 & 14 & 31 & 129 & 23 \\ \hline
        147 & hue-minimote--Hue-Party-Mode                                 & 4 & 31 & 120 & 19 & 31 & 93  & 38 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        148 & let-there-be-dark--vacation-lighting-director                & 5 & 31 & 150 & 14 & 31 & 150 & 23 \\ \hline
        149 & door-state-to-color-light-hue-bulb--Hue-Party-Mode           & 4 & 30 & 116 & 15 & 27 & 82  & 21 \\ \hline
        150 & monitor-on-sense--BetterLaundryMonitor                       & 3 & 28 & 81  & 12 & 28 & 72  & 20 \\ \hline
        151 & monitor-on-sense--rise-and-shine                             & 5 & 27 & 130 & 14 & 24 & 85  & 22 \\ \hline
        152 & turn-off-with-motion--vacation-lighting-director             & 5 & 27 & 130 & 13 & 24 & 101 & 19 \\ \hline
        153 & lights-off-with-no-motion-and-presence--turn-off-with-motion & 4 & 27 & 104 & 14 & 23 & 82  & 19 \\ \hline
        154 & good-night--smart-turn-it-on                                 & 6 & 25 & 144 & 13 & 25 & 119 & 18 \\ \hline
        155 & make-it-so--monitor-on-sense                                 & 5 & 25 & 120 & 14 & 25 & 102 & 23 \\ \hline
        156 & make-it-so--unlock-it-when-i-arrive                          & 5 & 25 & 120 & 14 & 25 & 102 & 22 \\ \hline
        157 & nfc-tag-toggle--sunrise-sunset                               & 4 & 25 & 96  & 17 & 25 & 86  & 37 \\ \hline
        158 & good-night-house--vacation-lighting-director                 & 4 & 23 & 88  & 12 & 23 & 88  & 23 \\ \hline
        159 & good-night-house--hue-minimote                               & 3 & 22 & 63  & 16 & 22 & 61  & 29 \\ \hline
        160 & darken-behind-me--lights-off-with-no-motion-and-presence     & 5 & 22 & 105 & 19 & 20 & 81  & 30 \\ \hline
        161 & big-turn-on--monitor-on-sense                                & 5 & 20 & 95  & 13 & 20 & 82  & 24 \\ \hline
        162 & auto-lock-door.smartapp--NotifyIfLeftUnlocked                & 4 & 19 & 72  & 16 & 19 & 68  & 20 \\ \hline
        163 & good-night-house--turn-off-with-motion                       & 3 & 18 & 51  & 10 & 18 & 49  & 21 \\ \hline
        164 & nfc-tag-toggle--NotifyIfLeftUnlocked                         & 4 & 18 & 68  & 15 & 16 & 43  & 26 \\ \hline
        165 & good-night-house--auto-lock-door.smartapp                    & 5 & 17 & 80  & 12 & 15 & 60  & 24 \\ \hline
        166 & big-turn-off--power-allowance                                & 5 & 14 & 65  & 11 & 14 & 60  & 19 \\ \hline
        167 & big-turn-on--make-it-so                                      & 4 & 14 & 52  & 10 & 14 & 52  & 17 \\ \hline
        168 & forgiving-security--turn-on-at-sunset                        & 3 & 14 & 39  & 10 & 14 & 28  & 15 \\ \hline
        169 & forgiving-security--turn-on-before-sunset                    & 3 & 14 & 39  & 9  & 14 & 28  & 11 \\ \hline
        170 & big-turn-off--smart-turn-it-on                               & 4 & 13 & 48  & 8  & 13 & 48  & 12 \\ \hline
        171 & its-too-hot--its-too-cold                                    & 2 & 13 & 12  & 10 & 13 & 12  & 17 \\ \hline
        172 & lock-it-at-a-specific-time--make-it-so                       & 5 & 13 & 60  & 10 & 13 & 54  & 15 \\ \hline
        173 & unlock-it-when-i-arrive--auto-lock-door.smartapp             & 5 & 13 & 60  & 12 & 13 & 56  & 21 \\ \hline
        174 & good-night-house--nfc-tag-toggle                             & 2 & 13 & 24  & 9  & 11 & 16  & 18 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        175 & hall-light-welcome-home--turn-on-at-sunset                   & 3 & 13 & 36  & 10 & 8  & 14  & 8  \\ \hline
        176 & hall-light-welcome-home--turn-on-before-sunset               & 3 & 13 & 36  & 8  & 8  & 14  & 8  \\ \hline
        177 & double-tap--smart-turn-it-on                                 & 2 & 12 & 8   & 11 & 12 & 8   & 10 \\ \hline
        178 & forgiving-security--hall-light-welcome-home                  & 3 & 12 & 33  & 10 & 12 & 24  & 15 \\ \hline
        179 & brighten-my-path--hall-light-welcome-home                    & 3 & 12 & 33  & 9  & 10 & 18  & 9  \\ \hline
        180 & good-night-house--NotifyIfLeftUnlocked                       & 3 & 12 & 33  & 10 & 9  & 20  & 14 \\ \hline
        181 & lock-it-when-i-leave--NotifyIfLeftUnlocked          & 4 & 11 & 40 & 11 & 9  & 26 & 17 \\ \hline
        182 & enhanced-auto-lock-door--NotifyIfLeftUnlocked       & 4 & 10 & 36 & 10 & 10 & 30 & 16 \\ \hline
        183 & forgiving-security--DeviceTamperAlarm               & 4 & 10 & 36 & 10 & 10 & 29 & 14 \\ \hline
        184 & hue-minimote--turn-on-by-zip-code                   & 2 & 10 & 18 & 10 & 10 & 18 & 17 \\ \hline
        185 & darken-behind-me--turn-off-with-motion              & 3 & 9  & 24 & 8  & 9  & 18 & 8  \\ \hline
        186 & enhanced-auto-lock-door--good-night-house           & 5 & 9  & 40 & 13 & 9  & 38 & 28 \\ \hline
        187 & hvac-auto-off.smartapp--WindowOrDoorOpen            & 4 & 9  & 32 & 12 & 9  & 2  & 13 \\ \hline
        188 & lock-it-at-a-specific-time--auto-lock-door.smartapp & 5 & 9  & 40 & 9  & 9  & 36 & 15 \\ \hline
        189 & nfc-tag-toggle--smart-turn-it-on                    & 2 & 9  & 16 & 8  & 9  & 16 & 14 \\ \hline
        190 & thermostat-auto-off--WindowOrDoorOpen               & 4 & 9  & 32 & 10 & 9  & 23 & 13 \\ \hline
        191 & energy-saver--power-allowance                       & 3 & 8  & 21 & 8  & 8  & 21 & 14 \\ \hline
        192 & thermostat-auto-off--hvac-auto-off.smartapp         & 2 & 8  & 14 & 8  & 8  & 14 & 10 \\ \hline
        193 & unlock-it-when-i-arrive--NotifyIfLeftUnlocked       & 3 & 8  & 21 & 9  & 8  & 17 & 13 \\ \hline
        194 & brighten-my-path--turn-on-at-sunset                 & 2 & 7  & 12 & 7  & 7  & 10 & 12 \\ \hline
        195 & brighten-my-path--turn-on-before-sunset             & 2 & 7  & 12 & 7  & 7  & 10 & 8  \\ \hline
        196 & darken-behind-me--good-night-house                  & 2 & 7  & 12 & 8  & 7  & 11 & 11 \\ \hline
        197 & energy-saver--smart-turn-it-on                      & 2 & 7  & 12 & 7  & 7  & 12 & 8  \\ \hline
        198 & turn-on-at-sunset--turn-it-on-when-it-opens         & 2 & 7  & 12 & 7  & 7  & 9  & 11 \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        199 & turn-on-at-sunset--turn-on-before-sunset            & 2 & 7  & 12 & 8  & 7  & 12 & 10 \\ \hline
        200 & turn-on-at-sunset--undead-early-warning             & 2 & 7  & 12 & 7  & 7  & 9  & 11 \\ \hline
        201 & turn-on-before-sunset--turn-it-on-when-it-opens     & 2 & 7  & 12 & 7  & 7  & 9  & 8  \\ \hline
        202 & turn-on-before-sunset--undead-early-warning         & 2 & 7  & 12 & 7  & 7  & 9  & 8  \\ \hline
        203 & brighten-dark-places--turn-on-at-sunset             & 2 & 7  & 12 & 7  & 6  & 8  & 8  \\ \hline
        204 & brighten-dark-places--turn-on-before-sunset         & 2 & 7  & 12 & 7  & 6  & 8  & 8  \\ \hline
        205 & brighten-dark-places--hall-light-welcome-home       & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        206 & hall-light-welcome-home--turn-it-on-when-it-opens   & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        207 & hall-light-welcome-home--turn-on-by-zip-code        & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        208 & hall-light-welcome-home--undead-early-warning       & 2 & 7  & 12 & 7  & 5  & 6  & 7  \\ \hline
        209 & brighten-dark-places--forgiving-security            & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        210 & brighten-my-path--forgiving-security                & 2 & 6  & 10 & 8  & 6  & 8  & 12 \\ \hline
        211 & brighten-my-path--turn-it-on-when-it-opens          & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        212 & brighten-my-path--undead-early-warning              & 2 & 6  & 10 & 7  & 6  & 8  & 10 \\ \hline
        213 & forgiving-security--turn-it-on-when-it-opens        & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        214 & forgiving-security--turn-on-by-zip-code             & 2 & 6  & 10 & 7  & 6  & 8  & 8  \\ \hline
        215 & forgiving-security--undead-early-warning            & 2 & 6  & 10 & 7  & 6  & 8  & 11 \\ \hline
        216 & lock-it-at-a-specific-time--NotifyIfLeftUnlocked    & 3 & 6  & 15 & 8  & 6  & 14 & 11 \\ \hline
        217 & brighten-dark-places--brighten-my-path              & 2 & 6  & 10 & 8  & 5  & 6  & 9  \\ \hline
        218 & enhanced-auto-lock-door--auto-lock-door.smartapp    & 4 & 5  & 16 & 9  & 5  & 15 & 18 \\ \hline
        219 & good-night-house--lock-it-at-a-specific-time        & 2 & 5  & 8  & 7  & 5  & 6  & 8  \\ \hline
        220 & good-night-house--turn-on-by-zip-code               & 2 & 5  & 8  & 7  & 5  & 6  & 8  \\ \hline
        221 & turn-on-at-sunset--turn-on-by-zip-code              & 2 & 5  & 8  & 6  & 5  & 6  & 7  \\ \hline
  \end{tabular}
  }

\end{table*}

\begin{table*}[!htb]
  \centering
  { \footnotesize
  \begin{tabular}{| r | p{42mm} | r | r | r | r | r | r | r |}
    \hline
    \textbf{No.} & \textbf{App} & \textbf{Evt.} & \multicolumn{3}{ c |}{\textbf{Without DPOR}} & \multicolumn{3}{ c |}{\textbf{With DPOR}}\\
    \cline{4-9}
        & & & \textbf{States} & \textbf{Trans.} & \textbf{Time} & \textbf{States} & \textbf{Trans.} & \textbf{Time}\\

    \hline
        222 & turn-on-before-sunset--turn-on-by-zip-code          & 2 & 5  & 8  & 6  & 5  & 6  & 7  \\ \hline
        223 & brighten-dark-places--turn-it-on-when-it-opens      & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        224 & brighten-dark-places--turn-on-by-zip-code           & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        225 & brighten-dark-places--undead-early-warning          & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        226 & brighten-my-path--turn-on-by-zip-code               & 2 & 4  & 6  & 6  & 4  & 5  & 6  \\ \hline
        227 & turn-it-on-when-it-opens--undead-early-warning      & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        228 & turn-on-by-zip-code--turn-it-on-when-it-opens       & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
        229 & turn-on-by-zip-code--undead-early-warning           & 2 & 4  & 6  & 6  & 4  & 5  & 7  \\ \hline
  \end{tabular}
  }

\end{table*}

\subsection{Event-Driven Concurrency Model}
\rahmadi{
We next present our event-driven concurrency model (see an example of event-driven
\invmcaiversion{systems in \appdx{Appendix A}).}
\inarxivversion{systems in \appdx{Appendix~\ref{sec:example-event-driven-system}}).}
}
We assume that
the event-driven system has a finite set $\Events{}$ of different event types.  Each event type $\Event{} \in \Events{}$ has a
corresponding event handler that is executed when an instance of the
event occurs.  We assume that there is a potentially shared state and
that event handlers have arbitrary access to read and write from this
shared state.

An event handler can be an arbitrarily long finite sequence of
instructions and can include an arbitrary number of accesses to shared
state.  We assume event-handlers are executed atomically by the
event-driven runtime system.
Events can be enabled by both external sources (\hbox{\emph{e.g.},}\xspace events in the
physical world) or event handlers.  Events can also be disabled by
the execution of an event handler.
We assume that the runtime system maintains an unordered set of
enabled events to execute.  It contains an event dispatch
loop that selects an arbitrary enabled event to execute next.

This work is inspired by smart-home systems that are widely deployed in the real world. However, the proposed techniques are general enough to handle other types of event-driven systems,
such as web applications, as long as the systems follow the concurrency model stated above.
\vspace{-.5em}

\subsection{Background on Stateless DPOR}

Partial order reduction is based on the observation that traces of concurrent systems are equivalent if they only reorder independent operations.  These equivalence classes are called Mazurkiewicz traces~\cite{tracetheory}.  The classical DPOR algorithm~\cite{flanagan2005dynamic} dynamically computes persistent sets for multithreaded programs and is guaranteed to explore at least one interleaving in each equivalence class.

The key idea behind the DPOR algorithm is to compute the next pending memory operation for each thread, and at each point in the execution to compute the most recent conflict for each thread's next operation.  These conflicts are used to set backtracking points so that future executions will reverse the order of conflicting operations and explore an execution in a different equivalence class. Due to space constraints, we refer the interested readers to \cite{flanagan2005dynamic} for a detailed description of the original DPOR algorithm.

\section{Introduction\label{sec:intro}}

Event-driven architectures are broadly used to build systems that react to events in the real world.  They include smart home systems, GUIs, mobile applications, and servers.
For example, in the context of smart home systems, event-driven systems include Samsung SmartThings~\cite{smartthings}, Android Things~\cite{androidthings}, Openhab~\cite{openhab}, and If This Then That (IFTTT)~\cite{ifttt}.

Event-driven architectures can have analogs of the concurrency bugs that are known to be problematic in multithreaded programming.  Subtle programming errors involving the ordering of events can easily cause event-driven programs to fail.  These failures can be challenging to find during testing as exposing these failures may require a specific set of events to occur in a specific order.
Model-checking tools can be helpful for finding subtle concurrency bugs or understanding complex interactions between different applications~\cite{trimanafse20}.  In recent years, significant work has been expended on developing model checkers for multithreaded concurrency~\cite{odpor,statelessmcr,jensen2015stateless,dporlocks,pldipsotso,inspect1,inspect2,inspect4}, but event-driven systems have received much less attention~\cite{jensen2015stateless,maiya2016por}.

Event-driven systems pose several challenges for existing \emph{stateless} and \emph{stateful} model-checking tools.  Stateless model checking of concurrent applications explores all execution schedules without checking whether these schedules visit the same program states.  Stateless model checking often uses dynamic partial order reduction (DPOR) to eliminate equivalent schedules. \mycomment{,\hbox{\emph{i.e.},}\xspace schedules that rearrange independent events}
While there has been much work on DPOR for stateless model checking of multithreaded programs~\cite{flanagan2005dynamic,odpor,dporlocks,pldipsotso,statelessmcr}, stateless model checking requires that the program under test terminates for fair schedules.  Event-driven systems are often intended to run continuously and may not terminate.  To handle non-termination, stateless model checkers require hacks such as bounding the length of executions to verify event-driven systems.

Stateful model checking keeps track of an application's states and avoids revisiting the same application states.  It is less common for stateful model checkers to use dynamic partial order reduction to eliminate equivalent executions.
Researchers have done much work on stateful model checking~\cite{jpf,spin,musuvathi2002cmc,gueta2007cartesian}.  While stateful model checking can handle non-terminating programs, they  miss an opportunity to efficiently reason about conflicting transitions to scale to large programs.  In particular, typical event-driven programs such as smart home applications have several event handlers that are completely independent of each other.  Stateful model checking enumerates different orderings of these event handlers, overlooking the fact that these handlers are independent of each other and hence the orderings are equivalent.

Stateful model checking and dynamic partial order reduction discover
different types of redundancy, and therefore it is beneficial to combine them to further improve model-checking scalability and efficiency.  For example, we have observed that some smart home systems have
several independent event handlers \mycomment{(\hbox{\emph{e.g.},}\xspace monitoring applications or
just independent systems)}in our experiments, and stateful model checkers can waste
an enormous amount of time exploring different orderings of these
independent transitions.  DPOR can substantially reduce the number of
states and transitions that must be explored.  Although work has been done to
combine DPOR algorithms with stateful model
checking~\cite{yang2008efficient,yi2006stateful} in the context of
multithreaded programs, this line of work requires that the application has
an \emph{acyclic state space}, \hbox{\emph{i.e.},}\xspace it terminates under all schedules.  In particular, the
approach of Yang \hbox{\emph{et al.}}\xspace\cite{yang2008efficient} is designed explicitly for
programs with acyclic state space and thus
cannot check programs that do not terminate.
Yi \hbox{\emph{et al.}}\xspace\cite{yi2006stateful} presents a DPOR algorithm for stateful model
checking, which is, however, incorrect for cyclic state spaces. For
instance, their algorithm fails to produce the asserting execution in
the example we will discuss shortly in
Figure~\ref{fig:example-non-terminating}.  As a result, prior DPOR
techniques all fall short for checking event-driven programs such as
smart home apps, that, in general, do not terminate.

\myparagraph{Our Contributions}
In this work, we present a stateful model checking technique for
event-driven programs that may not terminate. Such programs have cyclic state spaces, and existing algorithms can prematurely terminate an execution and thus fail to set
the necessary backtracking points to fully explore a program's
state space. Our \textbf{first} technical contribution is the \emph{formulation of a
sufficient condition to complete an execution of the application
that ensures that our algorithm fully explores the application's
state space}.

In addition to the early termination issue, for
programs with cyclic state spaces, a model checker can discover
multiple paths to a state \State{} before it explores the entire state
space that is reachable from state \State{}.  In this case, the
backtracking algorithms used by traditional DPOR techniques including
Yang \hbox{\emph{et al.}}\xspace\cite{yang2008efficient} can fail to set the necessary
backtracking points.  Our \textbf{second} technical contribution is \emph{a graph-traversal-based algorithm to appropriately set backtracking points on all paths that can reach the current state}.

Prior work on stateful DPOR only considers the multithreaded case
and assumes algorithms know the effects of the next transitions
of all threads before setting backtracking points.   For multithreaded programs, this assumption is \emph{not} a serious limitation as transitions model low-level memory operations (\hbox{\emph{i.e.},}\xspace reads, writes, and RMW operations), and each transition involves a \emph{single} memory operation.
However, in the context of event-driven programs,  events can involve
many memory operations that access multiple memory locations,
and knowing the effects of a transition requires actually
executing the event.  While it is conceptually possible to execute events and then rollback to discover their effects, this approach is likely to incur large overheads as model checkers need to know the effects of enabled events at each program state. As our \textbf{third} contribution, \emph{our algorithm avoids this extra rollback overhead by waiting until an event is actually
executed to set backtracking points and incorporates a modified backtracking
algorithm to appropriately handle events}.

\rahmadi{We have implemented the proposed algorithm in the Java Pathfinder model checker~\cite{jpf} and evaluated it on hundreds of real-world smart home apps. We have made our DPOR implementation publicly available~\cite{iotcheck-dpor-software}.}

\myparagraph{Paper Structure}
The remainder of this paper is structured as follows:
Section~\ref{sec:concurrency} presents the event-driven concurrency model that we use in this work.
Section~\ref{sec:definition} presents the definitions we use to describe our stateful DPOR algorithm.
Section~\ref{sec:basicideas} presents problems when using the classic DPOR algorithm to model check event-driven programs and the basic ideas behind how our algorithm solves these problems.
Section~\ref{sec:algorithm} presents our stateful DPOR algorithm for event-driven programs.
Section~\ref{sec:impeval} presents the evaluation of our algorithm
implementation on hundreds of smart home apps.
Section~\ref{sec:related} presents the related work; we conclude in
Section~\ref{sec:conclusion}.

\section{#1}\vspace{-.27em}}
\newcommand{\mysubsection}[1]{\vspace{-.27em}\subsection{#1}\vspace{-.27em}}
\newcommand{\mysubsubsection}[1]{\subsubsection{#1}}
\newcommand{\myparagraph}[1]{\noindent\textit{\textbf{{#1}.}}}
\newcommand{\mysubparagraph}[1]{\subparagraph{#1}}
\newcommand{\mycomment}[1]{}
\newcommand{\tuple}[1]{\ensuremath \langle #1 \rangle}
\newcommand{\Naive}[0]{Na\"{i}ve\xspace}
\newcommand{\naively}[0]{na\"{i}vely\xspace}
\newcommand{\Naively}[0]{Na\"{i}vely\xspace}
\newcommand{\naive}[0]{na\"{i}ve\xspace}
\newcommand{\interacts}[0]{\emph{interacts-with}\xspace}
\newcommand{\conflict}[0]{\emph{conflict}\xspace}
\newcommand{\Interacts}[0]{\emph{Interacts-with}\xspace}
\newcommand{\Conflict}[0]{\emph{Conflict}\xspace}
\newcommand{\Devicestate}[0]{\emph{Device}\xspace}
\newcommand{\Physical}[0]{\emph{Physical-Medium}\xspace}
\newcommand{\Globalstate}[0]{\emph{Global-Variable}\xspace}
\newcommand{\devicestate}[0]{\emph{device}\xspace}
\newcommand{\globalstate}[0]{\emph{global-variable}\xspace}
\newcommand{\physical}[0]{\emph{physical-medium}\xspace}
\newcommand{\conflictinreader}[0]{\emph{conflict in reader}\xspace}
\newcommand{\readop}[0]{\emph{read-from}\xspace}
\newcommand{\writeop}[0]{\emph{write-to}\xspace}
\newcommand{\readsop}[0]{\emph{reads-from}\xspace}
\newcommand{\writesop}[0]{\emph{writes-to}\xspace}
\newcommand{\doorlock}[0]{door-lock\xspace}
\newcommand{\doorunlock}[0]{door-unlock\xspace}
\newcommand{\doorlocks}[0]{door-locks\xspace}
\newcommand{\doorunlocks}[0]{door-unlocks\xspace}
\newcommand{\locationmode}[0]{\code{location.mode}\xspace}
\newcommand{\locationspacemode}[0]{\code{location.} \code{mode}\xspace}

\newcommand{\tool}[0]{\textsc{IoTCheck}\xspace}
\newcommand{\initialstateeventstreamer}[0]{\code{Initial}-\code{State}-\code{Event}-\code{Streamer}\xspace}
\newcommand{\bigturnon}[0]{\code{Big}-\code{Turn}-\code{ON}\xspace}
\newcommand{\firecoalarm}[0]{\code{FireCO2Alarm}\xspace}
\newcommand{\devicetamperalarm}[0]{\code{Device}-\code{Tamper}-\code{Alarm}\xspace}
\newcommand{\windowordooropen}[0]{\code{WindowOrDoorOpen}\xspace}
\newcommand{\lockitwhenileave}[0]{\code{Lock}-\code{It}-\code{When}-\code{I}-\code{Leave}\xspace}
\newcommand{\lockitataspecifictime}[0]{\code{Lock}-\code{It}-\code{at}-\code{a}-\code{Specific}-\code{Time}\xspace}
\newcommand{\autolockdoor}[0]{\code{Auto}-\code{Lock}-\code{Door}\xspace}
\newcommand{\thermostats}[0]{\code{Thermostats}\xspace}
\newcommand{\thermostatautooff}[0]{\code{Thermostat}-\code{Auto}-\code{Off}\xspace}
\newcommand{\neato}[0]{\code{Neato}-\code{(Connect)}\xspace}
\newcommand{\forgivingsecurity}[0]{\code{Forgiving}-\code{Security}\xspace}
\newcommand{\turnonatsunset}[0]{\code{Turn}-\code{On}-\code{at}-\code{Sunset}\xspace}
\newcommand{\lightupthenight}[0]{\code{Light}-\code{Up}-\code{the}-\code{Night}\xspace}
\newcommand{\bosesound}[0]{\code{Bose}-\code{SoundTouch}-\code{Control}\xspace}
\newcommand{\sprayercontroller}[0]{\code{Sprayer}-\code{Controller}-\code{2}\xspace}
\newcommand{\closethevalve}[0]{\code{Close}-\code{The}-\code{Valve}\xspace}
\newcommand{\influxdblogger}[0]{\code{InfluxDB}-\code{Logger}\xspace}
\newcommand{\buttoncontroller}[0]{\code{Button}-\code{Controller}\xspace}
\newcommand{\smartsecurity}[0]{\code{Smart}-\code{Security}\xspace}
\newcommand{\vacationlightingdirector}[0]{\code{Vacation}-\code{Lighting}-\code{Director}\xspace}
\newcommand{\lightingdirector}[0]{\code{Lighting}-\code{Director}\xspace}
\newcommand{\bigturnoff}[0]{\code{Big}-\code{Turn}-\code{OFF}\xspace}
\newcommand{\monitoronsense}[0]{\code{Monitor}-\code{on}-\code{Sense}\xspace}
\newcommand{\huemoodlighting}[0]{\code{Hue}-\code{Mood}-\code{Lighting}\xspace}
\newcommand{\doorstatetocolorlight}[0]{\code{Door}-\code{State}-\code{to}-\code{Color}-\code{Light}\xspace}
\newcommand{\greetingsearthling}[0]{\code{Greetings}-\code{Earthling}\xspace}
\newcommand{\hellohomephrasedirector}[0]{\code{Hello,}-\code{Home}-\code{Phrase}-\code{Director}\xspace}
\newcommand{\bonvoyage}[0]{\code{Bon}-\code{Voyage}\xspace}
\newcommand{\goodnight}[0]{\code{Good}-\code{Night}\xspace}
\newcommand{\wholehousefan}[0]{\code{Whole}-\code{House}-\code{Fan}\xspace}
\newcommand{\autohumidityvent}[0]{\code{Auto}-\code{Humidity}-\code{Vent}\xspace}
\newcommand{\keepmecozy}[0]{\code{Keep}-\code{Me}-\code{Cozy}\xspace}

\newcommand{\harry}[1] {{\textcolor{blue}{Harry: {#1}}}}
\newcommand{\rahmadi}[1]{{#1}}
\newcommand{\weiyu}[1]{{\color{purple}#1}}
\newcommand{\camready}[1]{#1}

\newcounter{isvmcaiversion}
\setcounter{isvmcaiversion}{0}

\ifthenelse{\value{isvmcaiversion}>0}
{
    \SetWatermarkText{\hspace*{6in}\raisebox{8.8in}{\includegraphics[scale=0.15]{vmcai-badges/aec-badge-vmcai}}}
    \SetWatermarkAngle{0}
    \pagenumbering{gobble}
    \newcommand{\appdx}[1]{\text{\color{blue} #1 in~\todo{\cite{ArXiv}}}}
    \newcommand{\invmcaiversion}[1]{#1}
    \newcommand{\inarxivversion}[1]{}
}
{
    \SetWatermarkText{}
    \SetWatermarkAngle{0}

    \newcommand{\appdx}[1]{#1}
    \newcommand{\invmcaiversion}[1]{}
    \newcommand{\inarxivversion}[1]{#1}
}

\maketitle

\thispagestyle{plain}
\pagestyle{plain}

\input{abstract}
\input{intro}
\input{concurrency}
\input{definitions}
\input{basicideas}
\input{algorithm}
\input{evaluation-new}
\input{related-work}
\input{conclusion}
\input{acknowledgment}

\bibliographystyle{splncs04}

\section{Correctness\label{sec:proof}}

This section proves the correctness of our stateful DPOR algorithm with the following proof strategy: we first prove that the
stateless analog of our dynamic partial order reduction algorithm is correct in
Theorem~\ref{thm:stateless}.  We then prove that the stateful
algorithm explores all transitions that the stateless algorithm
explores.

Given a transition sequence $\Trace{}$, we use the following notations in our proof:
\begin{itemize}
 \item $\lvert \Trace{} \rvert$ returns the length of $\Trace{}$.
 \item $\Trace{i}$ denotes the $i$-th transition in $\Trace{}$, and $\Trace{i...j}$ denotes the subsequence of $\Trace{}$ from the $i$-th position to the $j$-th position.
\end{itemize}

We also define the notion of transitive dependence as follows:
\begin{definition}[Transitive Dependence]
Given a transition sequence $\tau$:
$\State{0} \xrightarrow{\Trans{1}} \State{1} \xrightarrow{\Trans{2}} \State{2} ... \xrightarrow{\Trans{n-1}} \State{n-1} \xrightarrow{\Trans{n}} \State{n}$,
the transitive dependence relation $\rightarrow_{\tau}$ is the smallest partial
order on $\tau$ such that if $i < j$ and $\Trans{i}$ is dependent on $\Trans{j}$, then
$\Trans{i} \rightarrow_{\tau} \Trans{j}$.
\end{definition}

The notion of transitive dependence enables partial order reduction.
The transition sequence $\tau$ is one linearization of the partial order
$\rightarrow_{\tau}$.  A search algorithm only needs to explore one linearization,
because other linearizations of $\rightarrow_{\tau}$ yield
``equivalent'' transition sequences of $\tau$, which can be obtained
by swapping adjacent independent transitions.

\mysubsection{Correctness of the Stateless Algorithm}
For the stateless analog of the algorithm, the \code{if} conditions
in lines~\ref{line:termination-condition} and
\ref{line:cycle-current-exec-1} of Algorithm~\ref{alg:sdpor} are always false, and the data structures
\Trace{} and \hbox{$\mathcal{H}$}\xspace in Algorithm~\ref{alg:sdpormain} and~\ref{alg:sdpor} are \emph{don't-care} terms.
When all events in $\hbox{\textit{backtrack}}\xspace(\State{})$ have been explored, the search from $\State{}$ is over, and we say that the state $\State{}$ is ``backtracked''.

\begin{theorem}
\label{thm:stateless}
Let $\Events{}$ be a finite set of events, $\tau$ be a transition sequence
from the initial state $\State{0}$ explored by the stateless analog of
the algorithm in an acyclic transition
system, and $\State{0} \xrightarrow{\tau} \State{}$.  Then, when $\State{}$ is backtracked,
\begin{enumerate}
  \item the set of events that have been explored from $\State{}$ is a persistent set in $\State{}$, and
  \item every trace $\tau \cdot \lambda$ in the state space of $A_G$ is a prefix
    of a linearization of $\rightarrow_{\tau \cdot \lambda'}$ for
    some explored trace $\tau \cdot \lambda'$.
\end{enumerate}
\end{theorem}

\begin{proof}
Acyclicity implies that the system
satisfies the descending chain condition, and we can apply well-founded induction.
We need to prove the statements for the trace
$\State{0} \xrightarrow{\tau} \State{}$,
assuming that the statements hold for all longer traces
$\State{0} \xrightarrow{\tau} \State{} \xrightarrow{\tau'} \State{}'$,
where $\State{}'$ is reachable from $\State{}$ in finite steps.

Let $T := \hbox{\textit{backtrack}}\xspace(\State{})$.  For the first statement, \hbox{\emph{i.e.},}\xspace
$T$ is a persistent set when $\State{}$ is backtracked, we will prove it by contradiction.
If $T$ is not a persistent set, then there must exist a transition sequence from $\State{}$,
\[ \State{} \xrightarrow{\Trans{1}} \State{1} \xrightarrow{\Trans{2}} \State{2} ... \xrightarrow{\Trans{n-1}} \State{n-1} \xrightarrow{\Trans{n}} \State{n}, \]
where $\hbox{\emph{event}}\xspace{(\Trans{1})}, ..., \hbox{\emph{event}}\xspace{(\Trans{n})} \notin T$,
events $\hbox{\emph{event}}\xspace{(\Trans{1})}, ..., \hbox{\emph{event}}\xspace{(\Trans{n-1})}$ are read-write
independent with all events in $T$,
and $\hbox{\emph{event}}\xspace{(\Trans{n})}$ is dependent with some $x \in T$ at state $\State{n-1}$.
Select the shortest such transition sequence.
Let $\sigma$ be the transition sequence $\Trans{1} ... \Trans{n-1}$.
For simplicity, we will label $\hbox{\emph{event}}\xspace(\Trans{i})$ as $\tildeEvent{i}$ for $1 \leq i \leq n$.
Note that $\tildeEvent{i}$ and $\tildeEvent{j}$ could be the same event even if $i \neq j$.

Since $x$ is read-write independent with $\tildeEvent{i}$ for all
$1 \leq i \leq n-1$, we have the following transition diagram:
\[
\xymatrix{ \State{}' \ar[r]^{\sigma} & \State{n-1}' \\
           \State{} \ar[r]^{\sigma} \ar[u]^{x} & \State{n-1} \ar[u]^{x} }
\]

There are two cases: (1) $\tildeEvent{n}$ is enabled in $\State{}$;
and (2) $\tildeEvent{n}$ is disabled in $\State{}$.

\vspace{.3em}
\textbf{Case 1:} Suppose that $\tildeEvent{n}$ is enabled
in $\State{}$ but disabled in $\State{}'$.
Since $x \in T$ and $\State{} \xrightarrow{x} \State{}'$ has been explored
when $\State{}$ is backtracked, the \code{for} loop at line~\ref{line:add-disabled}
of Algorithm~\ref{alg:sdpor} will add $\tildeEvent{n}$ to $T$, contradicting the
assumption that $\tildeEvent{n} \notin T$.

Suppose that $\tildeEvent{n}$ is enabled in $\State{}$ and $\State{}'$.
Then, $x$ does not enable or disable $\tildeEvent{n}$.  Since $x$
is read-write independent with all corresponding events in $\sigma$,
$\tildeEvent{n}$ is also enabled in $\State{n-1}'$.
Recall that $\tildeEvent{n}$ is dependent with $x$ at $\State{n-1}$.
Since $x$ does not enable or disable $\tildeEvent{n}$,
$x$ and $\tildeEvent{n}$ must have
conflicting memory accesses $\Access{x, n}$ in the transition
sequence $\tau \cdot \sigma \cdot \Trans{x} \cdot \Trans{n}$, which is equivalent to
the transition sequence $\tau \cdot \Trans{x} \cdot \sigma \cdot \Trans{n}$,
where $\hbox{\emph{event}}\xspace({\Trans{x}}) = x$.
Since $x \in T$ and by the inductive assumption,
we have explored some transition sequence $\tilde{\tau}$ where
some linearization of $\rightarrow_{\tilde{\tau}}$ has prefix
$\tau \cdot \Trans{x} \cdot \sigma \cdot \Trans{n}$.
Thus, line~\ref{line:call-update-backtrack-set} in Algorithm~\ref{alg:sdpor}
would have called $\textsc{UpdateBacktrackSet}\xspace{}$ and found the conflicting memory
accesses $\Access{x, n}$, which is not empty until some
$\tildeEvent{i}$ is added to $T$.  Thus, we have a contradiction.

\vspace{.3em}
\textbf{Case 2:} Suppose that $\tildeEvent{n}$ is disabled in $\State{}$.
Since $\tildeEvent{n}$ is enabled in $\State{n-1}$,
there exists some transition $\Trans{i}$ in $\sigma$ that writes to the
shadow location of $\tildeEvent{n}$.  Since $x$ is
read-write independent with $\tildeEvent{1}, ...,
\tildeEvent{n-1}$ by assumption,
$x$ does not write to the shadow location of $\tildeEvent{n}$,
\hbox{\emph{i.e.},}\xspace $x$ does not enable or disable $\tildeEvent{n}$.
Then, $\tildeEvent{n}$ is also enabled in $\State{n-1}'$.
Therefore, the same argument in the second paragraph of Case 1 applies,
and we have a contradiction. \\

We have shown that $T$ is a persistent set.
We now move to prove the second statement in this theorem, that every trace
$\State{0} \xrightarrow{\tau} \State{} \xrightarrow{\lambda} u$ is a prefix of a linearization
of $\rightarrow_{\tau \cdot \lambda'}$ for some explored trace $\tau \cdot \lambda'$.
If $\lambda$ is the null sequence, then we are done,
because $\State{0} \xrightarrow{\tau} \State{}$
is already explored.

Let $\lambda = \pi \cdot \eta$, where $\pi$ is the maximal transition
sequence such that no events executed in $\pi$ is in $T$.
If $\eta$ is not the null sequence, then the first transition in $\eta$,
denoted by $\eta_1$, is in $T$.
Since $T$ is a persistent set, $\eta_1$ is independent with all transitions in
$\pi$.  Thus, $\tau \cdot \pi \cdot \eta$ is equivalent to
$\tau \cdot \eta_1 \cdot \pi \cdot \eta_{2...\lvert \eta \rvert}$.
Note that $\tau \cdot \eta_1$ is a transition sequence
that is explored by the algorithm.  Thus, by the inductive assumption,
$\tau \cdot \eta_1 \cdot \pi \cdot \eta_{2...\lvert \eta \rvert}$
is a prefix of a linearization of $\rightarrow_{\tau \cdot \eta_1 \cdot \lambda'}$
for some explored trace $\tau \cdot \eta_1 \cdot \lambda'$.
Then, $\tau \cdot \lambda = \tau \cdot \pi \cdot \eta$ is a prefix of
another linearization of $\rightarrow_{\tau \cdot \eta_1 \cdot \lambda'}$.
The same argument holds if $\pi$ is the null sequence.

Let $\eta$ be the null sequence.  Pick $y \in T$.
Then, $\tau \cdot \Trans{y}$ is an explored transition sequence,
where $\hbox{\emph{event}}\xspace(\Trans{y}) = y$.
The inductive assumption implies that $\tau \cdot \Trans{y} \cdot \pi$
is a prefix of a linearization of $\rightarrow_{\tau \cdot t_y \cdot \lambda'}$
for some explored trace $\tau \cdot \Trans{y} \cdot \lambda'$.
Because $\Trans{y}$ is independent with all transitions in $\pi$, $\tau \cdot \pi \cdot \Trans{y}$
is also a prefix of some linearization of $\rightarrow_{\tau \cdot t_y \cdot \lambda'}$.
Hence, $\tau \cdot \pi$ is a prefix of some linearization of
$\rightarrow_{\tau \cdot t_y \cdot \lambda'}$ as well.
\end{proof}

\section{Related Work\label{sec:related}}

There has been much work on model checking.  Stateless model checking
techniques do not explicitly track which program states have been
visited and instead focus on enumerating
schedules~\cite{godefroidbook,godefroid1997model,godefroid2005software,musuvathi2007chess}.

To make model checking more efficient, researchers have
also looked into various partial order reduction techniques.
The original partial order reduction techniques (\hbox{\emph{e.g.},}\xspace persistent/stubborn sets~\cite{godefroidbook,stubbornsets} and sleep sets~\cite{godefroidbook}) can also be used in the context of cyclic state spaces when combined with a proviso that ensures that executions are not prematurely terminated~\cite{godefroidbook}, and ample sets~\cite{clarke1999model,clarke1999state} that are basically persistent sets with additional conditions. However, the persistent/stubborn set techniques ``suffer from severe fundamental limitation''\cite{flanagan2005dynamic}: the operations and their communication objects in future process executions are difficult or impossible to compute precisely through static analysis, while sleep sets alone only reduce the number of transitions (not states).  Work on \emph{collapses} by Katz and Peled also suffers from the same requirement for a statically known independence relation~\cite{collapses}.

The first DPOR technique was proposed by Flanagan and Godefroid~\cite{flanagan2005dynamic} to address those issues.
The authors introduced a
technique that combats the state space explosion by detecting
\emph{read-write} and \emph{write-write} conflicts on shared variable
on the fly.
Since then, a significant effort has been made to further improve
dynamic partial order reduction~\cite{sen2006automated,sen2006race,lauterburg2010evaluating,saarikivi2012improving,tasharofi2012transdpor}.
Unfortunately, a lot of DPOR algorithms assume the context of shared-memory concurrency in that each transition consists of a single memory operation.
In the context of event-driven applications, each transition is an event that can consist of different memory operations. Thus, we have to execute the event to know its effects and analyze it dynamically on the fly in our DPOR algorithm
(see Section~\ref{sec:events-as-transitions}).

\mycomment{
There has been much work on model checking.  Stateless model checking
techniques do not explicitly track which program states have been
visited and instead focus on enumerating
schedules~\cite{godefroidbook,godefroid1997model,godefroid2005software,musuvathi2007chess}.
To make model checking more efficient, researchers have
also looked into different DPOR techniques.
The first DPOR technique was proposed by Flanagan and Godefroid~\cite{flanagan2005dynamic}.
The authors introduced a
technique that combats the state space explosion by detecting
\emph{read-write} and \emph{write-write} conflicts on shared variable
on the fly.
This technique is based on the notion of independence between commutative transitions that borrows a similar concept from \emph{collapses} introduced in~\cite{katz1992defining}.
Since then, a significant effort has been made to further improve
dynamic partial order reduction~\cite{sen2006automated,sen2006race,lauterburg2010evaluating,saarikivi2012improving,tasharofi2012transdpor}.
}

Optimal DPOR~\cite{odpor} seeks to
make stateless model checking more efficient by skipping equivalent
executions.  Maximal causality reduction~\cite{statelessmcr} further
refines the technique with the insight that it is only necessary to
explore executions in which threads read different values.
Value-centric DPOR~\cite{chatterjee2019valuecentricdpor} has the insight that executions are equivalent if all of their loads read from the same writes.  Unfolding~\cite{unfoldings} is an alternative approach to POR for reducing the number of executions to be explored.  The unfolding algorithm involves solving an NP-complete problem to add events to the unfolding.

Recent work has extended these algorithms to handle the TSO and PSO
memory models~\cite{nidhugg,pldipsotso,mcrpsotso} and the release
acquire fragment of C/C++~\cite{statelessrelacq}.  The RCMC tool
implements a DPOR tool that operates on execution graphs for the RC11
memory model~\cite{rcmc}.  SAT solvers have been used to avoid
explicitly enumerating all executions.  SATCheck extends
partial order reduction with the insight that it is only necessary to
explore executions that exhibit new behaviors~\cite{satcheck}.
CheckFence checks code by translating it into
SAT~\cite{checkfence}.
\rahmadi{Other work has also presented techniques orthogonal to DPOR, either in a more general context~\cite{desai2015systematic} or platform specific (\hbox{\emph{e.g.},}\xspace Android~\cite{ozkan2015systematic} and Node.js~\cite{loring2017semantics}).
}

Recent work on dynamic partial order reduction for event-driven programs has
developed dynamic partial order reduction algorithms for stateless
model checking of event-driven
applications~\cite{jensen2015stateless,maiya2016por}.
Jensen \hbox{\emph{et al.}}\xspace\cite{jensen2015stateless} consider a model similar to ours in which
an event is treated as a single transition, while Maiya \hbox{\emph{et al.}}\xspace\cite{maiya2016por}
consider a model in which event execution interleaves concurrently
with threads.  Neither of these approaches handle cyclic state spaces
nor consider challenges that arise from stateful model checking.

Recent work on DPOR algorithms reduces the number of executions for programs with critical sections by considering
whether critical sections contain conflicting
operations~\cite{dporlocks}.  This work considers stateless
model checking of multithreaded programs, but like our work it must
consider code blocks that perform multiple memory
operations.

CHESS~\cite{musuvathi2007chess} is designed to find and reproduce
concurrency bugs in C, C++, and C\#.  It systematically explores
thread interleavings using a preemption bounded strategy.  The Inspect
tool combines stateless model checking and stateful model checking to
model check C and C++
code~\cite{inspect1,inspect2,inspect4}.

In stateful model checking, there has also been substantial work such as
SPIN~\cite{spin}, Bogor~\cite{dwyer2003bogor}, and JPF~\cite{jpf}.
In addition to these model checkers, other researchers have
proposed different techniques to capture
program states~\cite{musuvathi2002cmc,gueta2007cartesian}.

Versions of JPF include a partial order reduction algorithm.  The
design of this algorithm is not well documented, but some publications
have reverse engineered the pseudocode~\cite{porjpf}.  The algorithm
is naive compared to modern DPOR algorithms---this algorithm simply
identifies accesses to shared variables and adds backtracking points
for all threads at any shared variable access.

redpajama_set_name: RedPajamaArXiv