text:
\section{Introduction}
\label{sec:intro}

Two central design decisions in modern conversational AI systems are the choices of state and action representations, which determine the scope of possible user requests and agent behaviors.
Dialogue systems with fixed symbolic state representations (like slot filling systems) are easy to train but hard to extend \cite{Pieraccini92Slots}. Deep continuous state representations are flexible enough to represent arbitrary properties of the dialogue history, but so unconstrained that training a neural dialogue policy ``end-to-end''
fails to learn appropriate latent states \cite{Bordes16EndToEndDialogue}.

This paper introduces a new framework for dialogue modeling that aims to combine the strengths of both approaches: structured enough to enable efficient learning, yet flexible enough to support
open-ended, compositional user goals that involve multiple tasks and domains.
The framework has two components: a new \textbf{state representation} in which dialogue states are represented as dataflow graphs; and a new \textbf{agent architecture} in which dialogue agents predict compositional programs that extend these graphs.
Over the course of a dialogue, a growing dataflow graph serves as a record of common ground: an executable description of the entities that were mentioned and the actions and computations that produced them (\autoref{fig:teaser}).

While this paper mostly focuses on representational questions,
learning is a central motivation for our approach.  Learning to interpret natural-language requests is simpler when they are understood to specify graph-building operations.
Human speakers avoid repeating themselves in conversation by using
anaphora, ellipsis, and bridging to build on shared context \cite{Mitkov14Anaphora}.
Our framework treats these constructions by translating them into explicit \emph{metacomputation operators} for reference and revision, which directly retrieve fragments of the dataflow graph\xspace that represents the shared dialogue state.
This approach borrows from corresponding ideas in the literature on program transformation
\cite{Visser01ProgramTransformation}
and results in compact, predictable programs whose structure closely mirrors
user utterances.

Experiments show that our rich dialogue state representation
makes it possible to build better dialogue agents for challenging tasks.
First, we release a newly collected dataset of around 40K natural dialogues in English
about calendars, locations, people, and weather---the largest goal-oriented dialogue dataset to date. %
Each dialogue turn is annotated with a program implementing the user request. Many
turns involve more challenging predictions than traditional slot-filling,
with compositional actions, cross-domain interaction, complex
anaphora, and exception handling (\autoref{fig:smcalendar-example}).
On this dataset, explicit reference mechanisms
reduce the error rate of a seq2seq-with-copying model \cite{See2017PointerNet}
by
5.9\% on all turns and by
10.9\% on turns with a cross-turn reference.
To demonstrate breadth of applicability, we additionally describe how to automatically convert the simpler MultiWOZ
dataset into a dataflow representation.  This representation again enables a
basic seq2seq model to outperform a state-of-the-art, task-specific model at traditional state tracking.
Our results show that within the dataflow framework,
a broad range of agent behaviors are both representable and learnable, and that
explicit abstractions for reference and revision are the keys to
effective modeling.

\section{Overview: Dialogue and Dataflow}
\label{sec:dataflow}

This section provides a high-level overview of our dialogue modeling framework, introducing
the main components of the approach.
Sections~\ref{sec:salience}--\ref{sec:recovery}
refine
this picture, describing the implementation and use of specific metacomputation operators.

We model a dialogue between a (human) \textbf{user} and an (automated)
\textbf{agent} as an interactive programming task where the human and computer
communicate using natural language.
Dialogue state is represented with a dataflow graph.
At each turn, the agent's goal is to translate the most recent user
utterance into a program.
Predicted programs nondestructively extend the dataflow graph,
construct any newly requested values or real-world side-effects,
and finally describe the results to the user.
Our approach is significantly different from a conventional dialogue system pipeline, which has separate modules for language understanding, dialogue state tracking, and dialogue policy execution \cite{young2013ieee}.
Instead, a single learned model directly predicts executable agent
actions and logs them in a graphical dialogue state.

\paragraph{Programs, graphs, and evaluation}

The simplest example of interactive program synthesis is \textbf{question answering}:

\begin{Dialogue}
    \User{When is the next retreat?}
    \AgentDo{%
      start(findEvent(EventSpec(\newline
      \strut~~name=\textquotesingle{}retreat\textquotesingle,\newline
      \strut~~start=after(now()))))
    }
    \AgentSay{It starts on April 27 at 9 am.}
\end{Dialogue}%

\noindent
Here the agent \defn{predicts} a program that invokes an API call (\AgentAction{findEvent}) on a structured input (\AgentAction{EventSpec}) to produce the desired query.\footnote{Note that what the agent predicts is not a formal representation of the utterance's \emph{meaning}, but a query that enables a contextually
appropriate \emph{response} (what \citet{Austin62How} called the ``perlocutionary force'' of the utterance on its hearer). The fact that \UserUtt{next} in this context triggered a search for ``events after now'' was learned from annotations. See \autoref{sec:data} for a discussion of how these annotations are standardized
in the SMCalFlow\xspace dataset.}
This is a form of \textit{semantic parsing} \citep{Zelle95GeoQuery}.

The program predicted above can be rendered as a \textbf{dataflow graph\xspace{}}:
\begin{center}
  \includegraphics[height=.9in,clip,trim=.12in 6.3in 8.1in 1.2in]{figs/dataflow1a.pdf}
\end{center}
Each function call in the program corresponds to a node
labeled with that function. This node's parents correspond to the arguments of the function call. The top-level call that returns the program's result is depicted with a solid border.
A dataflow graph\xspace is always acyclic, but is not necessarily a tree, as nodes may be reused.

Once nodes are added to a dataflow graph\xspace, they are  \textbf{evaluated} in topological order.  Evaluating a node applies its function to its parents' values:
\begin{center}
  \includegraphics[height=.8in,clip,trim=.12in 6.4in 8.17in 1.25in]{figs/dataflow1b.pdf}
\end{center}
Here we have annotated two nodes to show that the value of \AgentAction{after(now())} is a \AgentAction{DateTimeSpec}
and the value of the returned \AgentAction{start} node is a specific \AgentAction{DateTime}.  Evaluated nodes are shaded in our diagrams.  Exceptions (see \autoref{sec:recovery}) block evaluation, leaving downstream nodes unevaluated.

The above diagram saves space by summarizing the (structured) value of a node as a string.  In reality, each evaluated node has a dashed \defn{result edge} that points to the result of evaluating it:\looseness=-1
\begin{center}
  \includegraphics[height=.8in,clip,trim=.12in 6.2in 8.77in 1.35in]{figs/dataflow1bx.pdf}
\end{center}
That result is itself a node in the dataflow graph---often a new node \emph{added} by evaluation. It may have its own result edge.\footnote{\label{fn:returncode}In other words, a function does not have to return a terminal node.  Its result may be an existing node, as we will see in \autoref{sec:salience}.  Or it may be a new non-terminal node, i.e.,\xspace the root of a subgraph that implements the function. The new nodes in the subgraph are then evaluated further, giving them result edges, although they also remain available for reference and revision.  Of course, a library function such as  \AgentAction{findEvent} or \AgentAction{+} that invokes an API will generally return its value directly as a terminal node.  However, translating natural language to higher-level function calls, which have been defined to expand into lower-level library calls (reminiscent of macro expansion), is often more easily learnable and more maintainable than translating it directly to the expanded graph.\looseness=-1} A node's \defn{value} is found by transitively following result edges until we arrive at a node whose result is itself.  Such a \defn{terminal node} is either a primitive value (e.g.,\xspace \AgentAction{2020}), or a constructor (e.g.,\xspace \AgentAction{DateTime}) whose parent nodes' values specify its arguments.  A constructor has the same (capitalized) name as the type it constructs.

\paragraph{Reference and revision} We now sketch two metacomputation functions whose evaluation extends the graph in complex ways.

As a representation of dialogue state, the dataflow graph records entities that
have been mentioned and the relationships among them.
All nodes in the dataflow graph\xspace are eligible to be
referenced by subsequent utterances.
Suppose, for example, that the user continues the previous dialogue fragment
with a follow-up question:\looseness=-1

\begin{Dialogue}
    \User{What day of the week is that?}
    \AgentDo{dayOfWeek(\MetaAction{refer}\xspace{}())}
    \AgentSay{It's a Monday.}
\end{Dialogue}

\noindent
The user's word \UserUtt{that} becomes the \MetaAction{refer}\xspace call in our predicted program, as it is a reference to ``some salient previously mentioned node.''  Evaluating \MetaAction{refer}\xspace here chooses the top-level node, \AgentAction{start}, from the previous turn.
That node is then used as an argument to a  \AgentAction{dayOfWeek} node (nodes existing from previous turns are shown here in lighter ink):
\begin{center}
  \includegraphics[height=.9in,clip,trim=.12in 6.3in 8.05in 1.2in]{figs/dataflow1c.pdf}
\end{center}
and evaluating the latter node applies \AgentAction{dayOfWeek} to \AgentAction{start}'s value.
This diagram is actually a simplification:
we will show in \autoref{sec:salience} how the \MetaAction{refer}\xspace call itself is also captured in the dataflow graph.

The user may next ask a question that changes the \emph{upstream} constraint on the event's start time:

\begin{Dialogue}
    \User{What about in 2021?}
    \AgentDo{%
    \MetaAction{revise}\xspace{}(\newline
    \strut~~new=DateTimeSpec(year=2021),\newline
    \strut~~oldLoc=Constraint[DateTimeSpec](),\newline
    \strut~~rootLoc=RoleConstraint(output))}
\end{Dialogue} \\[-1.5em]
A ``\AgentAction{new}'' \AgentAction{DateTimeSpec}
(representing \UserUtt{in 2021}) is to be substituted for some salient existing \AgentAction{old} node that has value type \AgentAction{DateTimeSpec} (in this case, the node \AgentAction{after(now())}).  The \MetaAction{revise}\xspace{} operator non-destructively splices in this new sub-computation and returns a revised version of the most salient computation containing \AgentAction{old} (in this case, the subgraph for the previous utterance,
\AgentAction{root}ed at \AgentAction{dayOfWeek}):\looseness=-1

\begin{center}
    \includegraphics[height=.9in,clip,trim=.12in 6.1in 8.05in 1.4in]{figs/dataflow1d.pdf}
\end{center}

As in the \MetaAction{refer}\xspace{} example, the target program (though not the
above subgraph)
corresponds closely to the user's new utterance, making it easy to predict. Like the utterance itself, the program does not specify the revised subgraph in full, but describes how to find and reuse relevant structure from the previous dataflow graph\xspace{}.

Given a dataset of turns expressed in terms of appropriate
graph-manipulation programs, the learning problem for a dataflow agent is the
same as for any other supervised contextual semantic parser.
We want to learn a function that maps user utterances to particular programs---a
well-studied task for which standard models exist.
Details of the model used for our experiments in this paper are
provided in \autoref{sec:experiments}.

\paragraph{Aside: response generation}

This paper focuses on \emph{language understanding}: mapping from a user's
natural language utterance to a formal response, in this case the value of the outlined
node returned by a program. Dialogue systems must also
perform \emph{language generation}: mapping from this formal response to a
natural-language response.
The dataset released with this paper includes output from a learned generation
model that can describe the value computed at a previous turn, describe the
structure of the computation that produced the value, and reference other nodes
in the dataflow graph via referring expressions. Support for structured,
computation-conditional generation models is another advantage of dataflow-based
dialogue state representations. While a complete description of
dataflow-based language generation is beyond the scope of this paper, we briefly describe the
components of the generation system relevant to the
understanding system presented here.

The generation model is invoked after the evaluation phase. It conditions on a
view of the graph rooted at the most recent return node, so generated responses
can mention both the previously returned value and the computation that produced it.
As the generation model produces the natural language response,
it \emph{extends the dataflow graph}. For example, if after the user query
``\emph{What's the date of the next retreat?}'' the agent responds:

\begin{Dialogue}
\AgentSay{It starts on April 27 at 9 am, and runs for 8 hours.}
\end{Dialogue}

\noindent
then it will also extend the dataflow graph to reflect that the event's duration was mentioned:

\begin{center}
    \includegraphics[height=.9in,clip,trim=.12in 6.1in 8.05in 1.4in]{figs/dataflow1_gen.pdf}
\end{center}

\noindent
The duration of the event is now part of the
common ground in the conversation and available for future reference by either the
agent or the user.

The generation model is also important for agent initiative:

\begin{Dialogue}
\User{Put an event on my calendar.}
\AgentSay{What should it be called?}
\end{Dialogue}

\noindent
As discussed in detail in \autoref{sec:recovery}, questions of this kind
can be generated in response to \emph{exceptions} generated by underspecified
user requests.
In the accompanying dataset release, the agent's
utterances are annotated with their dataflow graphs as extended by the generation model.
\Fixme{check}

\section{Reference resolution}
\label{sec:salience}

In a dialogue, entities that have been introduced once may be referred to again.
In dataflow dialogues, the entities available for reference are given by
the nodes in the dataflow graph\xspace{}. Entities are \textbf{salient} to conversation
participants to different degrees, and their relative salience determines the ways
in which they may be referenced \citep{lappin1994algorithm}.
For example, \textit{it} generally refers
to the most salient non-human
entity, while more specific expressions like
\textit{the Friday meeting} are needed to refer to accessible but less
salient entities. Not all references to entities are overt: if the
agent says ``\AgentUtt{You have a meeting tomorrow}'' and the user responds
``\UserUtt{What time?}'', the agent must predict the \emph{implicit} reference to a salient event.

\paragraph{Dataflow pointers}
We have seen that \MetaAction{refer}\xspace is used to find referents for referring expressions. In general, these referents may be existing dataflow nodes or new subgraphs for newly mentioned entities. We now give more detail about both cases. %

Imagine a dialogue in which the dataflow graph\xspace{} contains the following fragment (which translates a mention of \UserUtt{Easter} or answers \UserUtt{When is Easter?}):
\begin{center}
    \includegraphics[height=0.3in,clip,trim=.12in 7.5in 9.1in .7in]{figs/dataflow2a.pdf}
\end{center}
\vspace{-.3em}
Suppose the user subsequently  mentions \emph{the day after that}. We wish to
produce this computation: \vspace{0.3em}
\begin{center}
    \includegraphics[height=.55in,clip,trim=.12in 6.9in 8.9in 1.1in]{figs/dataflow2b.pdf}
\end{center}
In our framework, this is accomplished by mapping \UserUtt{the day after that} to
  \AgentAction{+(\MetaAction{refer}\xspace{}(), Days(1))}.
  The corresponding graph is not quite the one shown above, but it evaluates to the same value:

 \begin{center}
    \includegraphics[height=.9in,clip,trim=.12in 6.45in 8.4in 1.2in]{figs/dataflow2_alt.pdf}
 \end{center}
This shows how the \AgentAction{\MetaAction{refer}\xspace{}()} call is reified as a node in the dataflow graph\xspace.  Its result is the salient \AgentAction{findDateTime} node from the previous turn---whose own result, a specific \AgentAction{DateTime}, now serves as the \emph{value} of \MetaAction{refer}\xspace{}.  We show both result edges here.  Evaluating \AgentAction{+} adds a day to this old \AgentAction{DateTime} value to get the result of \AgentAction{+}, a new \AgentAction{DateTime}.\looseness=-1

To enable dataflow graph\xspace{} manipulation with referring expressions, all that is required is an implementation of \MetaAction{refer}\xspace that can produce appropriate pointers for both simple
references (\emph{that}) and complex ones (\emph{the first meeting}).

\paragraph{Constraints}

A call to \MetaAction{refer}\xspace{} is essentially a query that retrieves a node from the dialogue history, using a salience model discussed below. %
\MetaAction{refer}\xspace takes an optional argument: a \defn{constraint} on the returned node.
Indeed, the proper translation of \emph{that} in the context \UserUtt{the day after that} would be \AgentAction{\MetaAction{refer}\xspace{}(Constraint[DateTime])}.\footnote{Fortunately, this constraint need not be manually annotated.  Given the rest of the program, it can be inferred automatically by Hindley-Milner type inference~\cite{hindley_principal_1969,milner_theory_1978}, which establishes that this \texttt{\scriptsize\underline{refer}}\xspace{} node must return a \AgentAction{DateTime} if the program is to type-check.}
Constraints are predicates built from boolean connectives and the constructions illustrated below:\footnote{\label{fn:constraint-alias}\texttt{\scriptsize\underline{refer}}\xspace and \texttt{\scriptsize\underline{revise}}\xspace are not the only operations that take constraints as arguments.  For example, constraint arguments to \AgentAction{\scriptsize{findEvent}} and \AgentAction{\scriptsize{createEvent}} specify what sort of event is to be retrieved or created.  In these other contexts, to avoid distracting the reader, this paper uses \AgentAction{\scriptsize EventSpec} as an alias for \AgentAction{\scriptsize Constraint[Event]}.  It similarly uses aliases \AgentAction{\scriptsize PersonSpec} and \AgentAction{\scriptsize DateTimeSpec}.  (Our dataset does not.)}
\begin{itemize}[noitemsep,topsep=0pt]
            \item
                \textbf{Type constraints}: \emph{the
                meeting} maps to the call
                \AgentAction{\MetaAction{refer}\xspace{}(Constraint[Event]())},
                where the constraint matches all nodes
                with values of type \AgentAction{Event}.
            \item
                \textbf{Property constraints}:
                Assuming a structured \AgentAction{Event} type with a
                \AgentAction{date} property, and a \AgentAction{Date} type with a
                \AgentAction{weekday} property, \emph{the Thursday
                meeting} maps to this nested call:\\
                \AgentAction{\MetaAction{refer}\xspace{}(Constraint[Event](date=\\
                \strut~~Constraint[DateTime](weekday=thurs)))}

              \item
                \textbf{Role constraints}: A role constraint specifies a keyword and matches nodes that are \emph{used as} keyword arguments with that keyword. For example, \emph{the month} maps to \AgentAction{\MetaAction{refer}\xspace{}(RoleConstraint(month))} and resolves to the constant node \AgentAction{apr} in the dialogues in  \autoref{sec:dataflow}, since that node was used as a named argument \AgentAction{month=apr}. We further allow keyword paths that select arguments of arguments: \AgentAction{thurs} in the previous bullet would satisfy \AgentAction{\MetaAction{refer}\xspace{}(RoleConstraint([date,weekday]))}.

\end{itemize}

To interpret a natural language referring expression,
the \emph{program prediction} model only needs to translate it into a contextually appropriate constraint $C$.  \AgentAction{\MetaAction{refer}\xspace{}($C$)} is then evaluated using a separate \emph{salience retrieval model} that returns an appropriate node.
The following dialogue shows referring expressions in action:\looseness=-1

\begin{Dialogue}
    \User{What's happening this morning?}
    \AgentDo{findEvent(EventSpec(\newline
    \strut~~start=and(today(), during(morning())))))}
    \AgentSay{You have a planning meeting at 9 am.}
    \User{What do I have after that?}
    \AgentDo{findEvent(EventSpec(start=after(end(\newline
    \strut~~\MetaAction{refer}\xspace{}(Constraint[Event]())))))}
    \AgentSay{Your birthday lunch is at 12:30 pm.}
    \User{How many people are coming to the 9~am meeting?}
    \AgentDo{length(attendees(\newline
    \strut~~\MetaAction{refer}\xspace{}(Constraint[Event](start=am(9)))))
    }
    \AgentSay{5 people will be at the planning meeting.}
\end{Dialogue}

\noindent Here \UserUtt{the 9 am meeting} refers to the one that is salient from the first response, not an arbitrary one.

\paragraph{Salience retrieval model}

The salience retrieval model returns the most salient node satisfying the
underlying constraint. Our dataflow framework is agnostic to the implementation
of this model. A sophisticated model could select nodes via a machine-learned
scoring function. In the experiments in this paper, however, we rank nodes using a
hard-coded heuristic.

The heuristic chooses the root node $r$ of the previous user utterance, if it satisfies the constraint.  More generally, the heuristic prefers nodes to the extent that they can be reached from $r$ in a small number of steps, where a step may move from a node to one of its input nodes, from an evaluated node to its result node, or from the root of an utterance to the root of an adjacent (user or system) utterance.
If no satisfying node is found in the past several utterances, the heuristic
falls back to generating code (see \autoref{fn:returncode}) that will search harder for a satisfying salient entity, perhaps by querying a database.
For example, our earlier \AgentAction{Constraint[Event](start=am(9))} may return the expression \AgentAction{findEvent(EventSpec(start=am(9)))} if no \emph{9~am meeting} has been mentioned recently,
and \AgentAction{Constraint[Person](name=`Adam')} may return \AgentAction{findPerson(PersonSpec(name=`Adam'))} if no \emph{Adam} has been mentioned.  (See \autoref{fn:constraint-alias}.)

\section{Revision}
\label{sec:revision}

Beyond referring to previously mentioned \emph{entities} (nodes), task-oriented
dialogues frequently refer to
previously executed \emph{computations} (subgraphs). This is one of the major
advantages
of representing the dialogue state as a dataflow graph\xspace of computations, not just a set of potentially salient entities.

\begin{Dialogue}
    \User{What time on Tuesday is my planning meeting?}
    \AgentDo{start(findEvent(EventSpec(\newline
      \strut~~name=\textquotesingle{}planning\textquotesingle{},\newline
      \strut~~start=DateTimeSpec(weekday=tuesday))))
  }
    \AgentSay{You meet with Grace at noon.}
    \User{Sorry, I meant all-hands.}
    \AgentSay{Your all-hands meeting is at 2:30 pm.}
\end{Dialogue}

\noindent
The second user utterance asks for the computation from the first user
utterance to be repeated, but with \UserUtt{all-hands} in place of \UserUtt{planning}.
The expected result is still a time, even though the second utterance makes no
mention of time.

In the dataflow framework,
we invoke a \MetaAction{revise}\xspace operator to construct the revised computation:

\begin{Dialogue}
    \User{Sorry, I meant all-hands.}
    \AgentDo{\MetaAction{revise}\xspace{}(%
            rootLoc=RoleConstraint(output),\newline
    \strut~~oldLoc=Constraint[String](),\newline
    \strut~~new=\textquotesingle{}all-hands\textquotesingle{})
    }
\end{Dialogue}

\noindent
Again, the content of the program closely reflects that of the corresponding utterance. The \MetaAction{revise}\xspace operator takes three arguments:
\begin{itemize}[nosep]
    \item \AgentAction{rootLoc}, a constraint to find the top-level node of the original computation;
    \item \AgentAction{oldLoc}, a constraint on the node to replace within the original computation;
    \item \AgentAction{new}, a new graph fragment to substitute there.
\end{itemize}
The \MetaAction{revise}\xspace node evaluates to the root of a \emph{modified copy} of the original computation, in which \AgentAction{new} now fills the role at the ``old'' location. Normally \AgentAction{rootLoc} is \AgentAction{RoleConstraint(output)}, which selects a node returned by a previous user utterance; thus, we \MetaAction{revise}\xspace that entire original request.

Revision is non-destructive---%
no part of the dialogue history is lost, so entities computed
by the original \texttt{target} and its ancestors remain available for
later reference.  However, the copy
shares nodes with the original computation where possible,
to avoid introducing unnecessary duplicate nodes that would have to be
considered by \MetaAction{refer}\xspace.

For the example dialogue at the beginning of this section, the first turn
produces the light gray nodes below.
The second turn adds the darker gray nodes, which specify the desired revision.
\begin{center}
  \includegraphics[height=1.3in,clip,trim=.12in 4.6in 7.9in 2.5in]{figs/dataflow3_alta.pdf}
\end{center}
Finally, evaluating the \MetaAction{revise}\xspace node selects the salient locations that match the \AgentAction{rootLoc} and \AgentAction{oldLoc} constraints (indicated in the above drawing by temporary dotted lines),
and constructs the revised subgraph (the new \AgentAction{start} node below and its ancestors).  The \emph{result} of evaluation (dashed arrow below) is the root of the revised subgraph.
\begin{center}
    \includegraphics[height=.85in,clip,trim=.12in 5.6in 8in 1.8in]{figs/dataflow3_altb.pdf}
\end{center}
Finally, evaluating these new nodes as well will establish that the value of the top-level \MetaAction{revise}\xspace is the start time of the `all-hands' meeting on Tuesday.

In the following example, the second utterance asks to replace a date specification.  However, the utterance appears in a context where the relevant \AgentAction{DateTimeSpec}-valued node to be ``replaced'' is an argument that has actually not yet been provided:

\begin{Dialogue}
  \User{When is lunch?}
  \AgentDo{start(findEvent(EventSpec(name=\textquotesingle{}lunch\textquotesingle{})))}
  \AgentSay{It's at 1 pm.}
  \User{What about tomorrow?}
  \AgentDo{\MetaAction{revise}\xspace{}(\newline
  \strut~~rootLoc=RoleConstraint(output),\newline
  \strut~~oldLoc=Constraint[DateTimeSpec],\newline
  \strut~~new=tomorrow())
  }
\end{Dialogue}

\noindent The revision replaces the missing \AgentAction{start} argument to the previous \AgentAction{EventSpec} (whose absence had resulted in some default behavior) with an \emph{explicit} argument (the \AgentAction{DateTimeSpec} returned by \AgentAction{tomorrow()}).  To achieve this, when the salience retrieval model is run with an \AgentAction{oldLoc} constraint, it must be able to return missing arguments that satisfy that constraint.  Missing arguments are implicitly present, with special value \AgentAction{missing} of the appropriate type.  In practice they are created on demand.%

\noindent
\begin{center}
    \includegraphics[height=1.2in,clip,trim=.6in 5in 7.25in 2in]{figs/dataflow5a.pdf}
\end{center}
Evaluating the \MetaAction{revise}\xspace node results in the new, revised subgraph pointed to by the dashed arrow, (which can then be evaluated):
\begin{center}
    \includegraphics[height=.8in,clip,trim=.2in 5.5in 7.7in 2in]{figs/dataflow5_alt.pdf}
    \vspace{-.5em}
\end{center}

Relatedly, a user utterance sometimes modifies a previously mentioned constraint such as an \AgentAction{EventSpec} (see \autoref{fn:constraint-alias}).  To permit this and more, we allow a more flexible version of \MetaAction{revise}\xspace to (non-destructively) transform the subgraph at \AgentAction{oldLoc} by applying a function, rather than by substituting a given subgraph \AgentAction{new}. Such functions are similar to rewrite rules in a term rewriting system \citep{Klop90TermRewriting}, with the \AgentAction{oldLoc} argument supplying the condition.
Our dataset (\autoref{sec:data}) makes heavy use of \MetaAction{reviseConstraint}\xspace calls, which modify a constraint as directed, while weakening it if necessary so that it remains satisfiable.  For example, if a 3:00--3:30 meeting is onscreen and the user says \emph{make it 45 minutes} or \emph{make it longer}, then the agent can no longer preserve previous constraints \AgentAction{start=3:00} and \AgentAction{end=3:30}; one must be dropped.

While the examples in this section involve a single update, real-world dialogues
(\autoref{sec:data})
can involve single user requests built up over
as many as five turns with unrelated intervening discussion.  Revisions of revisions or of constraints on reference are also seamlessly handled: \MetaAction{revise}\xspace can take another \MetaAction{revise}\xspace or a \MetaAction{refer}\xspace node as its \AgentAction{target}, leading to a longer chain of result edges (dashed lines) to follow.
Coordination of interactions among this many
long-range dependencies remains a challenge even for modern attentional architectures \cite{Bordes16EndToEndDialogue}.
With \MetaAction{revise}\xspace all the needed information is in one
place; as experiments will show, this is crucial for good performance in more
challenging dialogues.

\section{Recovery}
\label{sec:recovery}

Sometimes users make requests that can be fulfilled only with the help of followup exchanges,
if at all. Requests might be incomplete:

\begin{Dialogue}
    \User{Book a meeting for me.}
    \AgentSay{When should the meeting start?}
\end{Dialogue}

\noindent
referentially ambiguous:

\begin{Dialogue}
    \User{Who is coming to the planning meeting?}
    \AgentSay{Susan Chu and Susan Brown.}
    \User{What is Susan's email?}
\end{Dialogue}

\noindent
or have no identifiable referent (a presupposition failure):

\begin{Dialogue}
    \User{When is my first meeting on February 30?}
\end{Dialogue} \\[-2em]

\noindent Our solution is to treat such discourse failures as exceptions.  In principle, they are no different from other
real-world obstacles to fulfilling the user's request (server errors, declined
credit cards, and other business logic). To be useful, a dialogue model must
have some way to recover from all these exceptions, describing the problem to the user and guiding the dialogue past it.

Our dialogue manager consists \emph{mainly} of an exception recovery mechanism.  This contrasts with traditional slot-filling systems, where a scripted policy
determines which questions to ask the user and in which order.
Scripted policies are straightforward
but cannot handle novel compositional utterances.
Contextual semantic parsers treat compositionality, but
provide no dialogue management mechanism at all.  Our dataflow-based approach allows the user to express complex compositional intents, but also allows the agent to reclaim the initiative when it is unable to make progress.  Specifically, the agent can elicit interactive repairs of the problematic user plan: the user communicates such repairs through the reference and revision mechanisms described in preceding sections.

\paragraph{Exceptions in execution}

In the dataflow graph\xspace{} framework, failure to interpret a user utterance is signaled
by \textbf{exceptions}, which occur during evaluation.
The simplest exceptions result from
errors in function calls and constructors:

\begin{Dialogue}
    \User{What do I have on February 30?}
    \AgentDo{findEvent(EventSpec(\newline
    \strut~~start=DateTimeSpec(month=feb, day=30)))
    }
\end{Dialogue}

\noindent
Evaluation of the dataflow graph\xspace specified by this program cannot be completed. The \AgentAction{DateTimeSpec} constructor generates an exception,
and descendants of that node remain unevaluated. %

\begin{center}
    \includegraphics[height=.6in,clip,trim=.12in 6.6in 8in 1.3in]{figs/dataflow4.pdf}
\end{center}

An exception is essentially just a special result (possibly a structured value) returned by evaluation.  It appears in the dataflow graph, so the agent can condition on it when predicting programs in future turns.
When an exception occurs, %
the generation model (\autoref{sec:dataflow}) is invoked on the exceptional node. This can be used to produce prompts like:

\begin{Dialogue}
    \AgentSay{There is no 30th of February. Did you mean some other date?}
\end{Dialogue}%

\noindent
At this point, recovering from the exception looks like any other revision step:
the user supplies a new value, and the agent simply needs to patch it into the
right location in the dataflow graph\xspace. There are several answers the user could make,
indicating repairs at different locations:

\begin{Dialogue}
    \User{I meant February 28.}
    \vspace{-2pt}
    \AgentDo{\MetaAction{revise}\xspace{}(%
            rootLoc=RoleConstraint(output),\newline
    \strut~~oldLoc=Constraint[DateTimeSpec](),\newline
    \strut~~new=DateTimeSpec(month=feb, day=28))
    }
\end{Dialogue}

\vspace{-8pt}
\begin{Dialogue}
    \User{I meant March.}
    \vspace{-2pt}
    \AgentDo{\MetaAction{revise}\xspace{}(%
            rootLoc=RoleConstraint(output),\newline
    \strut~~oldLoc=Constraint[Month],\newline
    \strut~~new=mar)
    }
\end{Dialogue}

\noindent
The fact that exception recovery looks like any other turn-level prediction is
another key advantage of dataflow-based state representations. In the above
examples, the user specified a revision that would enable them to continue, but
they also would have been free to try another utterance (\UserUtt{List
all my meetings in February}) or to change goals altogether (\UserUtt{Never
mind, let's schedule a vacation}).

Because of its flexibility, our exception-handling mechanism is suitable for
many situations that have not traditionally been regarded as exceptions. For
example, an interactive slot-filling workflow
can be achieved via a sequence of
underspecified constructors, each triggering an exception and eliciting a revision from the user:

\begin{Dialogue}
    \User{Create a meeting.}
    \AgentDo{%
    createEvent()\newline
    $\dasharrow$ UnderconstrainedException!(name)
    }
    \AgentSay{What should it be called?}
    \User{Planning meeting.}
    \AgentDo{\MetaAction{revise}\xspace{}(%
            rootLoc=RoleConstraint(output),\newline
    \strut~~oldLoc=RoleConstraint(name),\newline
    \strut~~new=\textquotesingle{}Planning meeting\textquotesingle) \newline
    \strut $\dasharrow$ UnderconstrainedException!(start)}
    \AgentSay{When should it start?}
\end{Dialogue}

\noindent
The agent predicted that the user intended to revise the missing \texttt{name} because an exception involving the \AgentAction{name} path appeared in the dialogue history on the previous turn. %

Recovery behaviors are enabled by the phase separation between constructing the dataflow graph\xspace (which is the job of program synthesis from natural language) and evaluating its nodes.  The dataflow graph\xspace always
contains a record of the user's current goal, even when the goal could not be successfully evaluated. This goal persists across turns and remains accessible
to reference, and thus can be interactively refined and clarified
using the same metacomputation operations as user-initiated revision.
Exception handling influences the course of the dialogue,
without requiring a traditional hand-written or learned ``dialogue policy'' that reasons about full dialogue states.  Our policy only needs to generate language (recall \autoref{sec:dataflow}) that reacts appropriately to any exception or exceptions in the evaluation of the most recent utterance's program, just as it reacts to the ordinary return value in the case where evaluation succeeds.

\section{Data}
\label{sec:data}

\begin{figure}[t!]
{
\begin{Dialogue}[\scriptsize]
    \User{Can you remind me to go to the airport tomorrow morning at 8am?}
    \AgentDo{\scriptsize createCommitEventWrapper(\newline
    \strut~~createPreflightEventWrapper(\newline
    \strut~~~~EventBuilder(\newline
    \strut~~~~~~subject=\textquotesingle{}go to the airport\textquotesingle{},\newline
    \strut~~~~~~start=dateAtTime(\newline
    \strut~~~~~~~~~date=tomorrow(),\newline
    \strut~~~~~~~~~time=numberAM(8)))))
    }
    \AgentSay{Does this look right? [showing an event card with start at 8 am and the event name]}
    \User{Make the location LaGuardia Airport.}
    \AgentDo{\scriptsize \textsc{\underline{reviseConstraint}}( \newline
    \strut~~rootLoc=RoleConstraint(output), \newline
    \strut~~oldLoc=Constraint[EventBuilder](),\newline
    \strut~~new=EventBuilder(\newline
    \strut~~~~location=LocationKeyphrase(\newline
    \strut~~~~~~\textquotesingle{}LaGuardia Airport\textquotesingle{})))
    }
    \AgentSay{How is this? [showing an updated card]}
    \User{What can I expect the weather there to be tomorrow morning?}
    \AgentDo{\scriptsize weatherQueryApi(\newline
    \strut~~place=findPlace(PlaceSpec(\newline
    \strut~~~~keyphrase=\texttt{\scriptsize\underline{refer}}\xspace{}(\newline
    \strut~~~~~~Constraint[LocationKeyphrase]()))),\newline
    \strut~~time=DateTimeSpec(\newline
    \strut~~~~date=tomorrow(),\newline
    \strut~~~~constraint=morning()))
    }
    \AgentSay{It will be partly cloudy with a high of 77 °F and a low of 67 °F in LaGuardia Airport between 6:00 AM and 12:00 PM.}
    \User{How is the traffic on the way there?}
    \AgentDo{\scriptsize fenceNavigation()}
    \AgentSay{I can't answer questions about transit.}
    \User{That's fine. Book that please.}
    \AgentDo{\scriptsize confirmAndReturnAction()}
    \AgentSay{I've put that on your calendar. [showing the confirmed event card]}
\end{Dialogue}
}
\vspace{-1em}
\caption{A sample annotated dialogue in SMCalFlow\xspace.
Turn 1 features free-text subject and date/time.
Turn 2 features \MetaAction{reviseConstraint}\xspace.
Turn 3 features cross-domain interaction via \MetaAction{refer}\xspace and nested API calls (\AgentAction{findPlace} and \AgentAction{weatherQueryApi} are both real-world APIs).
Turn 4 features an out-of-scope utterance that is parried by a category-appropriate ``fencing'' response.
Turn 5 confirms a proposal after intervening turns.
}
\label{fig:smcalendar-example}
\Fixme{are these examples consistent with our current dataset? (\autoref{fn:constraint-alias} already notes that PlaceSpec and DateTimeSpec are aliases for Constraints that are actually in the dataset)}
\end{figure}

To validate our approach, we crowdsourced a large English dialogue dataset, SMCalFlow\xspace,
featuring task-oriented conversations about calendar events, weather, places, and people.
\autoref{fig:smcalendar-example} has an example. SMCalFlow\xspace has several key characteristics:

\textbf{Richly annotated}:
    Agent responses are executable programs,
    featuring API calls, function composition, and complex constraints built from strings,
    numbers, dates and times in a variety of formats. They are not key-value structures
    or database queries, but instead full descriptions of the runtime behavior needed
    to react to the user in a real, grounded dialogue system.

\textbf{Open-ended}:
    We did not constrain crowdworkers to scripts.  Instead, they were given general information about agent capabilities and were encouraged to interact freely.
    A practical dialogue system must also recognize and respond to out-of-scope requests.
    Our dataset includes many such examples (see the fourth user turn in \autoref{fig:smcalendar-example}).

\textbf{Cross-domain}:
    SMCalFlow\xspace spans four major domains: calendar, weather, places, and people.
    Cross-domain interaction is pervasive (\autoref{fig:smcalendar-example}).

To cover a rich set of back-end capabilities while encouraging worker creativity, we designed a wide range of \textit{scenarios} to guide dialogue construction.
There are over 100 scenarios of varying topic and granularity.
Dialogues are collected via a Wizard-of-Oz process. Every dialogue is associated with a scenario. At each turn, a crowdworker acting as the user is presented with a dialogue as context and is asked to append a new utterance.
An annotator acting as the agent labels the utterance with a program (which may include \MetaAction{refer}\xspace and \MetaAction{revise}\xspace) and then selects a natural-language response from a set of candidates produced by the language generation model described in \autoref{sec:dataflow}. The annotation interface includes an autocomplete feature based on existing annotations. Annotators also
populate databases of people and events
to ensure that user requests have appropriate responses. %
The process is iterated %
for a set number of turns or until the annotator indicates the end of conversation.
A single dialogue may include turns from multiple crowdworkers and annotators.

Annotators are provided with detailed guidelines containing example annotations
and information about available library functions.
Guidelines also specify conventions for pragmatic issues like the decision to annotate \emph{next} as \AgentAction{after} at the beginning of \autoref{sec:dataflow}. Crowdworkers are recruited from Amazon Mechanical Turk with qualification requirements such as living in the United States and with a work approval rate higher than 95\%.

Data is split into training, development, and test sets.
We review every dialogue in the \emph{test set} with two additional annotators. 75\% of turns pass through this double review process with no changes, which serves as an approximate measure of inter-annotator consensus on full programs.

\begin{table*}[t!]
    \centering
    \footnotesize
    \scalebox{0.94}{
    \begin{tabular}{@{}l cccccccc@{}}
    \toprule
         & \# Dialogues & \# User Turns &  \makecell{User Vocab. \\ Size} & \makecell{Library \\ Size} & \makecell{Utterance\\Length} & \makecell{Program\\Length} & \makecell{Program\\Depth} & OOS \\
         \midrule
         SMCalFlow\xspace     & 41,517 & 155,923 & 17,397 & 338 & (5, 8, 11) & (11, 40, 57) & (5, 9, 11) & 10,466\\
         MultiWOZ 2.1    & 10,419 & 71,410 & 4,105 & 35 & (9, 13, 17) & (2, 29, 39) & (2, 6, 6) & 0 \\
    \bottomrule
    \end{tabular}
    }
    \caption{Dataset statistics.
    ``Library\ Size'' counts distinct function names (e.g.,\xspace \AgentAction{findEvent}) plus keyword names (e.g.,\xspace \AgentAction{start=}).
    ``Length'' and ``Depth'' columns show (.25, .50, .75) quantiles. For programs, ``Length'' is the number of function calls and ``Depth'' is determined from a tree-based program representation.
    ``OOS'' counts the out-of-scope utterances.  MultiWOZ statistics were calculated after applying the data processing of \citet{Wu2019Trade}.
    Vocabulary size is less than reported by \citet{Goel2019HyST} because of differences in tokenization (see code release).\looseness=-1}
    \label{tab:data_statistics}
\end{table*}

For comparison, we also produce a version of the popular MultiWOZ 2.1 dataset \citep{Budzianowski18MultiWOZ,Eric2019MultiWOZ} with dataflow-based annotations.
MultiWOZ is a state tracking task, so in its original format the dataset annotates each turn with a dialogue state rather than an executable representation.
To obtain an equivalent (program-based) representation for MultiWOZ, at each user turn
we automatically convert the annotation to a dataflow program.\footnote{We release the conversion script along with SMCalFlow\xspace.}
Specifically, we represent each non-empty dialogue state as a call to an event booking function, \AgentAction{find}, whose argument is a \AgentAction{Constraint} that specifies the desired type of booking along with values for some of that type's slots.  Within a dialogue, any turn that initiates a new type of booking is re-annotated as a call to \AgentAction{find}.  Turns that merely modify some of the slots are re-annotated as \MetaAction{reviseConstraint}\xspace calls.  Within either kind of call, any slot value that does not appear as a substring of the user's \emph{current} utterance (all slot values in MultiWOZ are utterance substrings) is re-annotated as a call to \MetaAction{refer}\xspace with an appropriate type constraint, provided that the reference resolution heuristic would retrieve the correct string from earlier in the dataflow.  This covers references like \UserUtt{the same day}.  Otherwise, our re-annotation retains the literal string value. %

Data statistics are shown in \autoref{tab:data_statistics}.
To the best of our knowledge, SMCalFlow\xspace is the largest annotated task-oriented dialogue dataset to date.
Compared to MultiWOZ, it features a larger user vocabulary, a more complex space of state-manipulation primitives, and a long tail of agent programs built from numerous function calls and deep composition.

\section{Experiments}
\label{sec:experiments}

We evaluate our approach on SMCalFlow\xspace and MultiWOZ 2.1.
All experiments use the OpenNMT \cite{OpenNMT2017}
pointer-generator network \cite{See2017PointerNet},
a sequence-to-sequence model that can copy tokens from the source sequence while decoding.
Our goal is to demonstrate that dataflow-based representations benefit standard neural model architectures.
Dataflow-specific modeling might improve on this baseline, and we leave this as a challenge for future work.

For each user turn $i$,
we linearize the target  program into a sequence of tokens $z_i$.
This must be predicted from the dialogue context---namely the concatenated source sequence
$x_{i-c}\,z_{i-c}\cdots x_{i-1}\,z_{i-1}\,x_i$ (for SMCalFlow\xspace) or
$x_{i-c}\,y_{i-c}\cdots x_{i-1}\,y_{i-1}\,x_i$ (for MultiWOZ 2.1).
Here $c$ is a context window size,
$x_j$ is the user utterance at user turn $j$,
$y_j$ is the agent's natural-language response,
and $z_j$ is the linearized agent program.
Each sequence $x_j$, $y_j$, or $z_j$ begins with a separator token that indicates the speaker (user or agent).
Our formulation of context for MultiWOZ is standard \cite[e.g.,\xspace][]{Wu2019Trade}.
We take the source and target vocabularies to consist of all words that occur in (respectively) the source and target sequences in training data, as just defined.\looseness=-1

The model is trained using the Adam optimizer \cite{Kingma14Adam} with the maximum likelihood objective.
We use 0.001 as the learning rate. Training ends when there have been two different epochs that increased the %
development loss.

We use Glove800B-300d (cased) and Glove6B-300d (uncased) \cite{Pennington2014Glove} to initialize the vocabulary embeddings for the SMCalFlow\xspace and MultiWoZ experiments, respectively.
The context window size $c$,
hidden layer size $d$,
number of hidden layers $l$,
and dropout rates $r$
are selected based on
the agent action accuracy (for SMCalFlow\xspace)
or dialogue-level exact match (for MultiWoZ)
on the
development set
from \{2, 4, 10\}, \{256, 300, 320, 384\}, \{1, 2, 3\},
\{0.3, 0.5, 0.7\}
respectively.
Approximate 1-best decoding uses a beam of size 5.

\begin{table}[t]
    \centering
    \footnotesize
    \scalebox{.82}{
    \begin{tabular}{l|cc|cc|cc}
    \toprule
         & \multicolumn{2}{c|}{Full}
         & \multicolumn{2}{c|}{Ref.~Turns}
         & \multicolumn{2}{c}{Rev.~Turns} \\
         & dev & test & dev & test & dev  & test \\
         \midrule
         \# of Turns    & 13,499 & 21,224  & 3,554 & 8,965 & 1,052 & 3,315 \\
         \midrule
         Dataflow       & \bf .729 & \bf .665   & \bf .642 & \bf .574 & \bf .697 & \bf .565 \\
         inline         & .696 & .606   & .533 & .465 & .631 & .474 \\
    \bottomrule
    \end{tabular}
    }
    \caption{\textbf{SMCalFlow\xspace} results. Agent action
    accuracy is significantly higher than a
    baseline without metacomputation, especially on turns that involve
    a reference (Ref.\ Turns) or revision (Rev.\ Turns) to earlier turns in the dialogue ($p < 10^{-6}$, McNemar's test).}
    \label{tab:exp_smcal}
    \vspace{-7pt}
\end{table}

\paragraph{Quantitative evaluation} \autoref{tab:exp_smcal} shows results for the SMCalFlow\xspace dataset.
We report program
accuracy: specifically, exact-match accuracy of the predicted program after inlining metacomputation (i.e.,\xspace replacing all calls to metacomputation operations with the concrete program fragments
they return).\footnote{%
    Specifically, we inline all \texttt{\scriptsize\underline{refer}}\xspace calls and \texttt{\scriptsize\underline{revise}}\xspace calls that involve direct substitution of the kind described in \autoref{sec:revision}. We preserve {\footnotesize\underline{\texttt{reviseConstraint}}} calls to avoid penalizing baselines that do not have access to pre-defined constraint transformation logic.
}
We also compare to baseline models that \emph{train} on inlined metacomputation. These experiments make it possible to evaluate the importance of explicit dataflow manipulation compared to a standard contextual semantic parsing approach to the task: a no-metacomputation baseline can still reuse computations from previous turns via the model's copy mechanism.

For the full representation, $c$, $d$, $l$, and $r$ are 2, 384, 2, and 0.5, respectively.
For the inline variant, they are 2, 384, 3, and 0.5.
Turn-level exact match accuracy is around 73\% for the %
development set and 67\% for the test set.
Inlining metacomputation, which forces the model to explicitly resolve cross-turn computation,
reduces accuracy by 5.9\% overall, 10.9\% on turns involving references, and 9.1\% on turns involving revision. %
Dataflow-based metacomputation operations are thus essential for good model performance
in all three cases.

\begin{table}[t]
    \centering\footnotesize
    \scalebox{1}{
    \begin{tabular}{@{}l ccc@{}}
    \toprule
    & Joint Goal & Dialogue & Prefix \\
    \midrule
    Dataflow                & \bf .467  & \bf .220  & \bf 3.07 \\
    inline \texttt{\scriptsize\underline{refer}}\xspace       & .447      & \bf .202      & \bf 2.97 \\
    inline both             & \bf .467      & \bf .205      & 2.90 \\
    TRADE                   & .454      & .168      & 2.73 \\
    \bottomrule
    \end{tabular}
    }
    \caption{\textbf{MultiWOZ 2.1} test set results.
    TRADE \cite{Wu2019Trade} results are from the public implementation.
    ``Joint Goal'' \citep{Budzianowski18MultiWOZ} is average dialogue state exact-match, ``Dialogue'' is average dialogue-level
    exact-match, and ``Prefix'' is the average number of
    turns before an incorrect prediction.  Within each column, the best result is boldfaced, along with all results that are not significantly worse ($p < 0.05$, paired permutation test).  Moreover, \emph{all} of
    ``Dataflow,'' ``inline \MetaAction{refer}\xspace,'' and ``inline both'' have higher dialogue accuracy than
    TRADE ($p < 0.005$).}
    \label{tab:exp_multwoz}
    \vspace{-2pt}
\end{table}

\begin{table}[t]
    \centering
    \footnotesize
    \begin{tabular}{@{}lr@{}}
      \toprule
        Error category  & Count \\
        \midrule
        \bf Underprediction & \bf 21 \\
        \midrule
        \bf Entity linking & \bf 21 \\
        ~~~~Hallucinated & 7 \\
        ~~~~Wrong type & 7 \\
        ~~~~Wrong field & 2 \\
        ~~~~Boundary mismatch & 5 \\
        \midrule
        \bf Fencing & \bf 22 \\
        ~~~~Should have fenced & 9 \\
        ~~~~Shouldn't have fenced & 8 \\
        ~~~~Wrong message & 5 \\
        \midrule
        \bf Ambiguity & \bf 23 \\
        ~~~~Wrong in context & 8 \\
        ~~~~Acceptable (same semantics) & 8 \\
        ~~~~Acceptable (different semantics) & 7 \\
        \midrule
        \bf Miscellaneous & \bf 10 \\
        ~~~~Used wrong function & 4 \\
        ~~~~Other / Multiple & 6 \\ %
        \midrule
        \bf Error in gold & \bf 3 \\
      \bottomrule
    \end{tabular}
    \caption{Manual classification of 100 model errors on the SMCalFlow\xspace dataset. The largest categories are \emph{underprediction} (omitting steps from agent programs), \emph{entity linking} (errors in extraction of entities from user utterances, \emph{fencing} (classifying a user request as out-of-scope), and \emph{ambiguity} (user utterances with multiple possible interpretations). See \autoref{sec:experiments} for discussion.}
    \label{tab:error_analysis}
    \vspace{-1em}
\end{table}

We further evaluate our approach on
dialogue state tracking using MultiWOZ 2.1.
\autoref{tab:exp_multwoz} shows results.
For the full representation, the selected model uses $c=2$, $d=384$, $l=2$, and $r=0.7$.
For the inline \MetaAction{refer}\xspace variant, they are 4, 320, 3, and 0.3.
For the variant that inlines both \MetaAction{refer}\xspace and \MetaAction{revise}\xspace calls,
they are 10, 320, 2, and 0.7.
Even without metacomputation,
prediction of program-based
representations gives results comparable to the existing state
of the art, TRADE, on the standard ``Joint Goal'' metric (turn-level exact match).
(Our dataflow representation for MultiWOZ is designed so that dataflow graph
evaluation produces native MultiWOZ slot-value structures.)
However, Joint Goal does not fully characterize the effectiveness of a state
tracking system in real-world interactions, as it
allows the model to recover from an error at an earlier turn by
conditioning on gold agent utterances after the error. We thus evaluate
on \emph{dialogue}-level exact match and prefix length (the average
number of turns until an error).
On these metrics the benefit of dataflow over past approaches is clearer. %
Differences within dataflow model variants are smaller here than in \autoref{tab:exp_smcal}.
For the Joint Goal metric, the no-metacomputation baseline is better; we attribute this to the comparative simplicity of reference in the MultiWOZ dataset.
In any case, casting the state-tracking problem as one of program prediction with appropriate
primitives gives a state-of-the-art
state-tracking model for MultiWOZ using only off-the-shelf sequence prediction tools.\footnote{%
A note on reproducibility: Dependence on internal libraries prevents us from releasing a full salience model implementation and inlining script for SMCalFlow\xspace. The accompanying data release includes both inlined and non-inlined versions of the full dataset, and inlined and non-inlined versions of our model's test set predictions, enabling side-by-side comparisons and experiments with alternative representations. We provide full conversion scripts for MultiWOZ.
}

\paragraph{Error analysis}

Beyond the quantitative results shown in Tables \ref{tab:exp_smcal}--\ref{tab:exp_multwoz},
we manually analyzed 100 SMCalFlow\xspace turns where our model mispredicted.
\autoref{tab:error_analysis} breaks down the errors by type.

Three categories involve straightforward parsing errors.
In \textbf{underprediction} errors, the model fails to predict some computation (e.g.,\xspace a search
constraint or property extractor) specified in the user request. This behavior is not specific to
our system: under-length predictions are also well-documented in neural machine translation systems \cite{Murray18NMTLength}.
In \textbf{entity linking} errors, the model correctly identifies the presence of an entity mention in the input utterance, but uses it incorrectly in the input plan. Sometimes the entity that appears in the plan is \emph{hallucinated}, appearing nowhere in the utterance; sometimes the entity is cast to a \emph{wrong type} (e.g.,\xspace locations interpreted as event names) used in the \emph{wrong field} or extracted with \emph{wrong boundaries}.
In \textbf{fencing} errors, the model interprets an out-of-scope user utterance as an interpretable command, or vice-versa (compare to \autoref{fig:smcalendar-example}, turn 4).

The fourth category, \textbf{ambiguity} errors, is more interesting. In these cases, the predicted plan corresponds to an interpretation of the user utterance that would be acceptable in \emph{some} discourse context. In a third of these cases, this interpretation is ruled out by either dialogue context (e.g.,\xspace interpreting \emph{what's next?} as a request for the next list item rather than the event with the next earliest start time) or commonsense knowledge (\emph{make it at 8} means 8 a.m.\ for a business meeting and 8 p.m.\ for a dance party). In the remaining cases, the predicted plan expresses an alternative computation that produces the same result, or an alternative interpretation that is also contextually appropriate.

\section{Related work}
\label{sec:related}

The view of dialogue as an interactive process of shared plan synthesis dates
back to \citeauthor{Grosz86Intentions}'s earliest work on discourse structure
(\citeyear{Grosz86Intentions, Grosz88Plans}). That work represents the state of a
dialogue as
a predicate recognizing whether a desired piece of information has been
communicated or change in world state effected.
Goals can be refined via questions and corrections from both users and
agents.
The only systems to attempt full versions of this shared-plans framework \citep[e.g.,\xspace][]{Allen96Robust,Rich01Collagen} required inputs that could be parsed under a predefined grammar. Subsequent
research on dialogue understanding has largely focused on two
simpler subtasks:

\textbf{Contextual semantic parsing} approaches focus on complex language
understanding without reasoning about underspecified goals or agent initiative.
Here the prototypical problem is iterated question answering
\citep{Hemphill90ATIS,Yu2019Sparc}, in which the user asks a sequence of
questions corresponding to database queries, and results of query execution are
presented as structured result sets. \citet{Vlachos14SP} describe a semantic
parsing representation targeted at more general dialogue problems.
Most existing methods interpret
context-dependent user questions (\emph{What is the next flight to Atlanta? When
does it land?}) by learning to copy subtrees
\citep{Zettlemoyer09ContextATIS,Iyyer2017SequenceQA,Suhr2018ContextATIS} or
tokens \citep{Zhang2019EditingSQL} from previously-generated queries.
In contrast, our approach reifies reuse with explicit graph
operators.

\textbf{Slot-filling} approaches \citep{Pieraccini92Slots} model simpler utterances in the context of
full, interactive dialogues.
It is assumed that any user intent can be
represented with a flat structure consisting of a categorical dialogue act
and a mapping between a fixed set of
slots and string-valued fillers.
Existing fine-grained dialogue act schemes \citep{Stolcke2000DialogueActs} can distinguish among a
range of communicative intents not modeled by our approach,
and slot-filling representations have historically been
easier to predict \citep{Zue1994Pegasus}
and annotate \citep{Byrne2019Taskmaster}.
But while recent variants support interaction between
related slots \cite{Budzianowski18MultiWOZ} and fixed-depth hierarchies of slots
\cite{Gupta18HierarchicalSlots}, modern slot-filling approaches remain
limited in their support for semantic compositionality. By
contrast, our approach supports user requests corresponding to general
compositional programs.

More recent \textbf{end-to-end} dialogue agents attempt to map
directly from conversation histories to API calls and agent
utterances using neural sequence-to-sequence models without a
representation of dialogue state
\citep{Bordes16EndToEndDialogue,Yu19CoSQL}. While
promising, models in these papers fail to outperform rule- or template-driven
baselines. \citet{Neelakantan2019NeuralAssistant} report greater success on a
generation-focused task, and promising results have also been obtained from hybrid neuro-symbolic dialogue systems \cite{Zhao2016Towards,williams2017hybrid,Wen2017EACL,Gao19ConvSurvey}. Much of this work is focused on improving agent modeling for existing representation schemes like slot filling. We expect that many modeling innovations (e.g.,\xspace the neural entity linking mechanism proposed by \citeauthor{williams2017hybrid}) could be used in conjunction with the new representational framework we have proposed in this paper.

Like slot-filling approaches, our framework is aimed at modeling full dialogues
in which agents can ask questions, recover from errors, and take actions with
side effects, all backed by an explicit state representation. However, our
notions of ``state'' and ``action'' are much richer than in slot-filling
systems, extending to arbitrary compositions of primitive operators. We use semantic parsing as a modeling framework for dialogue
agents that can construct compositional states of this kind. While dataflow-based
representations are widely used to model execution state for programming
languages \cite{kam1976global}, this is the first work we are aware of that uses them to model conversational context and dialogue.

\section{Conclusions}
\label{sec:conclusion}

We have presented a representational framework for task-oriented dialogue modeling based on
dataflow graph\xspace{}s, in which dialogue agents predict a sequence of compositional updates
to a graphical state representation.
This approach makes it possible to represent
and learn from complex, natural dialogues.
Future work might focus on improving prediction by introducing learned implementations of \MetaAction{refer}\xspace and \MetaAction{revise}\xspace that, along with the program predictor itself, could evaluate their hypotheses for syntactic, semantic, and pragmatic plausibility.
The representational framework could itself be extended, e.g.,\xspace by supporting
declarative user goals and preferences
that
persist across utterances. %
We hope that the rich representations
presented here---as well as our new dataset---will facilitate greater use of context and
compositionality in learned models for task-oriented dialogue.

\section*{Acknowledgments}

We would like to thank Tatsunori Hashimoto, Jianfeng Gao, and the
anonymous TACL reviewers for feedback on early drafts of this paper.

redpajama_set_name: RedPajamaArXiv


So, if you are looking for a marketing method that is simple to track and monitor, that also gives a good ROI, then affiliate marketing should be in your toolbox.
However, if you are new to the concept, you may not understand how it works, or you might have preconceived ideas that are putting you off from progressing. To help you distinguish myth from fact, and get a better appreciation of how affiliate marketing could work for you, we've answered some of the most common questions marketing managers pose.
Q1. What is affiliate marketing?
Affiliate marketing involves offering your product or service to publishers (also known as affiliates) to promote on your behalf. In return, you pay a commission for every sale the publishers make. As a performance-based marketing method, you only pay for results.
Q2. How can affiliate marketing help my business?
Q3. How much is the global affiliate industry worth?
In our recent Awin report, we reported that the global annual investment in the industry is more than £12 billion annually.
Q4. What is an affiliate programme?
An affiliate programme refers to the package you create for your product, which is then made available to potential publishers. The programme will generally include details of the product, the retail value of the product, commission levels and promotional materials. Many affiliate programmes are managed via an affiliate network like Awin, which acts as a platform to connect publishers and advertisers, but it is also possible to offer your programme directly.
Q5. How do affiliate networks make money?
Affiliate networks make money through fees charged to the merchants, also known as advertisers. These fees are agreed with each individual advertiser and are usually paid as a percentage of each sale made.
Q6. What's the difference between affiliate marketing and drop-shipping?
Drop-shipping allows you to run an online store without having to stock products. You advertise the products as if you owned them, but when someone makes an order you then create a duplicate order with the company at a reduced price. The distributor takes care of the post and packaging on your behalf.
As affiliate marketing is based on referrals, when a customer buys through the affiliate link the affiliate does not have to handle any money as the customer is taken directly to the merchant's site to complete the purchase. The sale is tracked through cookies and a commission paid out to the referring affiliate.
The main difference is that affiliate marketing does not require the affiliate to fulfil the sales process or handle customer support. As a drop-shipper, you are still responsible for the sales procedure, even though you do not handle any inventory.
Q7. Can we use affiliate marketing for lead generation?
Normally affiliate marketing is used to generate sales. However, it can be used to create leads, by paying commission for every new subscriber created. Companies that offer a service rather than a product often opt for leads, as for them nurturing a lead is an important part of the sales funnel.
Remember also that every customer generated through affiliate marketing is a potential lead for further products. On top of this, Awin has a dedicated lead generation service that can be utilised for creating quality leads.
Q8. Are affiliate marketing and performance marketing the same thing?
Performance marketing is a method of marketing that encompasses different price structures, dependent on the performance of the ad. Examples include how many times the product is viewed or clicked, or how often a sale is made. It includes methods such as pay-per-click or display advertising.
Affiliate marketing is a form of performance marketing, as commissions are paid on performance, which is generally when a sale is made.
Q9. Is it possible to promote affiliate offers on mobile devices?
Smartphones are essentially computers in miniature, so can display the same websites and affiliate offers that a PC can. However, mobiles also have specific tools not so readily available on computers, and these can be used to good effect for publishers.
When setting up an ad, it can be optimised for mobile users, making it easy for this audience to access the offer. Publishers can also make good use of text and instant messaging promotions. As the mobile market is predicted to make up 80 percent of traffic in the future, publishers not only can promote on mobile devices, but they are missing a big trick if they are not.
Q10. What is a super affiliate?
A super affiliate is someone who has built a thriving and influential affiliate business and consistently drives a large majority of the sales from any programme they promote, compared to other affiliate partners involved in the programme. They make a large income from affiliate marketing, which is usually in the range of 5 or 6 figures.
Q11. Where do I find qualified publishers?
The best place to find qualified publishers is on reputable affiliate networks, like Awin, as they will have undergone a rigorous application process and compliance checks before joining the network. You can also advertise your programme on your website and invite publishers to sign up.
Q12. What is an affiliate disclosure statement?
An affiliate disclosure statement reveals to the reader that there may be affiliate links on the website, for which a commission may be paid to the publisher if these links are clicked and a sale is made.
Q13. How much commission should we pay publishers?
Commission rates are flexible and depend on various factors, including how well your website converts, and the industry standard for your type of product. However, the payment structure must be attractive enough to encourage publishers to join your programme, while also ensuring you make money on the product and leave some potential for increasing commissions for promotional placements.
Q14. We're worried about ad-blockers - are they an issue?
It's natural for affiliate marketers to be concerned about ad-blockers, as they can affect who sees your advertising. However, the online marketing world is constantly evolving to the changes thrown at it, and this is no exception. There is more positive news, as Chrome's new version ad blocker only blocks ads that don't comply with the Better Ads Standards. This means poor quality publishers are being driven out, while reputable publishers continue to thrive.
Q15. How do we deal with non-compliance issues?
The easiest way to deal with unscrupulous behaviour is to sign up to an affiliate network and publish your programme on the network's platform. Networks are skilled at compliance monitoring, which helps to prevent potential issues arising. Learn more about compliance in affiliate marketing and Awin's compliance team.
Q16. What happens if a publisher isn't performing in terms of sales?
Nothing – as you only pay out when the publisher creates a sale there is no risk created by underperforming publishers. However, having a good network of publishers is important for success as if your team is small your sales may be low.
Q17. How do we track publisher sales activity?
Every publisher has a unique publisher ID, which is used in tracking cookies to record which publisher generates what sales. This activity can then be viewed within the affiliate network dashboard.
Q18. Is social media activity an important part of affiliate marketing?
Publishers promote their programmes through various means, including blogs, websites, email marketing, pay-per-click and social media posts. However, social media has a huge interactive audience, making this platform a good source of potential traffic.
Q19. Can we have an affiliate programme in multiple countries?
Affiliate programmes can be promoted in any country that has a need of your product, and that your affiliate network operates in. If your site exists in multiple language versions, its best to set up an affiliate programme per language where possible so publishers can direct their audiences to the most suitable version.
Q20. What are the advantages of this type of marketing?
Because affiliate marketing only pays out commissions on sales, there is minimal risk that promotions will not generate ROI. It also exposes your product to new audiences, while tapping into the different skills of your publishers. Budgets are easy to manage and maintain due to the visibility and transparency the channel offers.
Q21. What are the disadvantages of this type of marketing?
If you decide not to use an affiliate network, managing your affiliate programme can be time-consuming. You will also need to be much more aware of compliance issues if you go it alone. Additionally, It can be challenging to build a team of loyal publishers who promote your programme long term.
Q22. How much growth is predicted for the affiliate industry?
Data gathered in 2018 showed that affiliate marketing spend in the UK has increased by 15 percent while in the US it is predicted to grow by 62 percent from 2015 to 2020. A 10.1 percent annual growth on spend is expected to continue.
Q23. Is affiliate marketing still expanding?
With the expected growth on spending set to increase, affiliate marketing is certainly still expanding and growing. Affiliate marketing and influencer marketing have been 'ruling the world of marketing' and this potent combination looks set to continue on its growth curve. By utilising both methods in tangent, expect to see affiliate marketing go from strength to strength.
"Affiliate marketing makes sure that you get more conversions while influencer marketing helps you reach a wider target audience" AM Navigator.
Q24. On average, how soon could we see a return on our investment?
Although there is not a fixed answer to this, expect to see a return after a year. Compare this to most small businesses, when breaking even after three years is a positive result.
Q25. What's the future of affiliate marketing?
Affiliate marketing is here to stay, even though it is likely to evolve and adapt to the changes that e-commerce throws at it. For example, Awin predicts that tech tracking will be in the spotlight, leading to companies reviewing their current software. There's also debate on the effect consolidation of some of the bigger affiliate marketing companies will have on the industry as a whole, as well as discussion on how International trade disagreements could have a less than positive impact on e-commerce growth across borders. However, the close relationships that publishers have with their consumers will likely stand the industry in good stead, as many marketers look to move to the 'Direct to Consumers' model.
If you're still not sure whether this will work for you, take a look at the advertisers on Awin's network who employ this method successfully. If they can do it, so can you.


text:
VoIP vendor Nuera snags fifth round of funding
This week's group of companies receiving venture funding included a surprising name: Nuera Communications Inc., a voice-over-IP product vendor that has been around since 1997.
The company, which has increasingly begun to focus on cable service providers looking to offer voice, raised US$20 million in its latest round. Not coincidentally, two of the lead investors are cable operators Comcast and Cox Communications. Sandler Capital Management and Argo Global Capital also participated in this round.
A Nuera spokeswoman says the funds will be used to further develop the company's cable telephony media gateways and its VoIP gateways, which are used for international long distance applications.
"This money will also sustain us until the cable operators start deploying in larger scale," she says. To date, Nuera has sold products to Comcast and overseas cable companies such as StarHub in Singapore and Liberty Media in Puerto Rico.
Cox invested in Nuera because "we want to support companies building products that work well in the cable plants that we have," says Jeff Brown, executive director of corporate development and investments. Brown says Cox has been impressed with Nuera's technology and pricing. The cable network operator, which also offers circuit-switched telephony in 10 markets, is trialing Nuera products for bundled telephony, cable TV and Internet access services.
To date, Nuera has raised US$109 million over five rounds of funding. That many rounds used to be unheard of, but extra rounds have become more common as venture firms try to keep companies afloat against a backdrop of limited exit strategies, such as IPOs or mergers. Nuera says it hopes to achieve profitability by year-end, but as a privately held company it does not disclose how much money it is losing or the amount of revenue it is generating.
Nuera burst onto the network scene six years ago pushing voice-over-frame relay products. The company spun out of an outfit called PCSI.
Over the years, Nuera has shifted its focus more to voice-over-IP gear via its ORCA gateways. The company, which competes with Cisco, General Bandwidth and Lucent, also counts a softswitch among its offerings.
Nuera is now working closely with cable industry group CableLabs to get its BTX media gateway certified. A higher-density model is being prepped for availability next year.
With the international toll bypass market nearing maturity, Nuera's focus on the cable telephony market makes sense, say Kevin Mitchell, directing analyst at market research firm Infonetics Research.
"There's lots of talk about triple play," where service providers offer packages of telephony, cable TV and Internet, he says. "VoIP is the best way to go down that path."
Nuera's close relationship with VoIP service provider Net2Phone is key, Mitchell says, as Net2Phone is increasingly looking to help cable companies get into VoIP and is using Nuera gear to do so.
IBM releases WebSphere Portal Collaboration Center
NewsThread

redpajama_set_name: RedPajamaCommonCrawl


text:
17108 Patricorbett è un asteroide della fascia principale. Scoperto nel 1999, presenta un'orbita caratterizzata da un semiasse maggiore pari a 2,8919171 UA e da un'eccentricità di 0,0275469, inclinata di 2,24912° rispetto all'eclittica.

Collegamenti esterni

Corpi celesti scoperti nel 1999
Patricorbett

redpajama_set_name: RedPajamaWikipedia


text:
Гега́м Га́гикович Кадимя́н (;  19 жовтня 1992, Арташат, Вірменія) — український та вірменський футболіст, нападник клубу «Неман» та збірної Вірменії.

Клубна кар'єра
Вихованець єреванського «Пюніка». Перший тренер — Серж Саргсян. У віці 14 років переїхав у Харків, де продовжив займатися футболом у школі харківського «Арсенала». Пізніше був запрошений армянським «Титаном», де відіграв 4 сезони.

Улітку 2013 року перейшов у ПФК «Суми». У перших 13 матчах, виходячи на заміни, забив 6 голів.

Через півроку перейшов в ужгородську «Говерлу». У чемпіонаті зіграв один раз проти харківського «Металіста».

Улітку 2014 року став гравцем «Олімпіка» (Донецьк). Дебютував у матчі проти «Зорі» (Луганськ). 9 грудня покинув клуб по закінченню контракту.

У лютому 2016 року приєднався до львівських «Карпат», але вже взимку 2016/17 залишив команду за обопільною згодою.

16 січня 2017 року офіційно став гравцем луганської «Зорі», проте в команді виходив вкрай рідко і по завершенні сезону перейшов у «Ворсклу».

У лютому 2019 року перейшов в «Арсенал-Київ».

Кар'єра у збірній
У березні 2016 року був викликаний до лав національної збірної Вірменії, у складі якої дебютував 25 березня в товариському матчі зі збірною Білорусі.

Примітки

Посилання

Уродженці Арташата
Українські футболісти
Вірменські футболісти
Гравці збірної Вірменії з футболу
Футболісти «Арсенала» (Харків)
Футболісти «Титана» (Армянськ)
Футболісти «Сум»
Футболісти «Говерли»
Футболісти «Олімпіка» (Донецьк)
Футболісти «Карпат» (Львів)
Футболісти «Зорі» (Луганськ)
Футболісти «Ворскли»
Футболісти «Арсенала» (Київ)


I'm a laid back girl trying to make it through this thing called life. I'm taking it one day at a time and making the best of every moment keeping my eyes wide open along the way and not taking a second for granted.
I was born in Hamburg, Germany moved around a lot growing up - spending half the year in Germany and the other half in the US - after finishing my education in Germany I finally found my home in NYC. I love traveling, as much as spending time at home with my family.
I stumbled upon Triathlon in 2009. I have no back ground in swimming, biking and until the summer of 2009 I had never run a mile. I started this journey with no expectations but along the way have found myself and a true passion.
After an amazing couple of years of racing and a success I could have never dreamed of my health threw me some curveballs. 2015 taught me to never take a single day for granted and forced me to take a step back from racing. I have overcome those hurdles had some good races in 2016 and was back in Kona 2017. 2018 was an off year for me with some family health struggles but those too are behind me and now I'm more hungry that ever to chase some big dreams and some big goals which include clocking the fasted bike split at Ironman Hamburg next year and getting back to the Big Island.


Ethiopian Culture & Fashion Ethiopia has a history of more than 3000 years and is home to more than 70 ethnic groups.
Ethiopians are the one who offer coffe for the world !
Kaldis the first Ethiopian who test Coffe !!
And Still Christian We Trust In Amighty God !
N.B If your computer didn't get The Amaric or Power Geez Soft ware this writing may not display or come as pices of boxes.
Hey Getche, i really admire you for all that you've done. Its great.But i think you should try to correct some of the spelling mistakes,especially the once below the pictures.


text:
Sixty-Something: The Quest For Adventure
T.J. Clemente
tjclemente@hotmail.com
| 2 Comments · Print Article
All my life I have had a quest for adventure. I just love exploration and enjoy discovering new things. It was during Cub-Boy Scouts that I met the three men who shaped my whole life's quest for adventure: Edward Giamo, Bob Hemm and Jack Brandreth. Only Bob Hemm is still alive. Recently, I was able to tell him just how much he influenced my life, from some talks he made to Boy Scout Troop 1, perhaps 55 years ago!
Edward Giamo Jr. was a brilliant real estate lawyer/baron, but when I met him he was my friend Bobby's older brother, and Edward was still attending Georgetown University. Edward had worked at Philmont Scout Ranch in New Mexico. He told me stories of climbing The Tooth of Time Mountain along with tales of encountering the bears, the snakes, and coyotes in the park. By the time I was 15-years-old, I went to Philmont and stood on the peak of The Tooth of Time, and had a bear incident of my own.
Bob Hemm is the father of a friend of mine, Drew Hemm. Bob is without a doubt a man who was years ahead of everyone. He even drove BMW's in the mid 1960's, when no one had them; yet, the Hemm's had three: a 1600, a 2002, and the luxury Bavaria model. Bob had a Beta Max camera system in the late 1960's. It would be 15 years before I even saw another. But it was Bob who talked of parachute jumps, flying his own plane, and gliders to a very attentive Troop 1, in Pelham New York. In all honesty, my own father had his own plane in 1969, and did soaring in gliders too, but as to parachuting he once explained he spent his time in the 8th Air Force during World War II attempting not to be forced to parachute out of his B-17 bomber. As he put it, "I am never going to jump out of a perfectly operating aircraft." I also know he witnessed fellow servicemen being shot by Germans as they parachuted out of crashing planes.
When I was in the Cub Scouts, my friend Lee Brandreth's mother, Bonnie Brandreth, was our den mother that first year. I never met her husband/Lee's dad until Lee and I were freshman at G.W.U., and he drove us from Atlanta to Washington D.C. after Thanksgiving weekend in 1971. During that 8-hour drive he told stories of attempting to climb the Matterhorn, hiking in the Alps, plus the fun of owning the James Bond model Aston Martin DB III. He was also on Admiral Chester Nimitz's staff during World War II. It was that day that I decided mountain climbing in the Alps was going to be for me. Lastly I have to add, that a woman, who was at dinner with Lee, his dad, and myself when we stopped on that drive in Richmond, commented that, "Jack Brandreth is the best looking man, I have ever seen!" No one has ever said that about me.
Now, at 65-years-old looking back, I realize it was Edward Giamo's stories that led me to visit and camp out in over 20 National Parks and climb the Tooth of Time in New Mexico. It was Bob Hemm who was responsible for my 1977 purchase of a BMW 320i in 1977 that I drove all over the northeast. It cost only $6775 back then, 4 years later the cost of a BMW 320i was $32,000. Throughout my life it was Jack Brandreth who was responsible for teaching me to think outside the box. It was his advice I used when purchasing the best outdoor gear, which in the end was very wise.
Lastly, I must also say thank you to Jack Brandreth for my adventures in the Alps. I went back 23 out of the next 25 years until my divorce and its ramifications ended that streak. (The other two years I missed were the years my daughters were born.) The end result was a successful summit of Mt. Blanc in the mid 90's, the tallest Mountain in the Alps. As I watched the sunrise over Western Europe from that summit I looked at the Matterhorn off in the distance and thought of Jack and his many attempts to summit that noble mountain. In later years, I skied the Matterhorn, and on a separate occasion, I took my mom and dad to Zermatt and my kids to the Italian side of the Matterhorn, Cervinia.
As of now, I have had the opportunity to do long hikes of the Appalachian trial in nine states. Along with my daughters, I did all the huts in the White Mountains. I also did ice climbs alone and with hikes in the Adirondacks. Mount Blanc traverses Italy, France and Switzerland, and I love that region, and even took my wife Cindi there this last March. I never attempted to climb the Matterhorn, because as a father of young daughters, I felt that mountain, with its brittle rock that led to a dozen deaths per year, was way too risky.
Now as a sixty-something, I sail Gardiner's Bay with my quest for adventure being an attempt to produce the best sailing sunset photo that day. However, still when I sit in my big bedroom chair, I still dream of climbing upward to see views I have never seen before.
Leave Your Legacy At South Fork Natural History Museum
Jerome Lawerence And Robert E. Lee's Inherit The Wind At Southampton Cultural Center
The Custer Institute Announces Fall Events
Guest (Tom Creekmore)	from Millsboro, Delaware	says::
I flew with the 8th Air Force in WW II and would like to talk with Bob Hemm. My number is 302 549 1116.
Jan 2, 2018 9:13 pm
Guest (Joanna ormond)	from Charleston, SC	says::
Great article. Always look forward to the insightful and entertaining articles from TJ Clemente. 


text:
Robert Nadon Ph.D.
Robert Nadon began his career as an experimental psychologist and transitioned to statistical analysis of high-throughput biology in the 1990s as Director of Informatics at Imaging Research (in St. Catharines, Ontario). While there, he led a team of statisticians, programmers, and biologists to develop novel statistical algorithms and software for microarray analysis. Among the first to champion currently broadly-accepted formal statistical procedures for gene expression microarrays, he developed the first commercially available software package for analysis of microarray data (ArrayStat). More recently, he conceived and spearheaded the development of the popular FlexArray software package.
Since joining McGill faculty in 2002, Dr. Nadon has given over 50 invited scientific presentations and workshops on high-throughput data analysis. He has been PI on three Génome Québec funded grants on high-throughput methods in genomics and a grant on high-throughput screening funded by FQRNT. He is currently an Associate Editor with BMC Bioinformatics and a panel member on the NSERC Genes, Cells and Molecules Evaluation Group. He has established ties and has forged collaborations with colleagues in France; he was a visiting researcher at the Commissariat à l'Énergie Atomique (CEA) in Grenoble, France (2009-2010) and took the McGill lead toward establishing formal links between McGill and l'Université de la Méditerranée in Marseille. He has consulted internationally for biotechnology and pharmaceutical companies at all stages of research projects for both traditional and high-throughput research studies, from initial concept to final data analysis. A recurrent consultation task is to determine and correct underlying causes of underperformance for ongoing research programs.
Dr. Nadon's primary research interests during the last 10 years have been the development of data analysis approaches for high-throughput data for the following biotechnology applications:
High-throughput screening (HTS) of small molecule and RNAi data
Genome-wide mRNA translation
Image-based high content screening (HCS)
Microarray gene expression
His research activities include:
Novel algorithm development and statistical modeling
Study design and data analysis protocol development
Collaborator on biological and medical research studies
Software development:
LPEadj : R package for correction of the local pooled error (LPE) method to replace the asymptotic variance adjustment with an unbiased adjustment based on sample size
FlexArray : User-friendly Windows software for microarray analysis (now maintained by Genome Quebec)
anota : R package for analysis of differential translation in polysome microarray or ribosome-profiling data sets
HTS Corrector : User-friendly Windows software for the analysis of high-throughput screening (HTS) data (primary developer, Vladimir Makarenkov, Université du Québec à Montréal)
Intensity quantile estimation and mapping (IQEM) : MATLAB code for the correction of image non-uniformity bias in high-content screening (HCS) dataCurrent research projects include developing statistical algorithms for analysis of Nuclear Magnetic Resonance (NMR) metabolomics data; assessing inter-laboratory reproducibility among five high-throughput screening (HTS) centres in France; HTS normalization and study design; development of software package for HTS analysis; high-content screening (HCS) statistical algorithms of image morphological measurements; RNAi study design and analysis.
Carl Murie McGill Ph.D. in Human Genetics
Aixiang Jiang McGill Ph.D. in Biostatistics; co-supervisor, James Hanley (McGill)
Elika Garg M.Sc. in Human Genetics
Abdulaziz Al Suwailem M.Sc. in Human Genetics
Iurie Caraus Ph.D. in Computer Science; primary supervisor, Vladimir Makarenkov (Université du Québec à Montréal), Geeta Bhambhani (B.Sc.), Alvin Kuate Defo (B.Sc.)
Jennifer Button M.A. Research Assistant


Designed by Kirk Pacenti - the famous and much-sought-after MTB fork crown for a 1" steerer. With 74mm inside width (just over 2.9"), there's plenty of room for many MTB tires, fenders, and mud. Biplane style.
214g - Weight is based on a representative sample. Actual weight may vary slightly. Imported.


Colin Thompson's book, The Paradise Garden, is about a little boy named Peter who is driven crazy by the urban noise all around him.
"Only in one place was there any peace. In the west of the city was a fabulous garden. There, behind its tall walls and thick trees, it was possible to escape the noise. For Peter, who had lived all his life in narrow streets, it was the closest place to paradise he had ever seen."
Peter decides to live in the park one summer but as summers will, that one came to a close and by autumn, Peter was lonely and he knew that it was time to go home. The story ends with Peter planting his own paradise garden in his back yard.
On the first day of the garden bloggers fling in San Francisco, our bus drove a short distance from our hotel (10 minutes) and stopped in front of an apartment building. Street, sidewalk, building edifice with no sign of green anywhere. Here's a picture from google maps.
Were we stopping to pick up a late arrival? Flat tire? No on both counts. We followed instructions to get off the bus and follow our leader. We dutifully folowed through a convolution of stairs and corridors which seemed to get smaller and darker as we went. There was one glimpse of daylight.
We were all a little relieved to find at the other end a garden, not just any garden, but a paradise garden in a courtyard created by the arrangement of several older apartment buildings, just like the one in Thompson's book. This magical place is the creation and home garden of Organic Mechanics James Pettigrew and Sean Stout.
A used votive candle holder, found in a church by OM sits on the left of the door at the end of the hall. I already know that this will be my kind of garden!
You'll see many images of this fabulous walk on the blogs of fling attendees as it was a big hit with the group and it's way cool!
Beautiful high brick walls enclose the garden giving it a sense of security and sanctuary from the bustle of the outside world. There are many wonderful plants and objects to be discovered in this garden. I've a feeling that we could spend several days here and we'd still be finding treasure.
A sunny spot is home to an interesting collection of drought tolerant plants.
So much to love in this garden. I'm thinking that a mantle like this might rot in my garden. I'll have to find one and see!
The plants were all so fabulous. The fragrance of the brushed foliage of this tagetes (yup, that tall yellow flowered thing is a marigold which is hardy here) is a delight.
Looking back toward the door to the left through which we originally came.
Makes you wonder what they'll pull of their hat next.
The bunny on the left is embarrassed by my bad attempt at humor.
So many reasons to adore this garden!
I wonder what Vicki's husband will think of her "singing duets" with a 25 year old named Simon? Oh well, what happens at the fling stays at the fling, right?
Here are our hosts, the Organic Mechanics themselves, Sean Stout and James Pettigrew.
I felt the same way, lucky and happy with this garden alone. To all of our delights, there were many more equally wonderful garden tours to follow!
I wish I could have come to SF..I grow up in the East Bay and lived for a while in SF. Didn't take a blind bit of notice to anything green.
It would have been great to have you there! I'd no idea that you grew up in that area - how lucky for you but also sad that you weren't interested in plants then as it's a horticultural paradise!
My reaction to this garden was much the same as yours. What a gem. A true oasis. Your pictures are wonderful and a great reminder of a special time.
It's a pleasure to see everyone else's posts about the fling gardens for the similarities and differences in what we noticed, liked, etc. and as you said they are great reminders of a special time! Glad that you liked this garden too!
I loved this garden too, it was so fabulous. You got some wonderful pictures too. And I loved the opening references to a book about a little boy named Peter. I would love to have a walkway like theirs in my own garden, but don't have a clue how to go about it.
Also, thanks for the advice about how to deal with my Hypericum. I didn't realize it was rust. I'll try cutting it back and cleaning it up thoroughly and see what happens.
Glad you liked the garden and the post. As for the walkway, first, I imagine that we'd need to start collecting cool stuff that would work in a pathway like that. Bedrock is a great source for broken pavers as are headstone places. Jones glass and used materials in Tacoma used to be a good source for unusual stuff. It's still there but I've not been back in a while. Perhaps we should go junking some day?
That hypericum has been fairly susceptible to rust for me so that's the only reason I knew anything about it.
I think this is my favorite write up so far on this garden. What a wonderful space!
Thanks Heather! This was an amazing space and reminded me a little of the Tales of the City books for some reason.
This is the best post on this garden yet! Of course I totally loved your intro, what a great story. I'm really enjoying seeing everyone's Fling posts, it's like reliving the tours all over again.
Aw, gosh, thanks Loree. I'm sure there will be many even better ones to come. I agree, seeing everyone else's posts keeps the memory fresh. Glad that some posts will come later to help lengthen the enjoyment!
I can't wait to see your post about this garden! It was a great place as were all of the gardens we were lucky enough to visit! What a fun time even if we roasted a little!
Wonderful intro to the post on an amazing city garden. The mystery walk was pretty good too since I found it a bit creepy even though I knew where you were going from Alison's post.
With all the treasures in this one garden it will be fun to see how many the other bloggers found as well. It would be fun to see you and Alison junking for a day. I've been collecting stuff for a path, it takes a while.
Can't wait to see the rest of your SF posts.
Glad you liked the post! This was a great place and I know I missed a lot but it was a treat to be there! I've been collecting stuff too including a manhole cover left by the previous owners of the house. I think it's still up in the attic. I love doing the SF posts because they bring back such fond memories.
Such an extraordinary garden. Not quite my type, but it looks great. It's wonderful that there are as many different gardens as gardeners and they all look great!
It's so true! Each garden has a personality and life of its own and each is wonderful is its own way! Happy weekend Satu!
It's so fun seeing everyone's take on these gardens...so interesting to see things you noticed that I totally didn't catch!
I agree! Interesting where other eyes focused when looking at these gardens!
Indeed! The path is pretty darned special! Note to self - start collecting stuff now!
If I lived in an apartment building, this is the kind of garden I would love to have in the courtyard. It's interesting that lots of people were drawn to take pictures of that clock and the votive candle holder.
There was something interesting about the play of time and ideas in that garden and those two objects seem to sum that feeling up most clearly. You mentioned in your post something about time standing still in that garden. The votive holder is there but has no glass candle holders in it. Is it for our own thoughts to ignite the flame of inspiration? Icons from several cultures, the head, broken on top - an open mind perhaps. The path - interesting reuse of discarded objects or the piecing together of something beautiful from the trash that life sometimes throws at us. While this garden may just be an interesting assemblage of objects and plants, one could also read a lot of meaning into the various pieces. This is the magic of art for me that the artist sends out a statement, a feeling, longing, whatever and it can be interpreted and enjoyed in as many ways as there are observers. Ain't it grand!
Glad you liked this garden! Your boyfriend's niece might also like the book Dangerously Ever After.
Lots of imagination at play here. I've been seeing quite a few paths and patios lately that are put together like a patchwork, but this one even has a jigsaw puzzle piece incorporated into the design.
This garden was lots of fun!
I just love your perspective on this incredible garden! Wish I'd had a chance to talk with you more. Next time, for sure!
Thanks Linda! This was my first fling and it was marvelous meeting so many great garden bloggers like yourself and seeing so many incredible gardens. With so much sensory stimulation, it was difficult to focus on anything or anyone for very long! I look forward to seeing you next year in Portland!
A Day in Portland Part Two: Name That Garden!
Exciting Developments in the Outlaw Garden.