text:
\section{Introduction}
{\it Introduction}.---  Several issues point out toward potential inconsistencies of General Relativity at ultraviolet and infrared scales. Specifically, while this theory perfectly works at solar system scale, shortcomings appear at quantum, astrophysical and cosmological levels. Quantum Gravity is the main conundrum as General Relativity cannot be dealt within standard Quantum Field Theory (QFT). On the other hand, we need dark energy and dark matter to achieve a self-consistent picture for the future standard model of cosmology. Up to now, no new particle beyond the current Standard Model of particles has been detected, while a huge amount of missing matter and a cosmic speed up are needed to fit the observed dynamics.

These issues can be softened by taking into account alternative theories of gravity, in particular scalar-tensor and higher-order gravity \cite{fujii:2003fi,*woodard:2007ln,*defelice:2010lr,*sotiriou:2010rp,*capozziello:2011pr,*clifton:2012fk,*capozziello:2015sc} --- that are semi-classical models where additional degrees of freedom are introduced, hence enlarging the dynamics of General Relativity. The method consists in adding minimally and/or non-minimally scalar fields and/or higher-order curvature invariant and see whether or not it helps to soften the issues of General Relativity. The presence of these terms can also be justified by perturbative QFT since matter-gravity interactions on perturbatively curved spacetimes result in such corrections \cite{birrell:1984bk}.

From the infrared perspective, one of the goals is to address the dark side with these additional gravitational degrees of freedom \cite{capozziello:2008gr, capozziello:2012ap,nojiri:2004pl,*bertolami:2007pd,*harko:2008pl,*bertolami:2008uq,*sotiriou:2008cq,*defelice:2010lr,*nojiri:2011pr,*bertolami:2012fl,*wang:2012cq,*tamanini:2013pr,*harko:2014ga}. Indeed these theories can be re-expressed effectively as the theory of Einstein plus additional effective terms entering the right hand side of the metric field equation. Therefore, in some sense, they can be considered as effective source terms like standard matter. For instance, a commonly considered straightforward extension of General Relativity is $f(R)$ gravity where the curvature term in the action is an algebraic function of the undifferentiated Ricci scalar, while the matter part is left unchanged.

However, because of the current tight constraints one has on gravity \cite{will:2014lr}, any proposed alternative theory of gravity must be such that it behaves like General Relativity in most situations \cite{damour:1993uq,*damour:1993kx,*khoury:2004uq,*khoury:2004fk,*khoury:2010ax,*defelice:2010lr,*sotiriou:2010rp,*capozziello:2011pr,*brax:2013cq}. Driven by this observation, and having in mind solar system experiments and constraints in particular, Minazzoli and Hees \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax} recently proposed another way to allow some scalar-tensor theories to satisfy experimental and observational constraints. Indeed, with a specific scalar-matter coupling scalar-tensor theories reduce to General Relativity in weak pressure regimes. For that reason the specific scalar-field(s) associated with this kind of theory has been dubbed ``\textit{pressuron(s)}'' \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax}.

On the other side, there is a well-known equivalence between $f(R)$ theories and a sub-class of scalar-tensor theories when the matter Lagrangian is minimally coupled to the gravitational field(s). Hence, one can expect a similar equivalence between a pressuron theory and a $f(R)$ theory with non-minimal gravity-matter coupling. Such non-minimal couplings have recently been investigated, notably in \cite{nojiri:2004pl,*bertolami:2007pd,*harko:2008pl,*bertolami:2008uq,*sotiriou:2008cq,*defelice:2010lr,*nojiri:2011pr,*bertolami:2012fl,*wang:2012cq,*tamanini:2013pr,*harko:2014ga}. They have been extended to $f(R,\mathcal{L}_m)$ and $f(R,\mathcal{L}_m,\phi,(\partial \phi)^2)$ theories later on \cite{harko:2010ep,*harko:2013rz}.

In what follows we present a simple and elegant $f(R,\mathcal{L}_m)$ theory that turns to be equivalent to a special case of pressuron theories. As we will see, the main surprise comes from the fact that the whole Lagragian is described by only one term where matter and geometry are related by a multiplicative coupling.\\

{\it On an unexpected action form}.---  The action of this theory can be set as  follows
\begin{equation}
S=-\frac{1}{2} \int d^4x \sqrt{-g} \kappa \frac{\mathcal{L}_m^2}{R}, \label{eq:actionfR}
\end{equation}
with $\kappa \equiv 8\pi G/c^4$, $G$ and $c$ are the gravitational constant \footnote{$G$ is different from the effective gravitational constant that is measured in Cavendish experiments. The effective gravitational constant can be deduced in this theory by doing a post-Newtonian expansion of the field equations --- see for instance \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb}.} and the speed of light respectively. One has to stress that action (\ref{eq:actionfR}) is the complete action of the theory.

First of all, note that in this theory, the dynamics can only exist if there is matter (ie. $\mathcal{L}_m \neq 0$). This would suggest that there is a deeper link between space-time and matter than usually assumed. In particular, Mach's principle is intrinsically fulfilled in this theory. This fact will be even more obvious later on after we rewrite the theory in its scalar-tensor form \footnote{Let us also remind that scalar-tensor theories were originally motivated by Mach's principle \cite{brans:1961fk,*brans:2014sc}.}. But consequently, it also means that the theory does not exist in vacuum. Fortunately, QFT tells us that true vacuum does not exist because of 0-point energy. Hence, action (\ref{eq:actionfR}) may actually be well defined everywhere, representing a realistic interplay between matter and geometry. Another thing to point out is that action (\ref{eq:actionfR}) is one of the simplest choices with the correct dimension that involves a multiplicative coupling between curvature and matter rather than an additional coupling.

The metric field equation of the theory writes
\begin{equation}
R_{\mu \nu}-\frac{1}{2} g_{\mu \nu} R= - \frac{R}{\mathcal{L}_m} T_{\mu \nu} + \frac{R^2}{\mathcal{L}_m^2} \left(\nabla_\mu \nabla_\nu - g_{\mu \nu} \Box \right)\frac{\mathcal{L}_m^2}{R^2}, \label{eq:fRmetricfield}
\end{equation}
with the stress-energy tensor defined as follows
\begin{equation}
T_{\mu \nu} \equiv - \frac{2}{\sqrt{-g}} \frac{\delta \left(\sqrt{-g} \mathcal{L}_m \right)}{\delta g^{\mu \nu}}. \label{eq:defSET}
\end{equation}
The trace of the metric field equation reads
\begin{equation}
3 \frac{R^2}{\mathcal{L}_m^2} \Box \frac{\mathcal{L}_m^2}{R^2}=  R-\frac{R}{\mathcal{L}_m} T. \label{eq:tracemetricFE}
\end{equation}
Let us remind that for pressureless perfect fluids, one has $\mathcal{L}_m=-c^2 \rho=T$, \footnote{$\rho=\sum_A (\sqrt{-g} U_A^0)^{-1} m_A \delta^{(3)}(x^\alpha-x_A^\alpha)$, where $ U_A^\alpha \equiv dx^\alpha/c d \tau_A$, $m_A$ is the conserved mass of a particle such that $dm_A/d\tau_A=0$ and $\delta^{(3})$ the Dirac distribution in three dimensions.} where $c^2 \rho$ is the rest mass energy density \cite{harko:2010pr,*minazzoli:2012pr,*minazzoli:2013zl}. Therefore, the right hand side of equation (\ref{eq:tracemetricFE}) is null for pressureless perfect fluids. This is precisely the pressuron mechanism that has been described in \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax}.

It turns out that as in usual $f(R)$ gravity one can re-write the action in an equivalent scalar-tensor theory form. Indeed, defining
\begin{equation}
\sqrt{\Phi}=h=- \kappa \frac{\mathcal{L}_m}{R},\label{eq:defscalar}
\end{equation}
the field equations can be re-written as follows
\begin{eqnarray}
&&R_{\mu \nu}-\frac{1}{2} g_{\mu \nu} R= \frac{\kappa}{h} T_{\mu \nu}+ \frac{1}{h^2}\left(\nabla_\mu \nabla_\nu - g_{\mu \nu} \Box \right) h^2, \label{eq:fEscalarmertric} \\
&&\frac{3}{h^2} \Box h^2= \frac{\kappa}{h} \left(T-\mathcal{L}_m \right).\label{eq:fEscalarscalar}
\end{eqnarray}
But such field equations can be derived from the following effective scalar-tensor theory action
\begin{eqnarray}
S&=&\int  d^4x \sqrt{-g} \left[\frac{\Phi R}{2 \kappa}+ \sqrt{\Phi} \mathcal{L}_m \right] \nonumber\\
&=&\int  d^4x \sqrt{-g} \left[\frac{h^2 R}{2 \kappa}+ h \mathcal{L}_m \right]. \label{eq:actionscalar}
\end{eqnarray}
Such an action is the one of a pressuron without kinetic terms (ie. $\omega(\Phi)=0$) \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax} \footnote{One can see that starting from action (\ref{eq:actionscalar}), one recovers the field equations (\ref{eq:fRmetricfield}-\ref{eq:tracemetricFE}) by inserting (\ref{eq:fEscalarscalar}) into (\ref{eq:fEscalarmertric}) and then re-expressing the scalar field equation without $h$. Hence actions (\ref{eq:actionfR}) and (\ref{eq:actionscalar}) are indeed totally equivalent as long as true vacuum energy does exist.}. But let us recall that even for $\omega=0$, pressuron theories satisfy the strong constraints one has from solar system experiments \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax}. For instance the post-Newtonian parameter $\gamma$ is exactly equal to one, just as in General Relativity \cite{minazzoli:2013fv}. In general, regardless the value of $\omega$, pressuron theories reduce to the General Relativity phenomenology in weak pressure regimes such as in the solar system or during the late cosmic period \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax}. Indeed, for a barotropic perfect fluid, the on-shell Lagrangian is minus the total energy density ($\mathcal{L}_m = - \epsilon$) \cite{harko:2010pr,*minazzoli:2012pr,*minazzoli:2013zl} and therefore equation (\ref{eq:fEscalarscalar}) reduces to \cite{minazzoli:2013fk,*minazzoli:2014ao,*minazzoli:2014pb,minazzoli:2015ax}
\begin{equation}
\frac{1}{h^2} \Box h^2= \kappa \frac{ P}{h},
\end{equation}
where $P$ is the barotropic pressure of the fluid \cite{harko:2010pr,*minazzoli:2012pr,*minazzoli:2013zl}, such that the scalar-field source disappears for $P=0$ --- in accordance with the dust case previously discussed. Another way to see the equivalence of actions (\ref{eq:actionfR}) and (\ref{eq:actionscalar}) goes as follows: using equation (\ref{eq:defscalar}), one can write
\begin{equation}
-\frac{1}{2} \kappa \frac{\mathcal{L}_m^2}{R}= -\frac{a+b}{2} \kappa \frac{\mathcal{L}_m^2}{R}=\frac{a}{2} h \mathcal{L}_m-\frac{b}{2} h^2 \frac{R}{\kappa},
\end{equation}
with $a$ and $b$ two constants such that $a+b=1$. Then from the definition of the stress-energy tensor (\ref{eq:defSET}) and in order to get the appropriate normalization of the material Lagrangian, one necessarily has $a=2$ \footnote{$a=2$ in order to recover for instance the fact that in the dust case limit one has $S_m=-c\int m h d\tau$. One can note that obviously, total inertia of massive particles depends on $h$ in this model, which depends on the content of the universe, therefore satisfying Mach's principle.}, therefore implying $b=-1$. Hence, one gets

\begin{eqnarray}
-\frac{1}{2} \kappa \frac{\mathcal{L}_m^2}{R}= \frac{1}{2} h^2 \frac{R}{\kappa}+ h \mathcal{L}_m. \label{eq:equivalence}
\end{eqnarray}
At the same time this choice for $a$ and $b$ is unique in the sense that the variation of $h$ with respect to the metric cancels for the two terms in this action.
This actually allows to treat $h$ as a fundamental scalar field, while it is a priori dependent on the metric, and establishes the equivalence to the pressuron
with $\omega=0$.
\\
However there is a strong difference with usual pressuron theories. Indeed, in this theory there cannot be any exact vacuum solutions as the theory is not even defined in a vacuum configuration \footnote{According to \cite{pais:1982bk} (p.287), in 1917 Einstein himself believed ``that the correct equations [for gravity] should have no solutions at all in the absence of matter''. This is because Einstein believed in Mach's principle --- that states roughly speaking that ``total inertia of a mass point is an effect due to the presence of all other masses, due to a sort of interaction with the latter'' --- while vacuum solutions would mean that a test particle's inertia would be defined even without any other form of matter, therefore in contradiction with Mach's principle \cite{pais:1982bk}.} (see action (\ref{eq:actionfR})). However, as in General Relativity with a cosmological constant, vacuum solutions may be good approximations in some situations. But in general, one has to consider the contribution of the vacuum energy in the field equations from the very definition of the theory, even though the issue of its nonphysical \footnote{\textit{Nonphysical} in the sense that it seems to be different from what is observed.} value derived from the usual QFT pertubative technics has not been settled yet \cite{bianchi:2010na}. Hence, part of the stress-energy tensor (\ref{eq:defSET}) must come from vacuum energy, otherwise the theory would not even be defined. \\

{\it Conclusion}.--- In this communication, we presented a $f(R,\mathcal{L}_m)$ theory of gravity with a very unusual action that reduces to General Relativity in pressure-less regimes. Because matter and curvature no-longer couple additively in the action but multiplicatively (\ref{eq:actionfR}), curvature and matter are even more intrinsically related than in General Relativity, such that it seems that dynamics do not exist without matter. This fulfills Einstein's initial proposal of having a theory of gravity that satisfies Mach's principle \cite{pais:1982bk}.

But let us stress that it comes from a change of paradigm. Indeed, curvature and matter are usually considered separately before their mutual effects are simply added up in the action. But here, curvature and matter are related multiplicatively in the action from the start. Therefore, in the present theory matter cannot be considered without curvature and vice versa. Hence, in some sense, one can see this theory as a unified theory of matter and geometry. As a consequence, the Planck mass and all particle masses are proportional in this theory (see equation (\ref{eq:equivalence})) \cite{minazzoli:2015ax}. However, it also means that the the 0-point energy value issue is even more pressing in this theory as the theory seems to be not even defined without considering such vacuum energy.

Otherwise, we showed that this theory reduces to a special case of the so-called scalar-tensor pressuron theory. This relation may help in order to study it in various regimes (eg. cosmological, black hole physics etc.). In particular, thanks to the equivalence between the actions (\ref{eq:actionfR}) and (\ref{eq:actionscalar}), one can use results from the literature related to theories with multiplicative scalar-matter coupling (see for instance \cite{shifman:1978pb,*taylor:1988pb,*ellis:1989pb,*damour:1994uq,*damour:1994fk,*gasperini:2002kx,*damour:2002ys,*uzan:2011vn,*hees:2014pr,*hees:2015gg} and references therein). In any case, a lot of work is still needed in order to figure out whether or not the particular action (\ref{eq:actionfR}) is suitable to describe actual space-time dynamics --- down to the quantum level.


text:
\section{Introduction}

Process Mining is a recently emerging discipline that links data analysis with process management. One of the central tasks of Process Mining is \textbf{Process Discovery}, where knowledge is extracted from event logs (readily available in today's information systems) in order to discover real processes \cite{pm}. Several tools, techniques, and algorithms have been proposed to discover business process models from event logs. The quality of discovered models and the efficiency of discovery techniques vary according to several criteria including: fitness, precision, generalization and simplicity.

This document is our submission to the Process Discovery Contest organized by the IEEE CIS Task Force on Process Mining\footnote{\url{http://www.win.tue.nl/ieeetfpm/doku.php}}, and which is collocated with the BPM Conference in Rio de Janeiro in September 2016.

The contest is dedicated to the assessment of tools and techniques that discover business process models from event logs. The objective is to compare the efficiency of techniques to discover process models that provide a proper balance between ``overfitting" and ``underfitting". A process model is overfitting (the event log) if it is too restrictive, disallowing behavior which is part of the underlying process.
Conversely, it is underfitting (the reality) if it is not restrictive enough, allowing behavior which is not part of the underlying process.

Within the Process-Discovery contest, 10 event logs are provided to the contestants as a training set. These event logs are generated from business process models that show different behavioral characteristics. The process models are not disclosed to the contestants.
The contestants should use the training event logs to discover process models that are the closest to the original process models, in term of balancing between ``overfitting" and ``underfitting".
To assess this balance, the contest has a classification perspective where a test event log will be used.
The test event log contains traces representing real process behavior and traces representing behavior not related to the process.
A model is as good in balancing ``overfitting" and ``underfitting" as it is able to correctly classify the traces in the ``test" event log:
\begin{itemize}
	\item Given a trace representing real process behavior, the model should classify it as allowed.
	\item Given a trace representing a behavior not related to the process, the model should classify it as disallowed.
\end{itemize}

With a classification view, the winner is the contestant who can classify correct the largest number of traces in all the test event logs. Moreover, in order to help the contestants tune their discovered models, two ``validation" sets are made available in April and May 2016.

In our submission to the contest, we discovered process models using two techniques: Inductive Miner \cite{indc} and Decomposition based discovery \cite{decmp1, decmp2, decmp3}. We chose these techniques because they guarantee rediscoverability; that is, models discovered using those techniques \emph{fit} (generate) all the traces in their original training event logs.
We used ProM tool (\url{http://www.promtools.org/}) to apply those techniques in discovering our submitted models.

In the reminder of this report, we first show in Section \ref{sec:2} how do we use two discovery techniques, namely: \emph{Inductive Miner} and \emph{Decomposition}, to discover process models from the training set using ProM tool. Second, we present in Section \ref{sec:3} how we \emph{replay} an event log on a process model. Then, we show how we use replay results:
\begin{enumerate}
	\item to check the rediscoverability of models (Section \ref{sec:31}), and
	\item to classify unseen traces (in validation and/or test logs) as fitting or not (Section \ref{sec:class}).
\end{enumerate}
Section \ref{sec:discussion} is dedicated for a discussion about the classification results of April and May validation logs, the complexity of discovered models, and how these two points affect our selection of models for submission.
In the end of the report, we show the pictures of our submitted models.

\section{Process Discovery}
\label{sec:2}

We used two well known discovery techniques: Inductive Miner \cite{indc} and Decomposed Process Mining \cite{decmp1, decmp2, decmp3}. We used those techniques to discover process models for each one of the 10 training event logs. However, since only one process model should be submitted for every event log, we decided to include in our submission a mix of those models.

Rationally, we have chosen those techniques because they guarantee a level of \emph{\textbf{rediscoverability}}, namely the ability of discovering again the models used to generate the traces of the event log. We consider that \textbf{rediscoverability} property is essential to accomplish the task of correctly classify new unseen traces.

The models for processes 1, 2, 4, 8 and 9 are discovered using \textbf{Inductive Miner}. We used ProM Lite \footnote{\url{http://www.promtools.org/doku.php?id=promlite}} for this group of models. Other models - for processes 3, 5, 6, 7, and 10 - are discovered using \textbf{Decomposition} technique. We used ProM 6.5.1 \footnote{\url{http://www.promtools.org/doku.php?id=prom651}} for this group of models. The justification of this selection of process models is given in Section \ref{sec:discussion}.

In the reminder of this section, we describe the step-by-step procedure to discover process models using the above mentioned techniques.

\subsection{Discovery using Inductive Miner}
Inductive Miner \cite{indc} is a discovery approach to construct a Process Tree for a given log.
The main advantages of this approach are:
\begin{itemize}
	\item All discovered models correspond to sound, block-structured workflow (WF) net systems.
	\item The model always fits, i.e., the model can generate the traces in the log.
\end{itemize}
The approach works recursively with \emph{divide and conquer} strategy: split log, construct part of process tree, and proceed with handling split parts of log separately. A process tree can be directly transformed into a Petri Net.
However, ProM has a plugin that can mine a Petri net directly using Inductive Miner technique.

To discover a model (Petri net) from an event log with Inductive Miner technique, we used ProM Lite\footnote{The Inductive Miner plugin is also available in ProM 6.5.1. However, we get different models when we used Inductive Miner in ProM Lite and ProM 6.5.1 (apparently, due to implementation discrepancies). Therefore, we would emphasize that we used \textbf{ProM Lite} for the submitted models that are discovered using Inductive Miner (as mentioned above: models 1, 2, 4, 8, and 9).} and followed the following steps:

\begin{enumerate}
	\item First, the log (in \textbf{XES} format) is loaded onto ProM and used.
Then, in \emph{Actions} window, the plugin ``\textbf{Mine Petri net with Inductive Miner}" by Leemans is selected (see Figure \ref{fig:discv_induc_1}).

\begin{figure}[htbp]
	\centering
		\includegraphics[width=0.5\textwidth]{discv_induc_1.png}
	\caption{Discovery using Inductive Miner: Plugin Selection}
	\label{fig:discv_induc_1}
\end{figure}

	\item A settings page appears then (Figure \ref{fig:discv_induc_2}). For \textbf{Variant}, we select \emph{Inductive Miner}. For \textbf{Event Classifier}, we select \emph{Event Name}.

\begin{figure}[htbp]
	\centering
		\includegraphics[width=0.5\textwidth]{discv_induc_2.png}
	\caption{Discovery using Inductive Miner: Settings Selection}
	\label{fig:discv_induc_2}
\end{figure}

\item Finally, we get a Petri net discovered using Inductive Miner. The discovered model can be visualized using \emph{Graphviz Petri net visualization} (Figure \ref{fig:discv_induc_3}).

\begin{figure}[htbp]
	\centering
		\includegraphics[width=0.5\textwidth]{discv_induc_3.png}
	\caption{Discovery using Inductive Miner: Model Visualization}
	\label{fig:discv_induc_3}
\end{figure}

\end{enumerate}

\subsection{Discovery using Decomposition}
Decomposition \cite{decmp1, decmp2, decmp3} is a process mining technique that discovers an accepting Petri net\footnote{Accepting Petri net is a labeled Petri net with an initial marking and a collection of final markings} from a log using decomposition.
First, the log will be split into a number of sublogs. Second, a subnet will be discovered (using the given discovery algorithm) for every sublog.
Third, these subnets will be merged into a single accepting Petri net. Fourth, known reduction techniques will be applied on this accepting Petri net, and the result is returned.

To discover a model from an event log using Decomposition technique, we used ProM 6.5.1
(this technique is not available in ProM Lite).

\begin{enumerate}
	\item First, the log (in \textbf{XES} format) is loaded onto ProM and used.
Then, in \emph{Actions} window, the plugin ``\textbf{Discovery using Decomposition}" is selected (see Figure \ref{fig:discv_decomp_1}). Actually, there are two available plugins in ProM 6.5.1 for discovery using Decomposition: an \textbf{interactive} plugin that allows for custom settings, and a \textbf{batch} one that uses some \emph{default} settings. We used the \textbf{batch} plugin, hence default settings are used; for instance, the underlying discovery algorithm is ILP Miner \cite{werf}\cite{decmp2}.

\begin{figure}[htbp]
	\centering
		\includegraphics[width=0.5\textwidth]{discv_decomp_1.png}
	\caption{Discovery using Decomposition: Plugin Selection}
	\label{fig:discv_decomp_1}
\end{figure}

\item We get an accepting Petri net discovered using Decomposition. It can be visualized using different visualizations such as \emph{Visualize Accepting Petri Net (Dot)} (Figure \ref{fig:discv_decomp_2}).

\begin{figure}[htbp]
	\centering
		\includegraphics[width=0.5\textwidth]{discv_decomp_2.png}
	\caption{Discovery using Decomposition: Model Visualization}
	\label{fig:discv_decomp_2}
\end{figure}

\item Finally, we convert the discovered accepting Petri net into a `regular' Petri net using \textbf{Unpack Accepting Petri Net} plugin.

\end{enumerate}

Our models are all regular Petri nets that are exported as PNML files.

\section{Replaying Semantics}
\label{sec:3}

To replay an event log on a process model, we used the ProM plugin \textbf{Replay a Log on Petri Net for Conformance Analysis} (available in both ProM Lite and ProM 6.5.1, with identical results obtained).

We need \textbf{replay} for two tasks:

\begin{enumerate}
	\item Verify that the discovered model can generate its originating event log (rediscoverability).
	\item Classify new unseen traces (in validation and test datasets) as fitting or not-fitting with respect to the discovered models.
\end{enumerate}

To replay a log on a process model:

\begin{enumerate}
	\item First, in \emph{Workspace} page, both the log and the model (Petri net) are selected and used.
	\item Second, in \emph{Actions} page, the plugin \textbf{Replay a Log on Petri Net for Conformance Analysis} is selected (Figure \ref{fig:replay_1}).

\begin{figure}[ht]
	\centering
		\includegraphics[width=0.5\textwidth]{replay_1.png}
	\caption{Replay Semantics: Plugin Selection}
	\label{fig:replay_1}
\end{figure}

\item Then, a dialog appears that allow for mapping between the transitions of the Petri net and the event classes of the log (Figure \ref{fig:replay_2}). We need only to choose \textbf{Event Name} as classifier; thus each transition is matched with the event class having the same name.

\begin{figure}[ht]
	\centering
		\includegraphics[width=0.5\textwidth]{replay_2.png}
	\caption{Replay Semantics: Mapping Petri Net to Log}
	\label{fig:replay_2}
\end{figure}

\item The next dialog allows us to select a replay algorithm. We use the following settings (as shown in Figure \ref{fig:replay_4}):

\begin{itemize}
	\item \textbf{The purpose of replay} -- measuring fitness.
	\item \textbf{Penalize improper completion}: yes.
	\item \textbf{Suggested algorithm} -- A* Cost-based Fitness Express, assuming at most 32767 tokens in each place.
\end{itemize}

\begin{figure}[ht]
	\centering
		\includegraphics[width=0.5\textwidth]{replay_4.png}
	\caption{Replay Semantics: Algorithm Selection}
	\label{fig:replay_4}
\end{figure}

\item The last page of the dialog is for selecting the costs of particular deviation (Figure \ref{fig:replay_5}). Again, we keep the defaults: the cost is always 1 except for \emph{tau} transitions where the cost is 0.

\begin{figure}[ht]
	\centering
		\includegraphics[width=0.5\textwidth]{replay_5.png}
	\caption{Replay Semantics: Deviation Cost Selection}
	\label{fig:replay_5}
\end{figure}

\end{enumerate}

Those are the required steps to replay a log on a process model. As mentioned above, we use those steps for rediscoverability checking and for classifying unseen traces.

\subsection{Rediscoverability Checking}
\label{sec:31}

Our choice of discovery techniques is mainly based on the rediscoverability property. That is, we chose Inductive Miner and Decomposition techniques because they both guarantee this property. Therefore, once we discover a model using one of those techniques, as discussed in Section \ref{sec:2}, we replay the originating/training event log on the discovered model to make sure that all the traces in the log \textbf{fit} properly in the model.

To do so, we follow the replay steps mentioned above using the training log and the discovered model. The result of the replay can be visualized in different ways.
For instance, using \textbf{Model Projected with Alignments} choice, the model can be visualized with alignments on top of it, as shown in Figure \ref{fig:replay_6}.

\begin{figure}[ht]
	\centering
		\includegraphics[width=0.5\textwidth]{replay_6.png}
	\caption{Replay Semantics: Rediscoverability Checking}
	\label{fig:replay_6}
\end{figure}

In this visualization, we can take a look at the \textbf{\emph{Global Statistics}} within the inspector window. Among the displayed statistics, we are mainly interested in three statistics: \textbf{Trace} Fitness, \textbf{Move-Model} Fitness, and \textbf{Move-Log} Fitness. The values of those statistics give an indicator about the rediscoverability property.
When those global statistics are all equal to 1, this means that all the traces fit in the model; thus the model is able to rediscover the event log.
Otherwise, if anyone of those statistics has a value less than 1, then there are some traces in the log that don't fit in the model.

In our submission, all the models perfectly satisfy the rediscoverability property.

\subsection{Classification of Unseen Traces}
\label{sec:class}

In the context of the Process Discovery Contest, process discovery is turned into a classification task with a training set and a test set. Moreover, two validation sets are made available to the contestants to allow them tune their discovered models. A process model needs to decide whether traces are fitting or not.

For this classification task, a test or validation log is replayed on the discovered model.
Again, we used the ProM plugin \textbf{Replay a Log on Petri Net for Conformance Analysis} according to the steps mentioned earlier.
When the replay results are obtained, we visualize them using \textbf{Project Alignment to Log} choice.
This will display log-model alignments for each case/trace, as shown in Figure \ref{fig:replay_results}.

For each trace, a table of statistics is shown to the left; and a colored picture of the trace alignment is shown to the right. The meaning of colors is illustrated in a legend shown on the right-top of the screen (Figure \ref{fig:legend}).

Among the displayed statistics, we are interested in: Trace fitness, Move-Log fitness, and Move-Model fitness; and we use them in the classification task.
We classify each trace as \emph{fitting} if, and only if, those three values are perfect (equal to 1). Otherwise, when anyone of those values is less than 1, then we classify the trace as \emph{not-fitting}.

For example, table \ref{tab:replay1} shows those values for replaying Log-1 of April validation dataset over our model-1 (discovered using Inductive Miner).

\begin{table}[htb]
	\centering
	 \scriptsize
		\begin{tabularx}{1\textwidth}{|l *{20}{|Y}|}
\hline
Case ID    & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 & 17 & 18 & 19 & 20 \\
\hline
Move-Model & 1 & 1 & .73 & 1 & .85 & .64 & .83 & 1 & .75 & .92 & 1 & .67 & .82 & 1 & .92 & 1 & 1 & .79 & 1 & 1 \\
Trace      & 1 & 1 & .73 & 1 & .87 & .67 & .83 & 1 & .74 & .79 & 1 & .65 & .82 & 1 & .88 & 1 & 1 & .63 & 1 & 1 \\
Move-Log   & 1 & 1 & .73 & 1 & .92 & .70 & .83 & 1 & .75 & .71 & 1 & .67 & .82 & 1 & .85 & 1 & 1 & .58 & 1 & 1 \\
\hline
Classification & $+$ & $+$ & $-$ & $+$ & $-$ & $-$ & $-$ & $+$ & $-$ & $-$ & $+$ & $-$ & $-$ & $+$ & $-$ & $+$ & $+$ & $-$ & $+$ & $+$ \\
\hline
	\end{tabularx}
	\caption{Results of replaying Log-1 of April validation dataset over the model discovered using Inductive Miner;
	Classification: $+$: fitting, $-$: non-fitting}
	\label{tab:replay1}
\end{table}

In the same sense, the colored picture of the trace alignment can be also used to determine fitness. If the picture consists of \textbf{green} and \textbf{gray} alignments \textbf{only}, then the trace is classified as \emph{fitting}. Otherwise, if the picture also contains other colors (like pink and/or yellow), then the trace is classified as \emph{not-fitting}.

\begin{figure}[htb]
	\centering
		\includegraphics[width=0.76\textwidth]{replay_results.png}
	\caption{Replay results visualized using \textbf{Project Alignment to Log}}
	\label{fig:replay_results}
\end{figure}

\begin{figure}[htb]
	\centering
		\includegraphics[width=0.32\textwidth]{legend.png}
	\caption{Alignment Legend}
	\label{fig:legend}
\end{figure}

\section{Discussion}
\label{sec:discussion}

In our solution to the Process Discovery Contest, we used Inductive Miner and Decomposition techniques - as presented in Section \ref{sec:2} - to discover process models\footnote{The pictures of process models of both groups are available at: \url{https://github.com/rajighawi/Process_Discovery_Contest_2016}} from the 10 \emph{training} logs.
Both groups of models are then used to replay again the training logs - as presented in Section \ref{sec:31}- to ensure \emph{rediscoverability}. We found that all our process models \emph{satisfy the rediscoverability} property.

Then, we replayed April and May \emph{validation} event logs over both groups of process models - processes discovered using Inductive Miner and processes discovered using Decomposition. The replay results are then used to classify the traces of the validation logs as fitting or non-fitting - as presented in section \ref{sec:class}. We communicated the obtained classifications\footnote{The detailed classifications are available at: \url{https://github.com/rajighawi/Process_Discovery_Contest_2016}} with the contest organizers, who replied by indicating how many traces have been correctly classified for each one of the 10 logs. Table \ref{tab:april_results} shows the classification results of April validation logs using the two groups of process models: Inductive Miner and Decomposition. Analogue classification results of May logs are shown in Table \ref{tab:may_results}.

\begin{table}[h]
	\centering
	\small
		\begin{tabularx}{\textwidth}{ L{3.4cm}|YYYYYYYYYY|Y}
 Log $\rightarrow$& 1 & 2 & 3 & 4  & 5  & 6  & 7  & 8  & 9  & 10 &  \\
		\hline
		\multicolumn{12}{c}{Inductive Miner Process Models} \\
		\hline
True Positive  & 10 & 10 & 10 & 10 & 10 & 10  & 10 & 10 & 10 & 10 & \\
False Positive & 0  & 0  & 8  & 0  & 1  & 3  & 0  & 5  & 0  & 0  & \\
True Negative  & 10 & 10 & 2  & 10 & 9  & 7  & 10 & 5  & 10 & 10 & \\
False Negative & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & \\
  \hline
Correctly Classified & 20 & 20 & 12 & 20 & 19 & 17 & 20 & 15 & 20 & 20 & \textbf{183}		\\
		\hline
		\hline
		\multicolumn{12}{c}{Decomposition Process Models} \\
		\hline
True Positive  & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & \\
False Positive & 0  & 0  & 1  & 0  & 0  & 1  & 0  & 6  & 0  & 0  & \\
True Negative  & 10 & 10 & 9  & 10 & 10 & 9  & 10 & 4  & 10 & 10 & \\
False Negative & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & \\
  \hline
Correctly Classified & 20 & 20 & 19 & 20 & 20 & 19 & 20 & 14 & 20 & 20 & \textbf{192}		\\
		\end{tabularx}
	\caption{Classification results of \textbf{April} set using Inductive Miner and Decomposition.}
	\label{tab:april_results}
\end{table}

\begin{table}[h]
	\centering
	\small
		\begin{tabularx}{\textwidth}{ L{3.4cm}|YYYYYYYYYY|Y}
 Log $\rightarrow$& 1 & 2 & 3 & 4  & 5  & 6  & 7  & 8  & 9  & 10 &  \\
		\hline
		\multicolumn{12}{c}{Inductive Miner Process Models} \\
		\hline
True Positive  & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & \\
False Positive & 0  & 0  & 9  & 0  & 1  & 4  & 3  & 2  & 0  & 2  & \\
True Negative  & 10 & 10 & 1  & 10 & 9  & 6  & 7  & 8  & 10 & 8  & \\
False Negative & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & \\
  \hline
Correctly Classified & 20 & 20 & 11 & 20 & 19 & 16 & 17 & 18 & 20 & 18 & \textbf{179}		\\
		\hline
		\hline
		\multicolumn{12}{c}{Decomposition Process Models} \\
		\hline
True Positive  & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & 10 & \\
False Positive & 0  & 0  & 0  & 0  & 1  & 0  & 1  & 4  & 0  & 0  & \\
True Negative  & 10 & 10 & 10 & 10 & 9  & 10 & 9  & 6  & 10 & 10 & \\
False Negative & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & \\
  \hline
Correctly Classified & 20 & 20 & 20 & 20 & 19 & 20 & 19 & 16 & 20 & 20 & \textbf{194}		\\

		\end{tabularx}
	\caption{Classification results of \textbf{May} set using Inductive Miner and Decomposition.}
	\label{tab:may_results}
\end{table}

A quick look at these classification results tells that Decomposition models classify better than Inductive Miner models. That is, out of 200 traces in April logs, 192 are correctly classified using Decomposition models versus 183 using Inductive Miner models. Similarly, in May logs, 194 are correctly classified using Decomposition models versus 179 using Inductive Miner models

However, with a more careful examination of the classification results, we made the following observations:
\begin{itemize}
	\item Logs 1, 2, 4, and 9 are always classified correctly: 20 out of 20 traces are classified correctly.
	\item Log 8 has a better classification using Inductive Miner model than using Decomposition model (April: $15>14$, May: $18>16$).
	\item Other logs - namely 3, 5, 6, 7, and 10 - has a better classification using Decomposition models than using Inductive Miner models.
\end{itemize}

Now, based on these observations, we should decide which models shall we include in our submission to the contest\footnote{As mentioned earlier, only one model per log must be submitted.}. From the second and third observation, we can easily decide to consider Inductive Miner model for log: 8, and Decomposition models for logs: 3, 5, 6, 7 and 10.

However, the first observation tells us that neither one of the two groups of process models has a preference over the other with respect to the classification of logs: 1, 2, 4 and 9. Therefore, we have to consider another quality criteria of process models, namely: \emph{simplicity}.

Actually, the Inductive Miner models are generally more \emph{simple} than Decomposition ones. One can clearly see this by looking at the pictures of both groups of models (see Appendix \ref{apx:models}). However, in order to prove this claim we computed different complexity metrics of process models \cite{lassen, blum}. Table \ref{tab:metrics} shows the complexity metrics for both Inductive Miner and Decomposition process models. These metrics are: \emph{ECaM}: Extended Cardoso Metric, \emph{ECyM}: Extended Cyclomatic Metric, $|E|$ and $|V|$: number of Edges and Nodes in the reachability graph, $|A|$, $|P|$ and $|T|$: number of Arcs, Places, and Transitions in the Net, and \emph{SM}: Structuredness Metric.

Based on the values listed in Table \ref{tab:metrics} we can see that the Inductive Miner models are, in general, \emph{simpler} than Decomposition ones. For instance, the average structuredness of Decomposition models is roughly one order of magnitude higher than for Inductive Miner models.

\begin{table}[ht]
\centering
\small
\begin{tabular}{c|c|c|c|cc|ccc|c}
Model & Density
& \emph{ECaM}
& \emph{ECyM}
& $|E|$ & $|V|$
& $|A|$ & $|P|$ & $|T|$
& \emph{SM}  \\
\hline
\multicolumn{10}{c}{Inductive Miner Process Models} \\
\hline
1 & 0.0387 & 35 & 177  & 185   & 78   & 72 & 31 & 30 & 293   \\
2 & 0.0589 & 20 & 67   & 85    & 27   & 70 & 18 & 33 & 6180  \\
3 & 0.0795 & 18 & 51   & 65    & 20   & 62 & 13 & 30 & 1550  \\
4 & 0.0376 & 31 & 174  & 174   & 77   & 70 & 31 & 30 & 212   \\
5 & 0.0499 & 24 & 215  & 247   & 50   & 90 & 22 & 41 & 29205 \\
6 & 0.1038 & 12 & 23   & 31    & 10   & 54 & 10 & 26 & $ ^{**}$\\
7 & 0.0586 & 22 & 3730 & 4082  & 482  & 82 & 20 & 35 & 4160  \\
8 & 0.0536 & 24 & 159  & 183   & 56   & 72 & 21 & 32 & 740   \\
9 & 0.034  & 35 & 142  & 142   & 72   & 74 & 34 & 32 & 242   \\
10 & 0.0322& 45 & 9418 & 10001 & 1852 & 98 & 39 & 39 & 16435 \\
\hline
Avg.& 0.0547& 26.6 & 1415.6 & 1519.5 & 272.4 & 74.4 & 23.9 & 32.8 & 6557.44 \\
\hline
\hline
\multicolumn{10}{c}{Decomposition Process Models} \\
\hline
1 & 0.0722 & 39  & 129   & 133   & 57    &  78 & 27 & 20 & 32000 \\
2 & 0.2325 & 120 &$ ^{*}$&$ ^{*}$&$ ^{*}$& 266 & 26 & 22 & 72270 \\
3 & 0.2167 & 156 &$ ^{*}$&$ ^{*}$&$ ^{*}$& 312 & 30 & 24 & 93240 \\
4 & 0.0745 & 40  & 862   & 900   & 282   & 93  & 26 & 24 & 79200 \\
5 & 0.088  & 80  &$ ^{*}$&$ ^{*}$&$ ^{*}$& 169 & 32 & 30 & 48600 \\
6 & 0.262  & 133 &$ ^{*}$&$ ^{*}$&$ ^{*}$& 283 & 27 & 20 & 23700 \\
7 & 0.0821 & 58  & 9501  & 10003 & 2140  & 128 & 26 & 30 & 65700 \\
8 & 0.5132 & 149 &$ ^{*}$&$ ^{*}$&$ ^{*}$& 349 & 17 & 20 & 31300 \\
9 & 0.0741 & 41  & 403   & 418   & 154   & 88  & 27 & 22 & 13455 \\
10 & 0.1705& 94  &$ ^{*}$&$ ^{*}$&$ ^{*}$& 180 & 24 & 22 & 89100 \\
\hline
Avg.& 0.1786& 91 & 2723.75 & 2863.5 & 658.25 & 194.6 & 26.2 & 23.4 & 54856.5 \\
\hline
\end{tabular}
\caption{Complexity Metrics for Our Process Models. $ ^{*}$: Reachability graph is unbounded. $ ^{**}$: Net is not a workflow net.}
\label{tab:metrics}
\end{table}

Now, given that for logs: 1, 2, 4 and 9, there is no preference of a particular group of models based on classification results; and given that Inductive Miner models are simpler than Decomposition models, we decided to consider Inductive Miner models for these logs, based on \emph{Occam's Razor} principle,

In summary, our submitted models to the contest are: models 1, 2, 4, 8 and 9 of Inductive Miner models, and models 3, 5, 6, 7 and 10 of Decomposition models. The pictures of these process models are depicted in Appendix \ref{apx:models}

Based on this selection of process models, the contest organizers replayed the \emph{June} test logs. The reported classifications are shown in Table \ref{tab:JuneResults}, whereas the classification results are shown in Table \ref{tab:JuneResults2}. Our selection of models correctly classified 192 out of 200 traces. The organizers confirmed later that this submission gave us the \textbf{second} place in the contest among 14 participant teams.

\begin{table}[h]
\small
	\centering
		\begin{tabularx}{\textwidth}{L{1cm}|YY|YYYYYYYYYYYYYYYYYYYY}
 &  &  & \multicolumn{20}{c}{ Trace }\\
Log $\downarrow$& + & - & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16 & 17 & 18 & 19 & 20 \\
\hline
1 & 10 & 10 & $+$ & - & + & + & - & + & - & - & + & + & - & + & + & - & + & - & + & - & - & - \\
2 & 12 & 8  & - & - & + & + & - & \cellcolor{Gray}+ & + & + & - & \cellcolor{Gray}+ & + & - & + & + & - & - & + & + & + & - \\
3 & 9  & 11 & + & \cellcolor{Gray}- & + & - & - & - & + & + & + & + & + & - & - & - & - & - & + & - & + & - \\
4 & 10 & 10 & - & + & - & + & + & + & - & + & - & - & + & + & - & - & - & + & + & - & + & - \\
5 & 10 & 10 & - & + & - & - & - & - & - & - & + & - & + & + & + & - & + & + & + & + & - & + \\
6 & 12 & 8  & - & + & - & + & + & + & + & \cellcolor{Gray}+ & - & - & - & + & \cellcolor{Gray}+ & - & + & - & + & + & + & - \\
7 & 10 & 10 & + & + & - & - & - & + & - & + & + & + & - & + & - & - & - & - & + & + & + & - \\
8 & 13 & 7  & + & - & + & + & + & - & + & + & + & + & \cellcolor{Gray}+ & \cellcolor{Gray}+ & - & + & - & \cellcolor{Gray}+ & - & - & + & - \\
9 & 10 & 10 & + & + & + & + & + & - & - & + & - & + & - & + & - & - & - & + & - & - & - & + \\
10& 10 & 10 & + & + & - & - & - & + & - & + & + & + & + & - & + & + & - & - & + & - & - & - \\

	\end{tabularx}
	\caption{Reported Classifications of \emph{June} Logs. Shaded cells denote incorrectly classified traces.}
	\label{tab:JuneResults}
\end{table}

\begin{table}[h]
\small
	\centering
		\begin{tabularx}{\textwidth}{ L{3.4cm}|YYYYYYYYYY|Y}
 Log $\rightarrow$& 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 &  \\
		\hline
True Positive  & 10 & 10 & 9  & 10 & 10 & 10 & 10 & 10 & 10 & 10 & \\
False Positive & 0  & 2  & 0  & 0  & 0  & 2  & 0  & 3  & 0  & 0  & \\
True Negative  & 10 & 8  & 10 & 10 & 10 & 8  & 10 & 7  & 10 & 10 & \\
False Negative & 0  & 0  & 1  & 0  & 0  & 0  & 0  & 0  & 0  & 0  & \\
  \hline
Correctly Classified & 20 & 18 & 19 & 20 & 20 & 18 & 20 & 17 & 20 & 20 & \textbf{192}		\\
		\end{tabularx}
	\caption{Classification Results of \emph{June} Logs.}
	\label{tab:JuneResults2}
\end{table}

redpajama_set_name: RedPajamaArXiv


text:
\section{A thought experiment}
\label{thought}

Say that we have three speech communities, which we will refer to as Elephant, Ocelot, and Coyote.
Each community has their own language, which is commonly characterized by language technology researchers as ``low-resource'' or ``under-resourced''.
Elephant's language is spoken by around 10 million first language (L1) speakers and millions more second language (L2) speakers, and it has a standard widely accepted orthography.
Ethnologue~\citep{lewis2015ethnologue,campbell2008ethnologue} classifies this language as ``used in education, work, mass media, and government at the national level.''
Ocelot's language has about 500,000 L1 speakers and three different orthographic representations.
This language is noted in Ethnologue as ``in vigorous use, with literature in a standardized form being used by some though not yet widespread or sustainable'', and is described as an ``indigenous'' language.
In comparison, the language of Coyote has only a handful of speakers that could be considered fluent L1 speakers, most of whom are elders.
The language currently lacks a standard orthography, and in Ethnologue it is classified as ``endangered''~\citep{meek2012we}.

Elephant wants to develop spoken language technology primarily to support the use of speech-enabled applications and smartphone features.
Ocelot and Coyote also need spoken language technology but their focus is often on using these technologies to support use of the language by the community through language preservation, documentation, and instruction. Creating chatbots or enabling hands-free cellphone use, while appealing, might not currently be a priority for Ocelot and Coyote.

\subsection{Collecting training data}
With support from local universities and the government, Elephant has started a project to build an automatic speech recognition (ASR) system.
Thus far it has collected around 60 hours of audio data gathered from radio stations as well as recordings produced in a studio.
To transcribe the recordings in the common orthography, Elephant resorted to online crowd-sourcing platforms.
It has gathered additional texts scraped from the web and from newspapers and books; the texts also followed the standard orthography of the language, totaling several million tokens.
All of the preparations required financial resources and time, but they were completed within the course of less than a year.
Elephant's language technology development work has received interest and accordingly collaboration offers from potential industry partners, including a company that makes a popular language learning application.
In return the company has asked for the language data from the project either to be made public or to be owned by the company. Elephant decided to cooperate.

Ocelot wanted to try ASR as well, though it did not know where to start at first and did not at the moment have government support. On the other hand, Ocelot has connections with a professor in Indigenous Studies who has advanced research knowledge of the language and is trusted by the language community.
Over the years the professor has collected around 100 hours of recordings produced by various community members.
Though there was not yet a standard orthography for the language, some agreement was reached among the community members.

Unlike Elephant, Ocelot was not able to use crowd-sourcing platforms since there are not that many people who have literacy in the language.
Fortunately, the professor was able to recruit students from the local university who are L1 speakers of the language to transcribe audio data.
They also managed to gather digitized texts of $\sim$250,000 tokens from available websites and converted them to the same orthography used for the transcriptions.
All of these preparations were finished in a few years.
When the same company that reached out to Elephant made the same offer, Ocelot said she would think about it.

Things were quite different for Coyote. She did not have support from the government or any connections with universities. A professor used to work with elders from the community and collected dozens of hours of recordings from fieldwork sessions throughout the years, but at a certain point he stopped giving the audio data back to the community and there were no additional copies. The few recordings made by anthropologists and linguists in the early 20th century were not yet digitized and remain archived outside the community. In other words there was no available data yet to build speech technology.
When the same company offered to make learning applications in exchange for ownership of the language data, Coyote said no because of their prior negative experiences with outside researchers and the government-funded attempted linguicide~\citep{hinton1994flutes,skutnabb2000linguistic} that occurred in their community just a few generations ago. At this point, many members of the Coyote language are reluctant to work with outsiders, even corporations willing to help them.

Coyote decided to be in charge of the documentation process herself. Right now there is one community member who is an L2 learner and is studying computer science as a student at a local university. She is trying to learn about how to build ASR tools. At the same time, she is working with internal researchers from Coyote, some of whom do not have formal training on the linguistic aspects of their language.
Their data collection process, however,  has turned out to face severe challenges.

First, recordings are gradually obtained from conversations with the elders, who are kindly working for free. In order to be respectful to the elders' schedules and to make sure they get enough rest during the long fieldwork sessions, the audio collections are  ongoing and comparatively much slower than that for the languages of Elephant and Ocelot.

Secondly, the lack of standard orthography for the language creates difficulty in choosing reasonable representations. Although the language has one available grammar book that was written back in the 1920s, there are many words in the recordings that do not appear in the grammar. Additionally, having representative community members come to a consensus on a single orthography requires extensive time and discussion.
Over the decades various linguists and anthropologists have come up with a total of eight different orthographic representations for the language, but having multiple written orthographies has resulted in several possible pronunciations for one word or one single utterance. Thus, a significant amount of deliberation is required in order to ensure that the speakers from Coyote do not have too much difficulty reclaiming words or sentences from written documentation.
Writing is seemingly so common for languages that are widely spoken or studied in the world that people tend to take it for granted; they do not exactly realize the writing is itself a luxury and a form of language technology that is not natural to many oral speech communities~\citep{bird-2020-decolonising,hinton1994flutes,ryon2002cajun,crystal2018}.

Thirdly, the transcribing process is an extremely time-consuming endeavor. Although the graduate student herself is an advanced L2 learner, many of the other learners do not have the same proficiency. In other words, there are very few people from Coyote who are capable of transcribing recordings of their language, commonly referred to as the ``transcription bottleneck"~\citep{zahrer-etal-2020-towards,shi-etal-2021-leveraging}. Therefore the transcriptions have to be cross-checked through consultations with the elders from the Coyote speech community from time to time.
At this pace, over five years, around only 16 hours of recordings for Coyote have been collected and transcribed, among which 6 hours are monolingual narratives and story-telling in the language, while the rest includes a large amount of code-switching with English.
Additional written texts for the language were digitized from the grammar book and an available Bible in the language, yielding around 40,000 tokens.

\subsection{Training the ASR models}
Now each of the three speech communities has some training data in their own language. They are ready to train ASR systems.
Elephant adopted the deep neural network (DNN) from the popular Kaldi toolkit~\citep{povey2011kaldi}. The transcripts of the audio data used as training data and additional written texts were combined to build a language model, which was then applied in the decoding process of the acoustic model. Word error rates (WERs) below 20\% were achieved. Elephant also tried the newer end-to-end neural ASR library, ESPNet~\citep{watanabe2018espnet}, which does not necessarily require language models and has been demonstrated to work well for languages with dozens of hours of audio data (e.g., Yolox{\'o}chitl Mixtec~\citep{shi-etal-2021-leveraging}) or more (e.g. English, Hindi~\citep{khare2021low}); a similarly strong WER was obtained.
Ocelot engaged in the same efforts and she was able to obtain WER numbers comparable to those for Elephant.

In contrast, when applying the same DNN architecture from Kaldi (with different hyperparameters) to her six hours of monolingual audio data, Coyote was able to derive a WER of just under 40\%. Coyote thought that perhaps this had something to do with the language model since it was trained on a very small number of words so she turned to ESPnet, opting for a training scheme without using the language model for decoding. The results were even worse. Similar results were produced using wav2vec-U~\cite{baevski2021unsupervised}. Even the ASR frameworks touted as particularly successful for low-resource languages yielded disappointing results for Coyote's language.

On the other hand, the number 40\% means something different for a community of speakers than it does for the research community. A WER this high might not be that impressive for (academic) researchers, but it may be \textit{good enough} in the meantime; a model trained on the available six hours of audio data can be used to generate transcriptions of new recordings from ongoing fieldwork sessions or untranscribed archival recordings. These transcriptions can then be corrected by L2 speakers, expediting the transcription process and creating new acoustic and textual training data~\citep{prudhommeaux2021}.

For speech communities like Elephant and Ocelot, which are relatively widely-spoken and often benefit from financial support or collaborative bonds with academics and industry partners, it is possible to collect more data, whenever it is needed. For endangered languages like Coyote's, however, it is unlikely that it will ever be possible to gather even dozens of hours of audio data, regardless of financial or time constraints.

\section{Academic perspectives}

\begin{table*}[ht!]
\small
\centering
\begin{tabular}{llll}
\hline
\textbf{Category} & \textbf{Description} & \textbf{Count} & \textbf{Examples}  \\ \hline
Elephant & widely spoken, well supported & 99 (60.7\%) & Bengali, Danish, Igbo, Pashto, Tagalog \\
Ocelot  & fewer speakers, well supported & 39 (23.9\%)  & Faroese, Maori, Quechua, Yiddish \\
Coyote & few speakers, little support  & 25 (15.3\%)  & Bribri, Kodi, Mi'kmaq, Veps, Yine \\    \hline
\end{tabular}
\caption{Number of unique languages named in ACL Anthology abstracts that include the phrases \emph{low resource, under resourced} or \emph{resource constrained language}, organized by low-resource language type. }\label{aclanth}
\end{table*}

As illustrated above, the current research field has coarsely referred both to widely spoken languages lacking an established tradition of natural language processing (NLP) and to endangered indigenous languages as ``low-resource" or ``under-resourced", without acknowledging or mentioning the drastically different conditions of their data availability.
Recent high-profile work on diversity in language technology included, in one case, just a few sentences encouraging researchers to prioritize endangered languages ~\citep{blasi2021systematic}, and in another case, no discussion of endangered languages at all~\citep{mager-etal-2020-tackling}.
While the language taxonomy based on resource availability proposed in~\citet{joshi-etal-2020-state} is quite impressive, the authors simply grouped all languages that are currently lacking resources into the same category. For instance, the Mixtec language (with different varieties), spoken in Mexico, has around half a million L1 speakers, while the Juruna language in Brazil has less than 300 L1 speakers; yet both were categorized together as ``still ignored in the aspect of language technology".

Roughly 1,050 of the nearly 30,000 abstracts available in the ACL Anthology bibliography contain the token phrases ``low resource'', ``under resourced'', or ``resource constrained language''. Of these abstracts, about half name a specific language that has been assigned an ISO-639 code. Excluding obviously high-resource languages that are mentioned in these abstracts as a point of comparison, as a source language for transfer learning, or as a source or target for machine translation, we are left with 163 unique languages (considering their dialectal variations) that have been characterized in at least one ACL abstract as low-resource. Table~\ref{aclanth} shows the distribution of these 163 languages with similar resource conditions/external support as Elephant, Ocelot, or Coyote respectively, along with a few examples of each language category. We see Elephant languages far outnumber both Ocelots and Coyotes, with Coyotes representing only 15\% of the languages identified as low-resource.

There is great variability both in the degree and in the nature of the challenges that arise when developing language technologies for languages with scarce training resources. It is crucial that NLP researchers actively acknowledge this variability and avoid giving the impression that models or architectures developed for Elephant will be suitable for Ocelot or Coyote, or vice versa.
We can begin by distinguishing languages classified as ``endangered'' from those that are not, and provide case-by-case detailed descriptions of the speaker population size and language data availability for the language being investigated.
Contingent on that, it is only recently that the academic community has started holding workshops devoted to endangered and indigenous languages, such as ComputEL~\citep{computel-2021-use}, held four times since 2014, and AmericasNLP~\citep{americasnlp-2021-natural}, which took place for the first time in 2021.
Work published in these and other venues has included research on several NLP tasks that pertain to language documentation and reclamation for endangered languages, from morphological segmentation~\citep{liu-etal-2021-morphological,kann-etal-2018-fortification}, finite-state morphological analyzers~\citep{lane-bird-2020-bootstrapping,lachler-etal-2018-modeling},
to machine translation~\cite{zhang-etal-2020-chren,bird-chiang-2012-machine} and ASR~\citep{thai2020fully,ethan2021,shi-etal-2021-highland}.

That being said, most of the work has focused on technology development, with relatively little regard for the ways in which the development of language technology for endangered languages might be different from that for languages with few existing resources but a much larger numbers of speakers.
Discussions of whether a proposed language technology would be useful for the workflow of the community's own language documentation efforts, or how it would be combined with the community's revitalization and instructional activities are also noticeably lacking, with a few notable exceptions such as the the verb conjugator, Kawenn{\'o}n:nis, developed for the Ohsweken dialect of Kanyen{'}k{\'e}ha~\footnote{https://kawennonnis.ca/about}~\citep{kazantseva-etal-2018-kawennon}; the online dictionary developed for Hupa~\footnote{http://nalc.ucdavis.edu/hupa/hupa-lexicon.php}, which is currently used for language-related activities in the community; and the Indigenous Languages Technology project at {NRC} {C}anada~\cite{kuhn-etal-2020-indigenous}.

The NLP community has formally recognized the importance of developing technologies for endangered languages, and we have the tools to support work in this area. Now we must try to answer this question: what priorities and considerations should researchers take into account when developing technology for endangered languages?

Often NLP technologies presume that a language has a standardized written form that may act as source or target for various computational tasks (e.g., ASR, machine translation, named-entity recognition). While standardization is typical of languages in most of the W.E.I.R.D societies \cite{henrich2010weirdest}, this is atypical for much of the rest of the world. For many endangered language contexts, the tradition of literacy is very recent, and writing is far less privileged than the oral medium.

Sociolinguistic research on small languages has identified significant variation, both dialectal and ideolectal~\citep{skilton2017three}, in these contexts as well. Standardization, frequently considered a first task in language documentation and the development of language technology, often tends to run counter to the goals of the speech communities working towards revitalization~\citep{whaley2011some}. For these communities, linguistic variation is not a problem to be solved but an important element of a vital language ecology.

A major goal in language revitalization involves facilitating the usage of the endangered language in a wider set of contexts and situations than it is being used in currently. This entails careful forethought into how language tools can assist in this broadening of usage. Providing state-of-the-art language technology to a community that is critically involved in training new speakers and developing new contexts for usage may not be the most efficient use of time and resources. Tools that assist in classroom education and in developing new usage situations are likely to be of much more immediate value to groups involved in revitalization.

The number of speakers of a language also impacts the maximum rate at which new data can be collected to add to the resources available for a language. Even for a language with just thousands of speakers, documentation projects can accumulate resources at a pace impossible for a highly-endangered language. In Yolox\'ochitl Mixtec, for instance, with around 5000 speakers, there exists a speech corpus of over 100 hours of running speech \citep{mitra2016automatic}.
For languages with few speakers capable and comfortable of speaking their language, who are mostly elders and also very occupied with other activities related to the revitalization of the language, the goal of collecting long-duration recordings in the language does not seem feasible or even reasonable.

Additionally, while through the decades certain linguistic academic scholars have responsibly and sensitively built trustworthy and collaborative bonds with indigenous communities~\cite{hale1992endangered} (e.g., Dr. Ken Hale working with Warlpiri~\cite{hale1983warlpiri} and Navajo~\cite{remember}; Dr. David Rood working with Lakhota~\cite{rood1996sketch}; Dr. R. M. W. Dixon working with Australian aboriginal languages~\cite{dixon1970proto}), for many endangered language communities in North America, the attitudes of earlier European-American ``settler" scholars towards indigenous communities and their languages have engendered distrust in the motives and biases of outside ``experts'' \citep{harveyNativeTonguesColonialism2015}. Earlier fieldwork often involved outsider linguists paying speaker consultants to participate in research that was designed and conducted solely by the researcher in what is now referred to as the linguist-focused model \cite{czaykowska2009research}. A more recent trend in linguistics is the movement toward Community-Based Language Research, in which community members collaborate with outsider linguists on the research which they themselves help design. In the development of language technology, providing the speech communities a central role in the design and implementation of language tools may improve the likelihood of the tools' success.

\section{Endangered language teachers perspectives}

\begin{table*}[h!]
\small
    \centering
    \begin{tabular}{l|c|c}
    \hline
   \textbf{Question} & \textbf{Yes} & \textbf{No}  \\
 \hline
 Do you think writing could be a technology? & 91.30\% & 8.70\% \\
 Can you read written form(s) of the language? & 95.65\% & 4.35\% \\
 Do you use written documentation when working on language revitalization? & 100\% & -\\
 Do you think morphological segmentation could be a useful technology? & 86.96\% & 13.04\% \\
 Do you think automatic speech recognition could be a useful technology? & 82.61\% & 17.39\% \\
 Do you think video parsing could be a useful technology? & 82.61\% & 17.39\% \\
 Do you think pedagogical learning applications are a useful technology? & 95.65\% & 4.35\% \\\hline

    \end{tabular}
    \caption{Summary statistics of an informal survey on language documentation and technology of language teachers from four endangered language communities of North America.}
    \label{tab:survey_statistics}
\end{table*}

While language technology is certainly of significant value from an academic's point of view, is it actually useful to stakeholders in endangered language communities?
To learn from community voices, we designed an informal survey (Table~\ref{tab:survey_statistics}) and received responses from a total of 23 language teachers coming from four endangered communities. Among them, five are community-designated Master Speakers~\footnote{Master speakers are indigenous community members who are fluent in their language and have accepted apprentices who study the language with them through the oral tradition~\citep{richardson1993now}.} of the language, two of whom are elders; the rest of the respondents include one young L1 speaker, and either semi-fluent or fluent L2 speakers of their languages.

In the survey, we asked for language teachers' thoughts on whether they would consider writing to be a technology, and additionally, whether they think writing, morphological segmentation~\citep{cotterell-etal-2016-joint}, ASR~\citep{jimerson-prudhommeaux-2018-asr}, video processing, and pedagogical learning applications~\citep{bettinson-bird-2017-developing}, which are all common in the research field of NLP, could be useful for them.

Overall, the majority of community language teachers think all of the five technological applications would be helpful but to different extents.
For example, most think that having written documentation would be valuable for aspects of language teaching, reclamation, and intergenerational transmission of cultural and linguistic knowledge. In the words of one respondent, ``Written documentation is useful because a lot of it is old.  It captures the way speakers talked and created words. The writing can be harvested, so that we reclaim our words and speak in the old style again." However, writing has not been shown to be an acceptable \textit{alternative} to learning under a Master Speaker through the oral tradition, which requires no written resources; and learning from writing alone is ``not necessary or sufficient for restoring linguistic and cultural knowledge".\footnote{We acknowledge that different indigenous speech communities have different perspectives on these matters. Writing alone is extremely useful for speech communities with no Master Speakers left. We note particularly the Breath of Life (BOL) speech communities that reclaim their languages from written documentation; the written documentation helps BOL community researchers become the next generation of Master Speakers. In other words, writing is not the end goal in the language learning process for these indigenous communities.}

With morphological segmentation, most language teachers stated that knowing the morphological structures of words would be informative for them to learn to ``piece together the meaning of phrases".  ``Before I attended language pods I had gone to some community language classes, but living far away a lot of my language learning came through reading the dictionary. I thought that having learned the morphologies of words because of how the dictionary shows those word parts was extremely helpful for me in order to not only learn those words but also be able to figure out how to create other ones without checking the dictionary."

In the case of ASR, some language teachers thought that it would be ``an interesting idea"; they could see themselves ``reviewing the transcripts" and the technology would help with ``sound recognition and pronunciation."; these transcripts, however, would not be as effective as ``listening to the audio". Others expressed strong feelings against ASR, saying that ``To me this is just a way for linguists to secure funding for themselves and their tech project, which takes money and resources away from speech communities.  This kind of thing is not language revitalization, as it doesn't create new speakers.  It generates new texts so they can sit on the shelf of some archive. Not helpful".

Almost all language teachers favor automatic processing of video materials. They mentioned this technology is of great value because ``(it) captures the  authenticity of the Speaker/Apprentice"; others said they would use videos ``to watch the elders' mouths when they speak as well as their facial expressions. Intonations
and body expression really add to conversations. Those things can be lost when just reading, writing or
even listening. Seeing the facial expressions and body language are important to understand the contexts
of how specific words are being used".

At last, regarding pedagogical applications, community members suggested that they could be beneficial given that the applications could ``provide repetition, drill and practice.  This would increase learner confidence in their own adequate exposure to the language (in a non-threatening manner).  It would allow the learner to understand how the language works, so that new speech beyond the app could be developed by the learner". They indicated, however, that for the applications to \textit{work}, people would have to actually ``use them".

The observations from this informal survey might be surprising to researchers, who typically consider language technology to be broadly useful and beneficial to all people.
An awareness that is lacking in the research field, however, is that the purpose of language technologies  and their development process might vary significantly when applied to indigenous and endangered languages. For  many of these languages, the priorities of the speech communities are how to more effectively document, teach, and reclaim their language; how to save the cultural heritage passed down from the elders; and how to let their language have a voice among other widely-spoken or dominant languages

\section{Elder perspectives}
\label{elder}

To linguists, the Karuk speech community is a critically endangered language community of North California.  To speech community members, the Karuk language is a vital language with approximately 25 speakers, including five Master Speakers and other language teachers, archivists and activists.  The Karuk language community is one which is thriving, though surviving through grass roots revitalization with very little infrastructure at the tribal language program level.

When it comes to modern language revitalization, data cannot and should not be separated from the Master Speakers. Their experience of government policies aimed at linguicide~\citep{hinton1994flutes,skutnabb2000linguistic} and their sense of loss of their mother tongue, as L1 speakers become more and more scarce, are realities that new speakers and field linguists need to acknowledge.  For speakers of the Karuk language, these issues come through in the community's internal documentation of their language. For instance, one Master Speaker of the language explained the loss of speakers when he said: ``(t)hose were all my friends.  That's what I was telling [the nurse]. I said I got a lot on my mind.  I said, I sit here all by myself and I'm thinking about all the people that left me.  I said, it's kinda, you don't feel good I mean, you know, when you think about them.  You're not supposed to, they're gone.  Xâatik, let it go.  But I just can't help it.  I think about all the funny things we did together, laughing and talking"~\footnote{All quotes in this paper were initially documented by an author of this paper during her fieldwork sessions.}. His words reflect the culture of language loss as it occurs on the human level.  Sometimes this reality stops potential Master Speakers from working with linguists and young speakers eager to reclaim their language/identity.

It is clear that when working with elderly Master Speakers, methodologies must include space for the elders to vent their grief.  Only after this is attended to, can the language be learned by young language workers striving to create a future where the language is worth more than the losses felt.  The young people and Master Speakers who attend to this re-envisioning of an indigenous future with endangered languages in perpetuity, are the cornerstone of hope and healing~\citep{whalen2016healing,hornberger2014bringing,leonard2011challenging}.

One speaker discusses her view of indigenous second language acquisition (SLA): ``Reading outta them books is OK.  You can probably gather a lot.  Yeah, you guys work hard at it.  I've been trying to get all these people together.  At least once a month all the speakers should work together.  You can just gather one day, all the language speakers, you know?  And start talking the language.  A lot of [speakers] get enough [language] so they can teach... But they're not really getting all of it.  If we're all together, maybe you had something that you wanted to say and you didn't know the words, you could ask somebody.  Somebody would know.  In other words, help one another".  Here we come to understand that linguistic documentation is useful for language reclamation, and writing ``them books'' is a fruitful task~\citep{leonard2017,rigney1999internationalization}.  But one theme that emerges from documentation of elderly Master Speakers of the Karuk language is that writing cannot replace the value of speech communities coming together and speaking their language, continuing the oral tradition.  Written data doesn't save a language; it safeguards knowledge.  The ultimate goal of endangered language communities is to someday house that same knowledge in the hearts and minds of their members.

\section{A case study in bridging the indigenous and academic communities}

What does a healthy and trustworthy workflow look like when bringing together an endangered indigenous language community with the (academic) research community?
After gaining perspectives from academics as well as language teachers and elders from indigenous endangered language communities, here we describe an ongoing workflow devoted to developing a morphological parser for the Cayuga language.

With approximately 50 L1 elder speakers and an ever-growing number of L2 speakers, the Cayuga language fits the description of a highly or critically endangered language. Community-based revitalization projects include an immersion language preschool, adult language courses, as well as teacher-training programs at the local Polytechnic.

Two authors of this paper are participants in this project,
and both are linguists in academia. Specifically, one of them has years of connections with the Cayuga speech community and advanced research knowledge in the language, while the other has extensive training in computational linguistics but no initial connections with the speech community.

This project started in late September, 2021, and is ongoing.
The overall workflow is simple and straightforward. First, the author known by the Cayuga speech community introduced the other author to the community. They described the general idea for the project and mentioned that if they were to carry out the project, they would begin with words already found in the published grammar. While morphological segmentation is of interest to linguists, and morphological supervision has potential utilities for certain NLP tasks such as dependency parsing~\citep{seeker-cetinoglu-2015-graph} and bilingual word alignment~\citep{eyigoz-etal-2013-simultaneous}, in this case, the main goal was to ask whether community members would find morphological segmentation useful for their own language teaching and documentation.
Community members mentioned that explicitly teaching students various inflectional elements of complex verbs, segmenting them, and in some instances color-coding morphemes have been useful for students to learn verbal arguments.

Second, after securing the go-ahead from community members, the two authors have been meeting almost every week for an hour to discuss progress. The author with extensive research knowledge of the language manually performs morphological segmentation of around 50 words every week. In particular, he provides annotations of both \textit{surface segmentation} and \textit{canonical segmentation}~\citep{cotterell-etal-2016-joint}.
The former is to be later incorporated into the workflow for building ASR systems for the language using recordings already collected; the latter has the objective of gaining a better understanding of the language from a linguistic perspective.
The key difference between these two types of segmentation is that for surface segmentation, the concatenation of the individual morphemes stays true to the initial (orthographic) representation of the word (e.g., \textit{onadowa:dǫh} $\rightarrow$ \textit{on-adowa:d-ǫh}; the word means \textit{they are hunting}), whereas for canonical segmentation, decomposing a word into its component morphemes involves the addition and/or deletion of characters (or phonemes) in order to outline the orthographic or phonological changes during the word formation process (e.g., \textit{onadowa:dǫh} $\rightarrow$ \textit{yodi-adowad-ǫh}).

With the new words annotated every week, the author with a computational background trains segmentation models in an iterative fashion, by combining the words of the current week with those from previous weeks to construct a data set for model training and evaluation.
Model performance, indexed by F1 score, is recorded weekly.
As of now we have annotations for 262 words. The F1 scores for both surface and canonical segmentation approximate 50\%.
Our follow-up step is to train models using all these words, then apply them to new data that has not yet been annotated to enhance and accelerate manual annotation. Once the F1 scores reach around 75\%, we plan to report back to the community, inform them about where things are, and discuss details of incorporating our research output with their own language work.

\section{Recommendations for ethical collaborations}

Considering academic output on endangered languages more holistically, we conclude that there are not enough narratives about the process of working with the community.
Academic (NLP) researchers working on indigenous languages, particularly endangered languages like Karuk and Cayuga that have historically been suppressed, should take the following steps when planning projects, describing their research, and collaborating with stakeholders in the speech communities:
\begin{enumerate}[(1)]
\item Make efforts to actually know the indigenous speech communities and build meaningful bonds with them. For instance, fieldwork researchers, when possible, should try to train young community internal researchers to document the language, if they have aspirations to become language teachers; training them can help the speech communities increase longevity and sustainability. Academic researchers should continue to assist the community that they have worked with when Master Speakers are no longer able to participate in language documentation.  Assistance might involve writing dictionaries that would later be given to the community, or helping heritage language learners with language revitalization and reclamation.

\item Consider that speakers and community researchers should be offered the opportunity to be co-authors on work they made meaningful contribution to, and/or be listed as contributors in appropriate ways.

\item Describe the data collection protocols followed and challenges faced in research output. Be attentive and respectful to indigenous community members' schedules, needs (e.g., elders might need to take medications during fieldwork sessions), and perspectives.
\item Speak clearly about plans for the sharing, archiving, and storing of the data~\citep{rigney2006indigenous}. In particular, make sure to be aware that Master Speakers want at the very least co-use copyright over \textbf{all} data which shall be inherited by their descendants.  In addition, physical copies of all data should be given to Master Speakers, and copies should be submitted to tribal archives or archivists.  The only exception to this rule occurs when Master Speakers ask for their data to be edited before being made public to remove gossip or culturally sensitive material before making copies available.
\item Create language technologies together in consultation with speech communities in order to ensure their usefulness to language programs.  The developed technology needs to be accessible to community language workers.
    \item Discuss concrete plans for how technology output can be incorporated into the documentation and revitalization work of the speech communities.
    For instance, a morphological parser needs to visualize morphemes and word construction in such a way as to be a valuable teaching tool for speech community members; ASR systems should not require data extraction or facilitate data ownership by community-external researchers or corporations.

\end{enumerate}

Lastly, each perspective and motivation for indigenous language documentation has value and is worth recording. We hope our work will encourage academics to focus on prioritizing the needs and preferences of endangered speech communities when working with them to develop technologies for their languages.
In particular, academics must keep in mind the relationship many endangered language communities have with their languages. One Master Speaker of the Karuk language captured this nature of this relationship when he said ``(t)he Karuk language is a canoe. It holds all of our baskets, our regalia, our materials, our food. The canoe holds all our practices, songs and stories. It holds all our people and all the Karuk people yet to be born. The canoe carries us all; without it, we can't get anywhere"~\cite{crystal2018}.

\section*{Acknowledgements}

We would like to thank the speech communities of Karuk, Gayog\underline{o}ho:n\k{o}' (Cayuga), Kanienkeha (Mohawk), and Onödowa'ga:' (Seneca). With the Karuk speech community, we are especially grateful to the following honored Elders who have passed away: Junie Donahue, Vina Smith, Charlie Thom, and Sonny Davis; we are also grateful to the current Master Speakers of the community who provided us with their invaluable insights: Julian Lang, Phil Albers Jr., Nancy Steele, and Susan Gehr; lastly, we want to thank the dedicated language teachers of the community: Florrine Super, Tamara Alexander, Lulu Alexander, Jason Hockaday.
Within the Gayog\underline{o}ho:n\k{o}' speech community, we are especially grateful to Gasenneeyoh Crawford, Renae Hill, Amos Keye, and Sose Smith.
This material is based upon work supported by the National Science Foundation under Grant \#2127309 to the Computing Research Association for the CIFellows Project, and Grant \#1761562.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation nor the Computing Research Association.

\section{Ethical consideration}

The data from the informal survey described in this paper were collected from language teachers and speakers from four indigenous speech communities of North America. The survey is language-agnostic in the sense that it could be used and expanded to other indigenous speech communities as well. We hope that the ethical challenges outlined here will motivate other researchers working on language technology to also be aware of the differences between languages in ``low-resource settings" but with much larger speaker populations, and indigenous endangered languages; and to be attentive to the needs of the speech communities of the latter.