\section{Introduction}

Compilers must often emit instruction sequences that accomplish particular data manipulations in the generated code. For example, a compiler may have to generate instructions that swap the contents of two scalar registers prior to an instruction with strict constraints on its register operands. Or, as another example, a compiler may have to emit instructions that broadcast the value in a scalar register to all elements of a vector register in the prologue of a vector loop. Using an efficient instruction sequence for each desired data manipulation reduces the runtime of any application that executes these data manipulations frequently.

Clearly, an optimizing compiler could try to find efficient instruction sequences during actual code generation. Although this possibly provides additional context for optimization, the major drawback of this approach is that search time directly contributes to compile-time during AOT compilation or, worse, runtime during JIT compilation. Alternatively, efficient instruction sequences for desired data manipulations could be searched for earlier, i.e. while the compiler is still being developed. Although this may provide less opportunities to exploit code context, search time is not at a premium in this approach, and an exhaustive state space search to find the best possible instruction sequences becomes possible. Once found, instruction sequences are hard-wired in the code generator and become at the immediate disposal of the compiler.

In this note, we explore using Prolog~\cite{colmerauer93} for such a state space search. To keep the presentation brief, we focus on finding efficient Intel SSE instruction sequences for a few simple SIMD data manipulations. However, the presented ideas easily generalize to other instructions sets and code generation problems.

\section{State Space Search}

Finding an efficient instruction sequence to accomplish a particular data manipulation can be expressed as a {\bf state space search} problem~\cite{luger,nilsson}, with the original contents of memory and registers as \emph{start state}, the machine instructions as \emph{transitions} from one state to another state, and the desired contents of memory and registers as \emph{goal state}. A path from the start state to the goal state provides a solution to the problem. The best solution is given by the shortest path, i.e. the path with minimal length if all transitions have the same cost, or the path with minimal total weight if different transitions have varying costs, such as different cycle counts for the instructions.

\subsection{State Space}

As stated before, for the sake of brevity, we focus on finding efficient Intel SSE instruction sequences for a few simple SIMD data manipulations. Furthermore, to keep the state space size manageable, we focus on just a subset of the SIMD state, data types, and instructions, abstracting away from details related to general-purpose registers and instructions, state flags, memory operands, etc. In this simplified view, the SIMD state is fully defined by eight \texttt{xmm}-registers, represented in Prolog as a list with eight variables (variables start with an uppercase letter).

\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"[ XMM0, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6, XMM7 ]"
\end{tabular}\color{black}\end{flushleft}

Here, each variable can be bound to a Prolog term that represent particular contents, such as a list \texttt{[17.5, 11.9]} to denote a packed double-precision floating-point data type with the given numerical values, an atom \texttt{xmm0} (atoms start with a lowercase letter) to denote a particular but otherwise non-exploitable value, or the anonymous variable \texttt{\_} to denote any term. For example, the following list denotes a SIMD state in which registers 0, 3, and 7 contain packed data types with the given numerical values, registers 1 and 2 have particular but different contents that are not subject to further inspection, and all other registers are undefined.

\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"[ [0,0,0,0], xmm1, xmm2, [8,7,6,5,4,3,2,1], _, _, _, [1.0, 2.5] ]"
\end{tabular}\color{black}\end{flushleft}

In the remainder of the paper, we will just consider \emph{packed dwords} represented by 4-elements lists, with the convention that the higher to lower packed elements appear left-to-right in the list.

\subsection{Transitions}

Each instruction transforms a SIMD state into another SIMD state. These transitions are modeled by a set of Prolog rules for each Intel SSE instruction in our simplified model. Each rule is this set will have the form
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(instruction, op1, op2, S, T)."
\end{tabular}\color{black}\end{flushleft}
to indicate that applying \texttt{instruction} to the given operands transitions from state \texttt{S} to state \texttt{T}. For example, the change in SIMD state by executing instruction
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"pxor xmm0, xmm0"
\end{tabular}\color{black}\end{flushleft}
is modeled by rule show below, which specifies that any contents of register \texttt{xmm0} (the anonymous variable \texttt{\_}) is zeroed out (the list \texttt{[0,0,0,0]}) while the contents of all registers remain unaffected.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(pxor, xmm0, xmm0," \\
\verb"  [         _, XMM1, XMM2, XMM3, XMM4, XMM5, XMM6, XMM7 ]," \\
\verb"  [ [0,0,0,0], XMM1, XMM2, XMM3, XMM4, XMM5, XMM6, XMM7 ])."
\end{tabular}\color{black}\end{flushleft}

Similarly, using Intel syntax, where the destination register appears first, the change in SIMD state after
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"paddd xmm1, xmm7"
\end{tabular}\color{black}\end{flushleft}
is modeled by the following rule, which adds the packed integral elements of one register to the packed integral elements of another register.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(paddd, xmm1, xmm7," \\
\verb"  [ XMM0, [A,B,C,D], XMM2, XMM3, XMM4, XMM5, XMM6, [E,F,G,H] ]," \\
\verb"  [ XMM0, [A+E,B+F,C+G,D+H], XMM2, XMM3, XMM4, XMM5, XMM6, [E,F,G,H] ]," \\
\end{tabular}\color{black}\end{flushleft}

Although this allows Prolog to reason about the instruction symbolically, sometimes we are also interested in evaluating the values using integral arithmetic. To that end, the following rule is added as well, which evaluates expressions in which all values are integers (such rules could be refined further to allow for partial evaluation).
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(paddd, xmm1, xmm7," \\
\verb" [ XMM0, [A,B,C,D], XMM2, XMM3, XMM4, XMM5, XMM6, [E,F,G,H] ]," \\
\verb" [ XMM0, [P,Q,R,S], XMM2, XMM3, XMM4, XMM5, XMM6, [E,F,G,H] ]) :-" \\
\verb" integer(A), integer(E), P is A+E, integer(B), integer(F), Q is B+F," \\
\verb" integer(C), integer(G), R is C+G, integer(D), integer(H), S is D+H."
\end{tabular}\color{black}\end{flushleft}

Similar rules are added for all other arithmetic, logical, comparison, and conversion instructions, and for all combinations of register pairs. A data shuffling instruction such as
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"punpckldq, xmm0, xmm3"
\end{tabular}\color{black}\end{flushleft}
is modeled as shown below.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(punpckldq, xmm0, xmm3," \\
\verb" [ [_,_,A,B], XMM1, XMM2, [X,Y,C,D], XMM4, XMM5, XMM6, XMM7 ]," \\
\verb" [ [C,A,D,B], XMM1, XMM2, [X,Y,C,D], XMM4, XMM5, XMM6, XMM7 ])."
\end{tabular}\color{black}\end{flushleft}

A shift instruction like
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"psrldq xmm2, 4"
\end{tabular}\color{black}\end{flushleft}
is modeled by the rule below.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(psrldq, xmm2, 4," \\
\verb"  [ XMM0, XMM1, [A,B,C,_], XMM3, XMM4, XMM5, XMM6, XMM7 ]," \\
\verb"  [ XMM0, XMM1, [0,A,B,C], XMM3, XMM4, XMM5, XMM6, XMM7 ])."
\end{tabular}\color{black}\end{flushleft}

The SIMD state change after the data movement instruction
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"movd xmm4, I"
\end{tabular}\color{black}\end{flushleft}
is modeled with this rule.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"i(movd, xmm4, I," \\
\verb" [ XMM0, XMM1, XMM2, XMM3,         _, XMM5, XMM6, XMM7 ]," \\
\verb" [ XMM0, XMM1, XMM2, XMM3, [0,0,0,I], XMM5, XMM6, XMM7 ])."
\end{tabular}\color{black}\end{flushleft}

Obviously, writing all these Prolog rules by hand would be too tedious and error-prone. Instead, a utility should be used to generate all rules automatically, preferably directly from an instruction set description in an electronic format. A complete and accurate rule set will obviously yield the best results.

\subsection{Search}

Given a complete Prolog rule set that model the SIMD state transitions of all instructions, we need a search mechanism to find a path in the state space from the \emph{start state} to the \emph{goal state}. This search mechanism is also expressed with Prolog rules.

A first reasonable attempt is shown below (we will refine these rules slightly later). The two rules states that any state \texttt{S} transitions into itself for an empty instruction sequence, or otherwise breaks down into the transition of a single instruction from state \texttt{S} to state \texttt{U} followed by the transition from state \texttt{U} to state \texttt{T} of an subsequent instruction sequence \texttt{J}. The list of 3-arity \texttt{i} predicates built by these rules ultimately indicate an instruction sequence that transitions state \texttt{S} to state \texttt{T}.

\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s(S, [], S)." \\
\verb"s(S, [i(I,R1,R2)|J], T) :- i(I, R1, R2, S, U), s(U, J, T)."
\end{tabular}\color{black}\end{flushleft}

Now suppose we are interested in finding the best way to zeroing the contents of register \texttt{xmm0}. It may be tempting to express that particular state space search problem with the following Prolog query.

\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s(S, I, [[0,0,0,0] | _ ])."
\end{tabular}\color{black}\end{flushleft}
However, this query returns the following first solution, with as interpretation that the shortest way of resetting register \texttt{xmm0} to zero is by executing no instructions at all (empty list \texttt{I}) but instead starting with all zeroes in that register (initial state \texttt{S}). Although correct, this is obviously not what we were searching for.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = []" \\
\verb"S = [[0,0,0,0]|_]"
\end{tabular}\color{black}\end{flushleft}

As a side note, this mistake can demonstrate a potential danger of using anonymous variables. The almost identical query
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s(_, I, [[0,0,0,0] | _ ])."
\end{tabular}\color{black}\end{flushleft}
would have given the solution
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = []"
\end{tabular}\color{black}\end{flushleft}
as well, but without even listing bindings for the two anonymous variables, obscuring the fact that the initial state was bound to a state with the first register already zeroed out. So each anonymous variable really denotes any \emph{suitable} term. Rather, named variables should be preferred when contents matter.

The correct way of formulating the original query is by explicitly stating the fact that all registers contain unusable and unrelated initial values, as shown below with eight different atoms.

\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s([xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7], I," \\
\verb"  [[0,0,0,0] | _ ])."
\end{tabular}\color{black}\end{flushleft}

This query will prompt the following list as a first solution, indicating a single instruction way of zeroing out register \texttt{xmm0}.

\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [i(pxor,xmm0,xmm0)]"
\end{tabular}\color{black}\end{flushleft}

\subsection{Iterative Deepening Search}

Prolog's DFS (depth-first search) is not very suited for this particular kind of state space search problem, since it will continuously append instructions to existing partial solutions in an attempt to reach the goal state. A BFS (breadth-first search) works much better, since it will report the \emph{shortest} instruction sequences from the initial state to the goal state first. We will implement such a search using Prolog's DFS, but without the inherently high memory demands of BFS, using IDS (iterative deepening search). To this end, the search rules given earlier are refined into the following set.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s(S, I, T) :- count(D, 0), s(S, I, T, D)." \\
\verb"s(S, [], S, 0 )." \\
\verb"s(S, [i(I,R1,R2)|J], T, X) :- X > 0, Y is X - 1," \\
\verb"                              i(I, R1, R2, S, U), s(U, J, T, Y)." \\
\end{tabular}\color{black}\end{flushleft}
The search rules themselves are as before, but restricted to a given depth. The \texttt{count} rules define a simple increment mechanism.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"count(X, X)." \\
\verb"count(X, Y) :- Z is Y + 1, count(X, Z)."
\end{tabular}\color{black}\end{flushleft}

Combined, these rules try to find solutions within subsequent instruction sequences of length 0, 1, 2, etc. As a result, shorter instruction sequences are reported first (note that with some effort, this search mechanism can be adapted for other criteria of the \emph{best} solution, such as finding the instruction sequences with the lowest total cycle counts). For example, running the query
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s( [ xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7], I," \\
\verb"   [ [-1,-1,-1,-1] | _ ] )." \\
\end{tabular}\color{black}\end{flushleft}
reports the desired solution
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [i(pcmpeqd,xmm0,xmm0)]"
\end{tabular}\color{black}\end{flushleft}
before it reports the following alternative, but longer solution, which basically just clobbers the register with an unused value before resorting to the shorter solution.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [i(pxor,xmm0,xmm0),i(pcmpeqd,xmm0,xmm0)]"
\end{tabular}\color{black}\end{flushleft}

Suppose we are interested in broadcasting a value to all elements in a SIMD register, an idiom that is frequently used in the prologue of a vector loop by a vectorizing compiler~\cite{bikbook}. An instruction sequence for such a broadcast can be found using the following query, where atom \texttt{c} denotes the value that needs broadcasting.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s( [ xmm0, xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7], I," \\
\verb"   [ [c,c,c,c] | _ ] )."
\end{tabular}\color{black}\end{flushleft}
Lacking a shuffle operation in our simplified rule set, the shortest instruction sequence for the broadcast consists of a data movement instruction followed by two unpack instructions.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [" \\
\verb"     i(movd,xmm0,c)," \\
\verb"     i(punpckldq,xmm0,xmm0)," \\
\verb"     i(punpckldq,xmm0,xmm0)" \\
\verb"    ]"
\end{tabular}\color{black}\end{flushleft}

\subsection{Usable Start State}

The examples so far searched for a particular goal state given an \emph{unusable} state state. Often, however, the start state may contain some known, usable information. As a simple example, the query
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s([xmm0, [1,2,3,4], xmm2, xmm3, xmm4, xmm5, xmm6, xmm7], I," \\
\verb"  [[1,2,3,4] | _ ])."
\end{tabular}\color{black}\end{flushleft}
yields the following solution, which indicates that the best way to assign particular contents to register \texttt{xmm0} given a state where register \texttt{xmm1} already has these contents is simply moving the register.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [i(movdqa,xmm0,xmm1)]"
\end{tabular}\color{black}\end{flushleft}

As a more practical application, this approach can be used to find the best sequence to sum up all elements in a SIMD register "horizontally", an idiom used by a vectorizing compiler~\cite{bikbook} to finalize the computation after converting a sum-reduction loop into SIMD code. Here the query
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"s( [ [a,b,c,d], xmm1, xmm2, xmm3, xmm4, xmm5, xmm6, xmm7], I" \\
\verb"   [ [_,_,_,(d+b)+(c+a)] | _ ] )."
\end{tabular}\color{black}\end{flushleft}
yields the following instruction sequence as first suitable solution.
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [" \\
\verb"     i(movdqa,xmm1,xmm0)," \\
\verb"     i(psrldq,xmm0,8)," \\
\verb"     i(paddd,xmm1,xmm0)," \\
\verb"     i(punpckldq,xmm0,xmm1)," \\
\verb"     i(paddd,xmm0,xmm1)," \\
\verb"     i(psrldq,xmm0,4)" \\
\verb"    ]"
\end{tabular}\color{black}\end{flushleft}
Subsequent solutions with the same length provide some true alternatives (here, cycle counts could help finding the truly best one).
\begin{flushleft}\color{blue}\footnotesize\begin{tabular}{lll}
\verb"I = [" \\
\verb"     i(movdqa,xmm1,xmm0)," \\
\verb"     i(psrldq,xmm0,8)," \\
\verb"     i(paddd,xmm1,xmm0)," \\
\verb"     i(movdqa,xmm0,xmm1)," \\
\verb"     i(psrldq,xmm1,4)," \\
\verb"     i(paddd,xmm0,xmm1)" \\
\verb"    ]"
\end{tabular}\color{black}\end{flushleft}
Other solutions of the same length that follow may simply provide the same instruction sequences using different intermediate registers.

Note that in this example, a \emph{statically} known property of the context in which the instruction sequence is needed allowed for adding some usable information to the start state (viz. the SIMD register contains four partial results that need to be summed up). As stated in the introduction, at runtime the compiler could even exploit some \emph{dynamically} known properties of the context to find better instruction sequences, but it is unlikely that exhaustive search (let alone Prolog) could be used under such circumstances.

\section{Conclusions}

In this note, we explored using Prolog for finding efficient data manipulating instruction sequences. The problem is expressed as a state search problem, with the initial memory and register contents as start state, machine instructions as transitions, and the desired memory and register contents as goal state. Modeling instructions with a complete and accurate Prolog rule set of transitions will yield the best results, where it preferable to extract such a rule set automatically from an instruction set description in an electronic format. Search is expressed with Prolog rules as well, enhanced with iterative deepening to work around obvious complications with the default depth-first search of Prolog.

Once the best solution is found after exhaustively searching the state space, the instruction sequence can be hard-wired in any code generator that needs the data manipulation, and becomes at the immediate disposal of the compiler. Here, the best can be defined as the shortest instruction sequence or, with some adaptation, as the instruction sequence with minimal total weight, such as summing the cycle counts. For the sake of brevity, we restricted our focus on finding efficient Intel SSE instruction sequences using just a subset of the SIMD state, instructions, and data types. However, the presented ideas easily generalize to broader instructions sets and code generation problems.


\section{}

Detached eclipsing binaries present a unique opportunity to measure accurate masses and radii of stars that are effectively in isolation \citep[e.g., ][]{Torres10}. The All-Sky Automated Survey for SuperNovae \citep[ASAS-SN, ][]{Shappee14, Kochanek17} observes the entire sky in optical light in the $V$-band (between 2012 and mid-2018) and $g$-band (since 2018). Although primarily designed for transient phenomena, the $V$-band observations have been used to classify $\sim 426,000$ variables, including $\sim150,000$ eclipsing binaries \citep{Jayasinghe19II, Jayasinghe21IX}. The ongoing ASAS-SN Citizen Science campaign \citep{Christy21} has also already identified a host of new variable stars in the $g$-band data.

Here we characterize the long-period detached eclipsing binary ASASSN-21co (RA=18:17:51.52, DEC$=-$~58:07:49.31). ASASSN-21co was first identified by \citet{Way21} as a long-period eclipsing binary with an eclipse depth of $g\sim0.6$~mag. $V$-band observations from the All-Sky Automated Survey \citep[ASAS, ][]{Pojmanski97} also show a similar eclipse around UT 2009-04-23. We use the {\tt astrobase} \citep{Bhatti20} implementation of the box least squares algorithm to derive an orbital period of $P=4344$~days (11.9 years). The absolute \textit{Gaia} magnitude \citep{GaiaCollaboration18}, corrected for extinction, is $M_G=-1.76$, with a \textit{Gaia} color of $G_{\rm{BP}}-G_{\rm{RP}}=2.23$~mag \citep{Anders19}, suggesting the star is a luminous red giant. The WISE and 2MASS colors, $W_1-W_2=-0.196$~mag and $J-K_s=1.093$~mag \citep{Curti12}, suggest that the photometric primary is an M giant \citep{Li16}. This is consistent with the spectra shown in Figure \ref{fig:lc_spectra}.

Given the orbital period, the binary separation is $a\simeq5.2 ({M_T}/{M_\odot})^{1/3}\ \rm{AU}$, where $M_T$ is the total mass of the binary. The eclipse total duration is $t_T \sim66$~days and the time of total eclipse is $t_F \sim23$~days. If we assume that the impact parameter is zero, the radii of the primary $R_p$ and secondary $R_s$ are

\begin{equation} \label{eqn:radii}
    R_p = \frac{\pi (t_T+t_F)}{2P}a = 36.1\ R_\odot \left(\frac{M_T}{M_\odot}\right)^{1/3}\ \text{and}\ R_s = \frac{\pi (t_T - t_F)}{2P}a = 17.4\ R_\odot \left(\frac{M_T}{M_\odot}\right)^{1/3},
\end{equation}

\noindent The \citet{Anders19} {\tt StarHorse} catalog of \textit{Gaia} DR2 sources gives $M_*=1.06\ M_\odot$. Using \texttt{Isoclassify} \citep{Huber17,Berger20}, we combine the {\tt StarHorse} stellar parameters with MIST tables of bolometric corrections \citep{Dotter16} to find $R_p=73.26\ R_\odot$. Given this radius, Equation \ref{eqn:radii} implies a large total mass $M_T=8.4\ M_{\odot}$, suggesting that the {\tt StarHorse} radius is overestimated.

Figure \ref{fig:lc_spectra} shows the ASAS and ASAS-SN $V$- and $g$-band observations. BVRI photometry taken during the 2021 eclipse shows that the eclipse is chromatic. Whereas the $g$-band magnitude falls by $\sim 0.6$, the $I$ magnitude decreases by only $\sim0.4$. The similarities between the asymmetric eclipse profile and disk eclipsing systems like $\epsilon$ Aurigae \citep{Kloppenborg10}, OGLE-LMC-ECL-11893 \citep{Dong14}, and ASASSN-V J192543.72$+$402619.0 \citep{Jayasinghe18detached} could suggest a dusty disk eclipse. This scenario would also produce a shallower eclipse in redder bands. In the case of a disk eclipse, the lack of secondary eclipse would be expected.

We use {\tt PHOEBE} \citep{Prsa16} to model the $g$- and $I$-band light curves of the most recent eclipse. We find that the eclipse depth and duration is consistent with two M giants with a ratio of radii $R_s/R_p=0.61$, close to our rough estimate of $0.48$ from the eclipse timing, and effective temperatures $T_{\rm{eff},s}/T_{\rm{eff},p}=1.09$. Since the redder bands have shallower eclipse depths, the observed eclipse occurs when the larger, colder primary is behind the smaller, hotter secondary. The eclipses observed in the ASAS and ASAS-SN data have the same duration and depth, while the eclipse structure implies that the secondary eclipse should have roughly half the depth of the primary eclipse, which rules out a period of $2P$. Shorter periods ($P/2$, $P/4$, etc.) have contradictions in their phased light curves because of missing eclipses. Finally, we note that the lack of a secondary eclipse near phase $0.5$ implies that the orbit must be elliptical.

The eclipse profile is asymmetric, with larger variability seen in the ASAS-SN $g$-band data than the BVRI photometry. This could be due to low amplitude rotational variability on the cooler star. Using {\tt Period04} \citep{Lenz04}, we find low amplitude variability in the ASAS-SN $g$-band data at a period of 9.7~days. We find the same periodicity in the TESS sector 13 light curve from the Quick-Look-Pipeline \citep[QLP, ][]{Huang20a, Huang20b} in addition to larger amplitude periodicity at longer periods $\sim 20$~days. Since only one sector of TESS data is available, we cannot accurately constrain the period of this long-term variability. The longer period, higher amplitude variability is consistent with what is expected from overtone pulsators with higher modes \citep{Tabur10, Ferreira15}, while the shorter period may be either overtone pulsations or solar-type oscillations \citep{Mosser13}.

Figure \ref{fig:lc_spectra} shows optical spectra taken from mid-eclipse to the completion of egress. The spectrum is consistent with that of an M giant and we see no temporal changes in the spectral features. The lack of spectral evolution during the eclipse is similar to the $\epsilon$ Aurigae system, whose spectrum does not vary with orbital phase despite the light curve suggesting a total eclipse \citep{Huang65}. The lack of spectral evolution does not rule out the two-giant scenario, as the effective temperature ratio found by the {\tt PHOEBE} model fits would also result in little spectral evolution during eclipse.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{21coLC_spectra.pdf}
    \caption{Top: ASAS and ASAS-SN light curves of 21co. The time of the TESS sector 13 is labeled by the blue arrow. We see a single eclipse in the ASAS $V$-band data and in the ASAS-SN $g$-band observations $\sim 11.9$ years later. Middle: the ASAS-SN $g$-band eclipse with additional BVRI measurements The {\tt PHOEBE} fits to the $g$- and $I$-band data are shown as solid lines. Bottom: scaled and shifted spectra of ASASSN-21co ordered in time from bottom to top. The orbital phase of each spectra is labeled in the middle panel with the corresponding number.}
    \label{fig:lc_spectra}
\end{figure}

\clearpage


\section{Introduction}
End-to-end (E2E) automatic speech recognition (ASR) models have been recently studied to streamline the complicated pipelines in conventional hybrid systems.
Online streaming is one of the most important tasks for real-world applications.
While frame-synchronous models such as connectionist temporal classification (CTC)~\cite{ctc_graves} and RNN transducer (RNN-T)~\cite{rnn_transducer} are natural choices for the streaming purpose, emerging progress of streaming attention-based encoder-decoder (AED) models, such as monotonic chunkwise attention (MoChA)~\cite{mocha}, make them good alternatives as well~\cite{inaguma2021alignment,kim2020review}.

Apart from the decoder topology, all streaming E2E models have a large emission latency because the alignment-unconstrained E2E training objective encourages the model to see as many future observations as possible.
To tackle this problem, various methods have been proposed; alignment path restriction~\cite{sak2015acoustic_asru,inaguma2020streaming,mahadeokar2020alignment}, encoder pre-training with frame-level supervision~\cite{hu2020exploring,inaguma2020streaming,mahadeokar2020alignment}, and a new training objective for direct latency minimization~\cite{inaguma2020streaming,inaguma2020_ctcsync,yu2021fastemit,yu2021dualmode,kim2021reducing}.
Among them, FastEmit~\cite{yu2021fastemit} successfully reduced the emission latency of RNN-T by modifying the loss to prioritize the generation of non-blank labels over blank labels \textit{without external alignment}.
However, the method is specific to RNN-T and cannot be applied to MoChA directly.
This is because MoChA detects token boundaries with a Bernoulli random variable in the internal attention module while RNN-T treats all labels, including a blank label, equally in the output softmax layer.

In this work, we propose a straightforward alignment-free regularization method, \textit{StableEmit}, to reduce the emission latency of MoChA.
StableEmit discounts selection probabilities for token boundary detection in MoChA with a constant factor during training.
Simultaneously, we encourage the expected attention weights calculated from the modified selection probabilities, to sum up to one to recover the attention mass (\textit{quantity regularization}~\cite{inaguma2020_ctcsync}).
Therefore, the confidence score for token emission is enhanced and reaches a threshold to emit a token earlier at test time.
This also reduces deletion errors, which stabilizes the search.

Moreover, it is possible to combine StableEmit with alignment regularization methods such as delay constrained training (DeCoT)~\cite{inaguma2020streaming} and CTC-synchronous training (CTC-ST)~\cite{inaguma2020_ctcsync}.
We further propose DeCoT-CTC, a new alignment path restriction method combining these ideas.

Experimental evaluations with Long Short-Term Memory (LSTM) and causal Conformer~\cite{gulati2020} encoders show that the proposed method significantly reduces the deletion errors and the emission latency of MoChA.
We also demonstrate that the combination with the alignment regularization methods has a complementary effect, leading to further improvements in the accuracy and latency.
The best MoChA model shows comparable performance to RNN-T on long-form speech utterances, which has been difficult to achieve so far.

\section{Background}\label{sec:background}

\subsection{Monotonic chunkwise attention (MoChA)}\label{ssec:mocha}
MoChA is a linear-time attention model on the basis of hard monotonic attention (HMA)~\cite{hard_monotonic_attention}.
To compensate for the performance degradation due to the hard attention to a single encoder output, MoChA introduces an additional soft attention  module restricted to a local chunk of $w$ frames from the boundary detected by HMA.
The test-time decoding complexity is linear, in which the discrete decision $z_{i,j}\in\{0,1\}$ to generate the $i$-th token on the $j$-th encoder output is sampled from a Bernoulli random variable parameterized by a selection probability $p_{i,j}$.

Because this procedure is not differentiable, the expected alignment probabilities $\alpha_{i,j}$ is calculated by summing all possible alignment probabilities during training as
\begin{eqnarray}
\alpha_{i,j} &=& p_{i,j}\sum_{k=1}^{j}\bigg(\alpha_{i-1,k}\prod_{l=k}^{j-1}(1-p_{i,l})\bigg) \nonumber \\
&=& p_{i,j}\bigg((1-p_{i,j-1})\frac{\alpha_{i,j-1}}{p_{i,j-1}}+\alpha_{i-1,j}\bigg). \label{eq:monotonic_attention_alpha}
\end{eqnarray}
While Eq.~\eqref{eq:monotonic_attention_alpha} requires the iterative update over $j$, there is an effective implementation which parallelizes the calculation by using the cumulative sum and product operations, denoted respectively as {\tt cumsum} and {\tt cumprod}, as
\scriptsize
\begin{eqnarray}
{\bm \alpha}_{i,:}={\bm p}_{i} \cdot \mbox{{\tt cumprod}}(1-{\bm p}_{i,:}) \cdot \mbox{{\tt cumsum}}\bigg(\frac{{\bm \alpha}_{i-1,:}}{\mbox{{\tt cumprod}}(1-{\bm p}_{i,:})}\bigg). \label{eq:monotonic_attention_alpha_parallel}
\end{eqnarray}
\normalsize
The chunkwise attention is estimated with $\alpha_{i,j}$ at training time and is calculated from the hard boundary at test time.
The interested readers are referred to~\cite{hard_monotonic_attention,mocha} for more details.

\subsection{Quantity regularization}\label{ssec:quantity_regularization}
An exponential decay of $\alpha_{i,j}$ due to the recurrence in Eq.~\eqref{eq:monotonic_attention_alpha} enlarges the mismatch between training and testing behaviors, which is the main cause of the poor performance of the original MoChA~\cite{online_hybrid_ctc_attention,inaguma2021alignment,online_hybrid_ctc_attention_taslp2020}.
To mitigate this problem, quantity regularization~\cite{inaguma2020streaming} was proposed by introducing a quantity loss $\mathcal{L}_{\rm qua}$, which makes the expectation of the total number of boundaries closer to the output sequence length $U$ as $\mathcal{L}_{\rm qua} = |U - \sum_{i,j}{\alpha_{i,j}}|.$

\subsection{Alignment path restriction}\label{ssec:alignment_path_restriction}
One of the effective methods to reduce the emission latency of MoChA is to ignore paths in Eq.~\eqref{eq:monotonic_attention_alpha} by referring to external alignment information extracted from the hybrid system~\cite{inaguma2020streaming}.
Given reference boundaries ${\bf b}^{\rm ref}=({\rm b}^{\rm ref}_{1}, \cdots, {\rm b}^{\rm ref}_{U})$ estimated from the word alignment, delay constrained training (DoCoT)~\cite{inaguma2020streaming} masks out inappropriate paths based on
\begin{eqnarray*}
\alpha_{i,j} = \begin{cases}
     p_{i,j}\bigg((1-p_{i,j-1})\frac{\alpha_{i,j-1}}{p_{i,j-1}}+\alpha_{i-1,j}\bigg) & (j \le {\rm b}^{\rm ref}_{i} + \delta) \\
     0 & ({\rm otherwise}),
\end{cases}
\end{eqnarray*}
where $\delta$ is a hyperparameter to control the delay.
To recover the reduced scale of $\alpha_{i,j}$, it is essential to use quantity regularization~\cite{inaguma2020streaming}.

Moreover, it is also possible to use CTC alignment to estimate $\bf{b}^{\rm ref}$.
We refer to this method as \textit{DeCoT-CTC} and note that it is newly investigated in this work.
Although the CTC alignment is not guaranteed to be accurate, it will be effective because the model still has some freedom to learn the optimal timing to emit tokens, unlike point-wise boundary restriction in the expected latency loss described in Section~\ref{ssec:expected_latency_loss}.
Moreover, the CTC alignment can be obtained without frame-level alignment supervision.
During training, we use the most probable CTC alignment as in~\cite{inaguma2020_ctcsync}.

\subsection{Expected latency loss}\label{ssec:expected_latency_loss}
Another strategy to reduce the latency is to use boundary supervision by minimizing the expected latency loss~\cite{inaguma2020streaming,inaguma2020_ctcsync} as
\begin{eqnarray}
{\rm b}^{\rm mocha}_{i} &=& \sum_{j=1}^{T'}j \cdot \alpha_{i,j}, \nonumber \\
\mathcal{L}_{\rm latency} &=& \frac{1}{U}\sum_{i=1}^{U}{|{\rm b}^{\rm ref}_{i} - {\rm b}_{i}^{\rm mocha}|}, \nonumber
\end{eqnarray}
where ${\rm b}^{\rm mocha}_{i}$ is the expected boundary index for the $i$-th token, and $T'$ is the time length of encoder outputs.
When using the alignment from a hybrid system, training becomes minimum latency training (MinLT)~\cite{inaguma2020streaming}.
When using the CTC alignment, it becomes CTC-synchronous training (CTC-ST)~\cite{inaguma2020_ctcsync}.
We have shown that CTC-ST outperforms MinLT in the accuracy and latency~\cite{inaguma2021alignment} and therefore focus on the former in this work.

\subsection{Total training objective}
To summarize the above methods, the generalized training objective of MoChA is formulated as follows:
\begin{multline}
\mathcal{L}^{\rm total}_{\rm mocha} = (1- \lambda_{\rm ctc}) \mathcal{L}_{\rm mocha} + \lambda_{\rm ctc} \mathcal{L}_{\rm ctc} \\
+ \lambda_{\rm latency} \mathcal{L}_{\rm latency} + \lambda_{\rm qua} \mathcal{L}_{\rm qua}, \label{eq:total_loss}
\end{multline}
where $\lambda_{*}$ is a corresponding task weight.
When $\lambda_{\rm latency} > 0$, $\lambda_{\rm qua}$ is set to 0.0 in this work~\cite{inaguma2020_ctcsync}.
We summarize the above alignment regularization methods in Table~\ref{tab:mocha_alignment_summary}.

\begin{table}[t]
    \centering
    \begingroup
    \caption{Summary of alignment regularization for MoChA}\label{tab:mocha_alignment_summary}
    \vspace{-2mm}
    \scalebox{1.0}{
    \begin{tabular}{c|l|c|c} \toprule
        \multicolumn{2}{c}{} & \multicolumn{2}{c}{Source of alignment} \\ \cmidrule(lr){3-4}
        &  & CTC & Hybrid ASR \\
        \midrule
        \multirow{4}{*}{\rotatebox{90}{Training}} & \multirow{2}{*}{\shortstack{Expected latency\\loss}} & \multirow{2}{*}{CTC-ST~\cite{inaguma2020_ctcsync}} & \multirow{2}{*}{MinLT~\cite{inaguma2020streaming}} \\
        &  &  &  \\ \cmidrule(lr){2-4}

       & \multirow{2}{*}{\shortstack{Alignment path\\restriction}} & \multirow{2}{*}{\shortstack{DeCoT-CTC\\(this work)}} & \multirow{2}{*}{DeCoT~\cite{inaguma2020streaming}} \\
       &  &  &  \\
      \bottomrule
    \end{tabular}
    }
    \endgroup
    \vspace{-2mm}
\end{table}

\section{StableEmit}\label{sec:proposed_method}
In this section, we propose \textit{StableEmit}, a simple but effective emission latency regularization method for MoChA without external alignment information.

\subsection{Motivation}
While the methods in Section~\ref{ssec:alignment_path_restriction} and Section~\ref{ssec:expected_latency_loss} can reduce the emission latency of MoChA, they require alignment information, although the CTC alignment can be obtained easily.
Moreover, at test time, token boundaries are detected when and only when $p_{i,j}$ surpasses a threshold $\tau=0.5$~\cite{hard_monotonic_attention}.\footnote{Setting $\tau=0.5$ is reasonable when assuming $p_{i,j}$ is well discretized after the training.}
Therefore, if the confidence of boundary prediction in some speech frames is below $\tau$, the corresponding token cannot be generated because it does not appear in the subsequent search process.
As a result, this leads to increasing deletion errors, especially for long-form speech data.

\subsection{Formulation}
To solve this problem, we propose \textit{StableEmit}, which discounts $p_{i,j}$ \textit{during training only} to detect token boundaries earlier with high confidence at test time.
Specifically, we introduce a constant discount factor $\lambda_{\rm se}$ $(0 \leq \lambda_{\rm se} < 1)$ and multiply $p_{i,j}$ by $1 - \lambda_{\rm se}$.
Therefore, Eq.~\eqref{eq:monotonic_attention_alpha} is reformulated as
\begin{eqnarray}
p'_{i,j} &=& (1 - \lambda_{\rm se}) \cdot p_{i,j}, \\ \label{eq:stable_emit}
\alpha_{i,j} &=& p'_{i,j}\bigg((1-p'_{i,j-1})\frac{\alpha_{i,j-1}}{p'_{i,j-1}}+\alpha_{i-1,j}\bigg). \nonumber
\end{eqnarray}
Accordingly, Eq.~\eqref{eq:monotonic_attention_alpha_parallel} is rewritten as
\footnotesize
\begin{eqnarray*}
{\bm \alpha}_{i,:}={\bm p}'_{i,:} \cdot \mbox{{\tt cumprod}}(1-{\bm p}'_{i,:}) \cdot \mbox{{\tt cumsum}}\bigg(\frac{{\bm \alpha}_{i-1,:}}{\mbox{{\tt cumprod}}(1-{\bm p}'_{i,:})}\bigg).
\end{eqnarray*}
\normalsize
Similarly to FastEmit, StableEmit is also a \textit{sequence-level} regularization because the discount of $p_{i,j}$ at every grid $(i,j)$ influences $\alpha_{i,j}$ globally.
As the discount decreases $\sum_{j} \alpha_{i,j}$, we further need to use the quantity regularization to recover it.
With this technique, $p_{i,j}$ is optimized to detect token boundaries with high confidence because it is encouraged to compensate the discount so that $\sum_{j} \alpha_{i,j}$ gets closer to 1.
In other words, the reasonable value of $\tau$ increases from 0.5 to $0.5 / (1 - \lambda_{\rm se})$ through the training.
However, at test time, we keep $\tau = 0.5$ as in the original formulation, unless otherwise noted.
This is effective for reducing the emission latency because $p_{i,j}$ can reach $\tau$ earlier in terms of $j$ if it is trained to recover the discount properly.

Since StableEmit just modifies $p_{i,j}$ during training, it can be combined with alignment regularization methods that manipulate $\alpha_{i,j}$ such as DeCoT and CTC-ST.
The effect of combination will be described in Section~\ref{sec:experiment}.

\section{Experimental evaluations}\label{sec:experiment}
\subsection{Experimental setup}\label{ssec:setup}
We used the TEDLIUM release v2 (TEDLIUM2)~\cite{tedlium}, a 210-hour English lecture speech corpus, and the evaluation sets include long-form utterances (40/30 seconds in the \textit{dev}/\textit{test} sets, respectively).
For the GPU memory capacity, however, we removed utterances longer than 16 and 20 seconds in the training data for LSTM and Conformer models, respectively.
We extracted 80-channel log-mel filterbank coefficients computed with a 25-ms window that was shifted every 10ms with Kaldi~\cite{kaldi}.
We applied three-fold speed perturbation~\cite{speed_perturbation} and SpecAugment~\cite{specaugment}.
The vocabularies were constructed by the byte pair encoding (BPE) algorithm ~\cite{sennrich2015neural} with 10k and 1k units for AED and RNN-T models, respectively.

For streaming encoders, we used the unidirectional LSTM (UniLSTM) and causal Conformer (UniConformer)~\cite{gulati2020} architectures, both of which follow four CNN layers with a $3 \times 3$ filter.
A max-pooling layer with a stride of $2 \times 2$ was inserted at the 2nd and 4th CNN layer for LSTM and the 4th CNN layer for UniConformer.
We used five layers of UniLSTM encoder with 1024 units~~\cite{inaguma2021alignment}.
We adopted Conformer (M) configuration in~\cite{gulati2020} while reducing the number of blocks from 16 to 12.
Moreover, we replaced batch normalization~\cite{ioffe2015batch} in each convolution module with layer normalization~\cite{ba2016layer}.\footnote{We also tried group normalization~\cite{wu2018group}, but layer normalization was the best in our implementations.}
We used Shaw's style relative positional encoding (RPE)~\cite{shaw-etal-2018-self} and clipped the relative distance to 10 in each block.
For streaming, we adopted causal depthwise separable convolution with a kernel size of 7 and a causal self-attention mask.
Therefore, the lookahead latency was introduced in the frontend CNN layers only.
Furthermore, we used hierarchical downsampling~\cite{dong2019self}, where the max-pooling with a stride of 2 was performed in the last frontend CNN layer, 4th, and 8th Conformer blocks, respectively.
This resulted in the downsampling factor of 8 for the UniConformer in total while 4 for the LSTM.
The total lookahead latency in the UniLSTM and UniConformer encoders were 60ms and 40ms, respectively.
The decoder consisted of a single layer of LSTM with 1024 units for MoChA, with a chunk size $w = 4$.
For RNN-T, we used a two-layer LSTM prediction network with 1024 units and a joint network with 512 units.

The Adam algorithm~\cite{adam} was used for optimization.
We set $\lambda_{\rm ctc}$ to 0.3 in all models, including RNN-T.\footnote{RNN-T is optimized via $\mathcal{L}^{\rm total}_{\rm rnnt} =(1- \lambda_{\rm ctc}) \mathcal{L}_{\rm rnnt} + \lambda_{\rm ctc} \mathcal{L}_{\rm ctc}$.}
We set ($\lambda_{\rm latency}$, $\lambda_{\rm qua}$) to (1.0, 0.0) for CTC-ST, otherwise (0.0, 2.0).
During inference, we use the beam width $b$ of 10 with a four-layer LSTM language model having 1024 units.
We used length normalization~\cite{murray-chiang-2018-correcting} and softmax smoothing with a temperature of 0.7~\cite{chorowski2017towards}.
Our implementation is publicly available.\footnote{\url{https://github.com/hirofumi0810/neural_sp}.}

\begin{table}[t]
    \centering
    \begingroup
    \caption{Results on TEDLIUM2. Token emission latency (TEL) is calculated on the test set. ${}^{\spadesuit}$SpecAugment was NOT used.
    }\label{tab:result_tedlium2}
    \vspace{-3mm}
    \scalebox{0.85}{
    \begin{tabular}{lccccc} \toprule
     \multirow{2}{*}{Model} & \multicolumn{2}{c}{WER [$\%$] ($\downarrow$)} & \multicolumn{3}{c}{TEL [ms] ($\downarrow$)} \\ \cmidrule(lr){2-3} \cmidrule(lr){4-6}
     \multicolumn{1}{c}{} & dev & test & 50th & 90th & 95th \\ \midrule
     BiLSTM - Global AED & \phantom{0}8.1 & \phantom{0}7.5 & -- & -- & -- \\

     UniLSTM - RNN-T ($b=5$) & 11.0 & 11.0 & -- & -- & -- \\ %
     UniLSTM - RNN-T & 10.7 & 10.7 & -- & -- & -- \\ %
     \hline
     UniLSTM - MoChA${}^{\spadesuit}$ & 16.3 & 14.9 & 280 & 680 & 1120 \\
     \ + StableEmit${}^{\spadesuit}$ & \bf{13.9} & \bf{12.7} & \bf{240} & \bf{440} & \phantom{0}\bf{640} \\

     \ \ \ + DeCoT-CTC ($\delta=12$) & \bf{12.4} & \bf{10.7} & \bf{240} & \bf{400} & \phantom{0}\bf{480} \\
     \ \ \ + DeCoT-CTC ($\delta=16$) & 12.5 & 11.1 & 280 & 440 & \phantom{0}520 \\
     \hline
     \ + DeCoT ($\delta=16$) & 12.4 & \bf{10.7} & 280 & 440 & \phantom{0}560 \\
     \ \ \ + StableEmit & \bf{12.0} & \bf{10.7} & \bf{240} & \bf{360} & \phantom{0}\bf{440} \\ \hline
     \ + CTC-ST & 13.5 & 11.3 & 200 & 360 & \phantom{0}480 \\
     \ \ \ + StableEmit & \bf{11.7} & \bf{10.9} & 200 & \bf{320} & \phantom{0}\bf{440} \\
     \midrule \midrule

     Conformer - Global AED & \phantom{0}7.3 & \phantom{0}7.0 & -- & -- & -- \\

     UniConformer - CTC & 12.1 & 11.2 & 240 & 320 & \phantom{0}400 \\
     UniConformer - RNN-T & \phantom{0}8.5 & \phantom{0}8.2 & -- & -- & -- \\ \hline %

     UniConformer - MoChA & \phantom{0}9.6 & \phantom{0}8.8 & 320 & 480 & \phantom{0}640 \\
     \ + StableEmit & \phantom{0}\bf{9.1} & \phantom{0}\bf{8.4} & \bf{240} & 480 & \phantom{0}\bf{480} \\
     \ \ \ + DeCoT-CTC ($\delta=8$) & \phantom{0}\bf{8.8} & \phantom{0}\bf{8.4} & \bf{240} & \bf{400} & \phantom{0}\bf{480} \\
     \ \ \ + DeCoT-CTC ($\delta=12$) & \phantom{0}\bf{8.5} & \phantom{0}\bf{8.4} & 320 & 480 & \phantom{0}\bf{480} \\
     \hline

     \ + DeCoT ($\delta=8$) & \phantom{0}8.5 & \phantom{0}8.5 & \bf{240} & 400 & \phantom{0}480 \\
     \ \ \ + StableEmit & \phantom{0}8.7 & \phantom{0}8.7 & \bf{240} & \bf{320} & \phantom{0}\bf{400} \\
     \ + DeCoT ($\delta=12$) & \phantom{0}8.7 & \phantom{0}8.5 & 320 & 480 & \phantom{0}560 \\
     \ \ \ + StableEmit & \phantom{0}\bf{8.6} & \phantom{0}\bf{8.2} & \bf{240} & \bf{400} & \phantom{0}\bf{480} \\ \hline
     \ + CTC-ST & \phantom{0}\bf{8.9} & \phantom{0}8.8 & \bf{240} & \bf{320} & \phantom{0}\bf{400} \\
     \ \ \ + StableEmit & \phantom{0}9.3 & \phantom{0}\bf{8.5} & \bf{240} & \bf{320} & \phantom{0}\bf{400} \\
     \bottomrule
    \end{tabular}
    }
    \endgroup
    \vspace{-4mm}
\end{table}

\subsection{Main results}\label{ssec:main_result}
\vspace{-1mm}
We summarized the results in Table~\ref{tab:result_tedlium2}.
We used forced alignment results to evaluate the token emission latency (TEL)~\cite{inaguma2021alignment} from the ground-truth acoustic boundary.
We report median, 90th, and 95th percentile latency calculated on the \textit{test} set.

\vspace{-1mm}
\subsubsection{UniLSTM encoder models}
\vspace{-1mm}
We first confirmed significant improvement by StableEmit over the baseline UniLSTM MoChA when SpecAugment was not used.
We activated SpecAugment with alignment regularization methods after pre-training the baseline model to stabilize the training~\cite{inaguma2020_ctcsync,inaguma2020streaming}.
While both CTC-ST and DeCoT were effective, the combination with StableEmit brought further improvements with an additional TEL reduction.
We confirmed that DeCoT-CTC was also effective with a smaller $\delta$ than that in DeCoT because CTC also had a delay.
Specifically, applying StableEmit to CTC-ST showed relative WER improvements of 13.3\% (\textit{dev}) and 3.5\% (\textit{test}), and absolute TEL reductions of 40ms and 80ms in the 90th and 95th percentile, respectively.
Compared to the baseline model, the TEL was reduced by 80ms, 160ms, and 240ms in the 50th, 90th, and 95th percentile, respectively.
We also observed that StableEmit was effective in reducing the tail of TEL distributions for DeCoT.

\begin{table}[t]
    \centering
    \begingroup
    \caption{Effects of boundary threshold and loss weight on TEDLIUM2. The encoder is UniConformer.}\label{tab:threshold_stableemit}
    \vspace{-3mm}
    \scalebox{0.82}{
    \begin{tabular}{ccccccc} \toprule
      \multirow{2}{*}{$\lambda_{\rm se}$} & \multirow{2}{*}{$\tau$} & \multicolumn{2}{c}{WER [$\%$] ($\downarrow$)} & sub / ins / del & \multicolumn{2}{c}{TEL [ms] ($\downarrow$)} \\ \cmidrule(lr){3-4} \cmidrule(lr){5-5} \cmidrule(lr){6-7}
      &  & dev & test & dev & 50th & 90th \\
      \midrule

      \multirow{5}{*}{0.0\phantom{0}} & 0.5 & \phantom{0}9.58 & 8.76 & 4.88 / 1.55 / 3.15 & 320 & 480 \\
      & 0.4 & \phantom{0}\bf{9.50} & 8.48 & 4.96 / 1.77 / 2.77 & 320 & 480 \\
      & 0.3 & \phantom{0}9.64 & 8.39 & 4.95 / 2.16 / 2.52 & 320 & 480 \\
      & 0.2 & 10.11 & \bf{8.32} & 5.00 / 2.79 / 2.31 & 320 & 480 \\
      & 0.1 & 11.70 & 8.70 & 5.00 / 4.78 / 1.91 & \bf{240} & \bf{400} \\
      \midrule

      0.0* & 0.5 & \phantom{0}9.80 & 9.00 & 4.92 / 1.45 / 3.43 & 320 & 560 \\
      \midrule \midrule

      \multirow{5}{*}{0.1\phantom{0}} & 0.5 & \phantom{0}9.13 & 8.45 & 4.94 / 1.48 / 2.71 & 240 & 480 \\
      & 0.4 & \phantom{0}9.10 & 8.47 & 4.93 / 1.57 / 2.60 & 240 & \bf{400} \\
      & 0.3 & \phantom{0}\bf{9.06} & 8.39 & 5.06 / 1.61 / 2.40 & 240 & \bf{400} \\
      & 0.2 & \phantom{0}9.12 & \bf{8.29} & 5.16 / 1.77 / 2.19 & 240 & \bf{400} \\
      & 0.1 & \phantom{0}9.63 & 8.45 & 5.26 / 2.28 / 2.15 & 240 & \bf{400} \\ \midrule

      0.1* & 0.5 & \phantom{0}9.13 & 8.55 & 4.97 / 1.42 / 2.74 & 240 & 480 \\
      \midrule \midrule

      0.05 & \multirow{3}{*}{0.5} & \phantom{0}9.15 & 8.56 & 4.92 / 1.51 / 2.73 & 320 & 480 \\
      0.15 &  & \phantom{0}9.38 & 8.45 & 4.90 / 1.73 / 2.76 & \bf{240} & \bf{400} \\
      0.2\phantom{0} &  & 10.01 & 8.99 & 4.86 / 2.47 / 2.67 & \bf{240} & \bf{400} \\
      \bottomrule
    \end{tabular}
    }
    \vspace{-2mm}
    \begin{flushright}
    {*$\lambda_{\rm se}$ was set to 0.1 at test time.}
    \end{flushright}
    \endgroup
    \vspace{-4mm}
\end{table}

\vspace{-1mm}
\subsubsection{UniConformer encoder models}
\vspace{-1mm}
Next, we applied StableEmit to the UniConformer models.
Unlike the UniLSTM models, we activated SpecAugment from scratch and activated $\mathcal{L}_{\rm latency}$ and $\mathcal{L}_{\rm qua}$ after running for 10 epochs, without pre-training.
Firstly, we observed a significant improvement from the UniLSTM models with the Conformer encoder.
StableEmit was also effective in this case, leading to relative WER improvements of 5.2\% (\textit{dev}) and 4.5\% (\textit{test}) and absolute reduction of the median TEL by 80ms.
Leveraging alignment information further reduced the TEL and WER in some cases.
In this regard, we observed that alignment path restriction (DeCoT and DeCoT-CTC) was more effective than CTC-ST in terms of WER while CTC-ST reached the same level of TEL as the pure CTC model without external alignment.
Although the gains became smaller than those of the UniLSTM models, we attributed it to the larger downsampling factor, which improved the baseline performance significantly, as shown in Section~\ref{ssec:ablation}.

\vspace{-1mm}
\subsection{Analysis}\label{ssec:analysis}
\vspace{-1mm}
Next, we study the effect of a discount factor $\lambda_{\rm se}$ (for training) and a threshold $\tau$ for selection probabilities (for testing).
We used the UniConformer encoder for this purpose.
The results in Table~\ref{tab:threshold_stableemit} showed that reducing $\tau$ from 0.5 slightly reduced WER regardless of the use of StableEmit.
However, the model trained with StableEmit consistently outperformed the baseline in both WER and TEL and was robust to a small $\tau$.
Furthermore, we also set $\lambda_{\rm se}$ to 0.1 during testing, but this did not improve WER nor TEL.
This verifies that the confidence of $p_{i,j}$ at test time is increased by StableEmit.
For simplicity, we keep $\tau = 0.5$ in all other experiments.

We also investigate the effect of different values of $\lambda_{\rm se}$.
Increasing $\lambda_{\rm se}$ reduced the 90th percentile TEL by 80ms while at the cost of WER.
Regarding error types, StableEmit successfully reduced the deletion errors while a strong constraint by a small $\tau$ or a large $\lambda_{\rm se}$ resulted in increasing the insertion errors.

\begin{table}[t]
    \centering
    \begingroup
    \caption{Ablation study on TEDLIUM2}\label{tab:result_ablation}
    \vspace{-3mm}
    \scalebox{0.82}{
    \begin{tabular}{lcc} \toprule
      \multirow{2}{*}{Model} & \multicolumn{2}{c}{WER [$\%$] ($\downarrow$)} \\ \cmidrule(lr){2-3}
       & dev & test  \\ \midrule
      UniConformer - MoChA + StableEmit & \phantom{0}\bf{9.1} & \phantom{0}\bf{8.4} \\
      \ \ w/o quantity reg. ($\lambda_{\rm qua}=0$ in Eq.~\eqref{eq:total_loss}) & \phantom{0}9.7 & \phantom{0}9.9 \\
      \ \ w/o StableEmit ($\lambda_{\rm se}=0$ in Eq.~\eqref{eq:stable_emit}) & \phantom{0}9.6 & \phantom{0}8.8 \\
      \ \ w/o hierarchical downsampling & 10.3 & 10.0 \\
      \ \ \ \ w/o StableEmit & 11.1 & 10.4 \\
      \ \ w/o CTC loss ($\lambda_{\rm ctc}=0$ in Eq.~\eqref{eq:total_loss}) & \phantom{0}9.2 & \phantom{0}8.8 \\
      \ \ LayerNorm $\to$ BatchNorm & 31.3 & 23.4 \\
      \midrule
      UniConformer - RNN-T & \phantom{0}\bf{8.5} & \phantom{0}\bf{8.2} \\
      \ \ w/o hierarchical downsampling & \phantom{0}8.7 & \phantom{0}8.4 \\
      \ \ w/o CTC loss ($\lambda_{\rm ctc}=0$) & \phantom{0}8.7 & \phantom{0}8.6 \\
      \ \ LayerNorm $\to$ BatchNorm & \phantom{0}8.8 & \phantom{0}\bf{8.2} \\
      \bottomrule
    \end{tabular}
    }
    \endgroup
    \vspace{-2mm}
\end{table}

\vspace{-1mm}
\subsection{Ablation study}\label{ssec:ablation}
\vspace{-1mm}
We also conduct the ablation study in Table~\ref{tab:result_ablation}.
We observed that the quantity regularization was essential for StableEmit.
The hierarchical downsampling and layer normalization were also key ingredients when the UniConfomer encoder was used.
Moreover, we found that the hierarchical downsampling, the auxiliary CTC loss, and layer normalization were also beneficial for RNN-T.

\begin{figure}[t]
  \centering
  \begingroup
  \includegraphics[width=1.0\linewidth]{visualization_v2.png}
  \vspace{-5mm}
  \caption{Visualization of selection probabilities and CTC outputs from UniLSTM MoChA and the input spectrogram. Yellow regions represent high probabilities.}
  \label{fig:visualization_p_choose}
  \endgroup
  \vspace{-1mm}
\end{figure}

\vspace{-1mm}
\subsection{Visualization of selection probabilities}\label{ssec:visualization_selection_prob}
\vspace{-1mm}
We visualize the selection probabilities $p_{i,j}$ in the UniLSTM MoChA models in Figure~\ref{fig:visualization_p_choose}.
The yellow regions represent high probabilities close to 1.
We used the whole encoder outputs for visualization.
We also plotted the CTC probabilities obtained from the model trained with CTC-ST and StableEmit.
We can see that the baseline MoChA had blurred probabilities, and StableEmit sharpened the distribution, indicating the confidence to emit a token was increased.
CTC-ST shifted some token boundaries to the left and sharpened the probability distribution~\cite{inaguma2021alignment}.
Moreover, its combination with StableEmit further made them clearer, showing a complementary effect.
The delay became almost the same as that of CTC.

\begin{table}[t]
    \centering
    \begingroup
    \caption{Results on Librispeech. TEL was averaged over the test-clean and test-other sets.}\label{tab:result_librispeech}
    \vspace{-3mm}
    \scalebox{0.78}{
    \begin{tabular}{lcccccc} \toprule
     \multirow{3}{*}{Model} & \multicolumn{4}{c}{WER [$\%$] ($\downarrow$)} & \multicolumn{2}{c}{TEL [ms] ($\downarrow$)} \\ \cmidrule(lr){2-5} \cmidrule(lr){6-7}
     \multicolumn{1}{c}{} & \multicolumn{2}{c}{dev} & \multicolumn{2}{c}{test} & \multirow{2}{*}{50th} & \multirow{2}{*}{90th} \\
     \multicolumn{1}{c}{} & clean & other & clean & other &  &   \\ \midrule
     UniConformer - RNN-T  & 2.8 & 8.2 & 3.1 & 8.5 & -- & -- \\
     UniConformer - MoChA & 2.6 & 7.6 & 2.8 & 7.9 & 400 & 560 \\
     \ \ + StableEmit & 2.9 & \bf{7.5} & 3.0 & 8.2 & \bf{320} & \bf{480} \\
     \bottomrule
    \end{tabular}
    }
    \endgroup
    \vspace{-4mm}
\end{table}

\vspace{-1mm}
\subsection{Results on Librispeech}
\vspace{-1mm}
Finally, we present results on Librispeech (960h)~\cite{librispeech} in Table~\ref{tab:result_librispeech}.
We used the same architecture as previous experiments.
We used $\lambda_{\rm qua}=0.1$ for the baseline and $(\lambda_{\rm qua}$, $\lambda_{\rm se}$) = (0.2, 0.2) for StableEmit.
We observed TEL reductions of 80ms in the 50th and 90th percentile at the cost of a slight WER degradation.
This is because MoChA did not suffer from the deletion errors on this corpus.
However, it outperformed a strong RNN-T model having the same encoder architecture.

\section{Conclusions}\label{sec:conclusion}
In this work, we have proposed StableEmit, a simple alignment-free solution to reduce the emission latency of MoChA for streaming ASR applications.
We encouraged the model to learn high selection probabilities in the token boundary detection by discounting them during training as a regularization.
We experimentally demonstrated that StableEmit reduced not only the emission latency but also the deletion errors.
Combining it with the alignment regularization further reduced the latency.

\bibliographystyle{IEEEtran}