text:
\section{Introduction}
\label{sec:intro}

These guidelines include complete descriptions of the fonts, spacing, and
related information for producing your proceedings manuscripts. Please follow
them and if you have any questions, direct them to Conference Management
Services, Inc.: Phone +1-979-846-6800 or email
to \\\texttt{papers@2021.ieeeicassp.org}.

\section{Formatting your paper}
\label{sec:format}

All printed material, including text, illustrations, and charts, must be kept
within a print area of 7 inches (178 mm) wide by 9 inches (229 mm) high. Do
not write or print anything outside the print area. The top margin must be 1
inch (25 mm), except for the title page, and the left margin must be 0.75 inch
(19 mm).  All {\it text} must be in a two-column format. Columns are to be 3.39
inches (86 mm) wide, with a 0.24 inch (6 mm) space between them. Text must be
fully justified.

\section{PAGE TITLE SECTION}
\label{sec:pagestyle}

The paper title (on the first page) should begin 1.38 inches (35 mm) from the
top edge of the page, centered, completely capitalized, and in Times 14-point,
boldface type.  The authors' name(s) and affiliation(s) appear below the title
in capital and lower case letters.  Papers with multiple authors and
affiliations may require two or more lines for this information. Please note
that papers should not be submitted blind; include the authors' names on the
PDF.

\section{TYPE-STYLE AND FONTS}
\label{sec:typestyle}

To achieve the best rendering both in printed proceedings and electronic proceedings, we
strongly encourage you to use Times-Roman font.  In addition, this will give
the proceedings a more uniform look.  Use a font that is no smaller than nine
point type throughout the paper, including figure captions.

In nine point type font, capital letters are 2 mm high.  {\bf If you use the
smallest point size, there should be no more than 3.2 lines/cm (8 lines/inch)
vertically.}  This is a minimum spacing; 2.75 lines/cm (7 lines/inch) will make
the paper much more readable.  Larger type sizes require correspondingly larger
vertical spacing.  Please do not double-space your paper.  TrueType or
Postscript Type 1 fonts are preferred.

The first paragraph in each section should not be indented, but all the
following paragraphs within the section should be indented as these paragraphs
demonstrate.

\section{MAJOR HEADINGS}
\label{sec:majhead}

Major headings, for example, "1. Introduction", should appear in all capital
letters, bold face if possible, centered in the column, with one blank line
before, and one blank line after. Use a period (".") after the heading number,
not a colon.

\subsection{Subheadings}
\label{ssec:subhead}

Subheadings should appear in lower case (initial word capitalized) in
boldface.  They should start at the left margin on a separate line.

\subsubsection{Sub-subheadings}
\label{sssec:subsubhead}

Sub-subheadings, as in this paragraph, are discouraged. However, if you
must use them, they should appear in lower case (initial word
capitalized) and start at the left margin on a separate line, with paragraph
text beginning on the following line.  They should be in italics.

\section{PRINTING YOUR PAPER}
\label{sec:print}

Print your properly formatted text on high-quality, 8.5 x 11-inch white printer
paper. A4 paper is also acceptable, but please leave the extra 0.5 inch (12 mm)
empty at the BOTTOM of the page and follow the top and left margins as
specified.  If the last page of your paper is only partially filled, arrange
the columns so that they are evenly balanced if possible, rather than having
one long column.

In LaTeX, to start a new column (but not a new page) and help balance the
last-page column lengths, you can use the command ``$\backslash$pagebreak'' as
demonstrated on this page (see the LaTeX source below).

\section{PAGE NUMBERING}
\label{sec:page}

Please do {\bf not} paginate your paper.  Page numbers, session numbers, and
conference identification will be inserted when the paper is included in the
proceedings.

\section{ILLUSTRATIONS, GRAPHS, AND PHOTOGRAPHS}
\label{sec:illust}

Illustrations must appear within the designated margins.  They may span the two
columns.  If possible, position illustrations at the top of columns, rather
than in the middle or at the bottom.  Caption and number every illustration.
All halftone illustrations must be clear black and white prints.  Colors may be
used, but they should be selected so as to be readable when printed on a
black-only printer.

Since there are many ways, often incompatible, of including images (e.g., with
experimental results) in a LaTeX document, below is an example of how to do
this \cite{Lamp86}.

\section{FOOTNOTES}
\label{sec:foot}

Use footnotes sparingly (or not at all!) and place them at the bottom of the
column on the page on which they are referenced. Use Times 9-point type,
single-spaced. To help your readers, avoid using footnotes altogether and
include necessary peripheral observations in the text (within parentheses, if
you prefer, as in this sentence).

\begin{figure}[htb]

\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{image1}}
  \centerline{(a) Result 1}\medskip
\end{minipage}
\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image3}}
  \centerline{(b) Results 3}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image4}}
  \centerline{(c) Result 4}\medskip
\end{minipage}
\caption{Example of placing a figure with experimental results.}
\label{fig:res}
\end{figure}

\section{COPYRIGHT FORMS}
\label{sec:copyright}

You must submit your fully completed, signed IEEE electronic copyright release
form when you submit your paper. We {\bf must} have this form before your paper
can be published in the proceedings.

\section{RELATION TO PRIOR WORK}
\label{sec:prior}

The text of the paper should contain discussions on how the paper's
contributions are related to prior work in the field. It is important
to put new work in  context, to give credit to foundational work, and
to provide details associated with the previous work that have appeared
in the literature. This discussion may be a separate, numbered section
or it may appear elsewhere in the body of the manuscript, but it must
be present.

You should differentiate what is new and how your work expands on
or takes a different path from the prior studies. An example might
read something to the effect: "The work presented here has focused
on the formulation of the ABC algorithm, which takes advantage of
non-uniform time-frequency domain analysis of data. The work by
Smith and Cohen \cite{Lamp86} considers only fixed time-domain analysis and
the work by Jones et al \cite{C2} takes a different approach based on
fixed frequency partitioning. While the present study is related
to recent approaches in time-frequency analysis [3-5], it capitalizes
on a new feature space, which was not considered in these earlier
studies."

\vfill\pagebreak

\section{REFERENCES}
\label{sec:refs}

List and number all bibliographical references at the end of the
paper. The references can be numbered in alphabetic order or in
order of appearance in the document. When referring to them in
the text, type the corresponding reference number in square
brackets as shown at the end of this sentence \cite{C2}. An
additional final page (the fifth page, in most cases) is
allowed, but must contain only references to the prior
literature.

\bibliographystyle{IEEEbib}

\section{Experiments}
\label{sec: experiment}

We first use our proposed framework to evaluate the effect of style diversity.
As discussed, increasing the sampling bias reduces the style diversity of synthetic data.
\figref{figure: exp cer} shows the CERs of the English recognizer when varying the sampling bias and the amount of synthetic training data.
To avoid introducing new content, the  data was synthesized using the text corpus of the original training data.
The results indicate that:
(1) A large sampling bias (bias $\ge$ 0.4; \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot neat training data) leads to low \WER{}{r}{s} and high \WER{}{s}{r}, indicating a decrease in the diversity of the synthetic data (\figref{figure: exp wer test rs}), corresponding to Case 2.
(2) A small sampling bias (0.05 $\le$ bias < 0.4) increases the diversity of the data, covering a large distribution of styles, leads to lower  \WER{}{s}{r} and \WER{}{b}{r} than \WER{}{r}{r}, corresponding to Case 3.
Due to the increase in data diversity, we also observe an increase in \WER{}{r}{s} and \WER{}{s}{s}.
(3) Further reducing the sampling bias (bias < 0.05) leads to an increase in \WER{}{s}{r} and \WER{}{b}{r} due to the occasional generation of out-of-distribution samples, corresponding to Case 1. %
When we use a very small sampling bias, we sample from a mixture of Gaussian distributions with large standard deviations.
Therefore, the larger standard deviations make it more likely to generate out-of-distribution samples.

These results demonstrate the effectiveness of data incubation to improve recognizers and of the proposed recipe to analyze synthetic data and optimize synthesis.
By selecting the right sampling bias, we are able to reduce the CER of the English recognizer by 28\% compared to the model trained only using real data (Table~\ref{table: final}) because we are able to increase the diversity of writing styles.

Further, for our multilingual recognizer, we show that using additional content helps improve the CER even further in a scenario where we don't have good coverage of all relevant content.
When training our system with the same text corpus, we obtain an improvement of 46\% on the normal multilingual test corpus.
However, in this scenario, we only have 60k training samples for each of the non-English languages (compared to 300k for English). Therefore we increase the content diversity by synthesizing from a large multilingual text corpus. This leads to an improvement of 66\% (Table~\ref{table: final}).

Finally, we also show results on a special patterns (SP) evaluation set containing content like URLs, emails, addresses, and hashtags which are underrepresented in the collected training data.
As shown in the last column of Table~\ref{table: final}, expanding the synthesis corpus enables our multilingual model to also achieve 66\% reduction in CER compared to the model trained using only the real data.

\input{table_final_td}
\input{figure_test}

\section{Introduction}
\label{sec: introduction}

\begin{figure}[th!]
	\centering
	\includegraphics[width=.8\linewidth]{figures/illustration/illustration_v5.pdf}
	\caption{
	    Data incubation starts with (a) collected data sparsely covering the content and the style space.
	    %
	    %
	    We then use a controllable generative model that can (b) synthesize samples with unseen content, and (c) mix new style from existing training samples.
	    %
	    (d) Finally, we synthesize new samples that close gaps in content and style, leading to a better coverage of the real data distribution for handwriting recognition.
}
	\label{figure:illustration}
\end{figure}

Data is the most valuable asset for machine learning applications, but are we using it efficiently?
The standard approach is relatively simple: (i) collect and annotate data, and (ii) train a recognizer using the data.
While this approach has achieved success in many standard machine learning tasks, such as image classification~\cite{tan2019efficientnet}, we will show that it is sub-optimal for handwriting recognition.

Handwriting data can be described
(a) by its content (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, the text that was written, a sequence of characters)
and (b) by its style (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, all other information about a handwriting sample except content {\textemdash} printed or cursive, neat or messy, round or square corners, tight or loose spacing, \emph{etc}\onedot} \def\vs{\emph{vs}\onedot).
The content space is combinatorial and some handwriting styles are rare compared to others.
Therefore, it is difficult to collect sufficient data to cover the entire content and style space well.

In this paper, we propose a method we refer to as \textit{data incubation} to grow the original collections and improve the generalization of recognizer training.
This capability is enabled by a controllable generative model that is used to augment training with synthesized data covering underrepresented content and rare styles (\figref{figure:illustration}).

How does a controllable generative model improve data efficiency for a downstream handwriting recognizer?
After all, we learn the generative model with the same dataset that we use to train the recognizer.
Our approach is based on the following observation:
The total number of handwriting samples that can be collected is limited and thus the diversity of contents and styles in the collected samples is also limited (\figref{figure:illustration}a).
By utilizing our controllable synthesizer that models the data distribution as a factored distribution of style and content, we can combine information across collected samples, complement the real training set with synthesized handwriting, and fill both the content and the style gaps in the collected training data (\figref{figure:illustration}b-d).

Effectively, we introduce prior knowledge about content and style being factorizable into the creation of synthetic training data.
This inductive bias allows us to strategically increase the size of our training set to target a more balanced distribution of writing styles and content as well as training recognizers with better generalization.

Training with synthetic data, however, comes with its own challenges.
Synthetic data might introduce a shift from the \textit{empirical distribution} that is formed by the collected real data. %
While this shift can be due to the artifacts produced by the generative model, it can also come from generating content and styles that are underrepresented or missing in the collected data, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, shifting the empirical distribution shown in \figref{figure:illustration}a to that in \figref{figure:illustration}d.
In this aspect, this distribution shift can be seen both as a problem but also as a feature.
It is critical to make careful and deliberate choices about which data to synthesize.
If we only synthesize handwriting from well-formed English sentences with highly legible, printed style, a recognizer trained on this data will end up performing poorly on messy, cursive styles and content including slang, uncommon words, or abbreviations.
On the other hand, we can carefully choose which data to synthesize and with that we will be able to close the gaps in the empirical distribution.

In this paper, we demonstrate a systematic recipe to train a handwriting recognizer with data synthesized by a controllable generative model.
This recipe enables us to examine and understand the problems associated with the synthetic data.
We build on the recently-proposed controllable generative model~\cite{chang2021style}, which separately controls the content and the handwriting style.
As we will demonstrate, utilizing data incubation significantly improves our multilingual handwriting recognizer, achieving a 66\% reduction in Character Error Rate when compared to a recognizer trained with only original data.

\section{Conclusion}
We presented data incubation{\textemdash}a new framework to improve recognition by (i) training a controllable generative model to synthesize missing data, (ii) optimizing data synthesis, and (iii) training recognition models by strategically combining synthetic and real data. We applied this framework to handwriting recognition and achieved a $66\%$ reduction in CER.

\bibliographystyle{IEEEbib}

\section{Generative and recognizer models}
\vspace{-0.2cm}
\label{sec: models}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{figures/arch/arch_v6.pdf}
	\vspace{-2.2mm}
	\caption{
	(a) Online handwriting is a sequence of stylus movements.
    (b) Our handwriting generative model takes an input content ${\bf c}$ and outputs handwriting ${\bf x}$.  The handwriting style is controlled using a latent style variable $z_t$. The model can either mimic the style of an existing style reference handwriting or it can sample a new style from a prior distribution to generate unseen styles.
	(c) Our recognizer takes an input handwriting ${\bf x}$ and outputs the probabilities of all symbols at each time step, $p(\pi | {\bf x})$, which is converted to $p({\bf c} | {\bf x})$ by CTC decoding.
	}
	\vspace{-2mm}
	\label{figure: arch}
\end{figure}

\subsection{Problem formulation and dataset}

An online handwriting sample ${\bf x} = \left[ ({\Delta x}_t, {\Delta y}_t, u_t) \right]_{t=1 \dots T}$ is a sequence of stylus movements on a writing surface, where ${\Delta x}_t \, {\in} \,  {\mathbb R}$ and ${\Delta y}_t \, {\in} \, {\mathbb R}$ are the movement of the stylys in $x$ and $y$ directions respectively at time $t$, and $u_t \, {\in} \, \{0, 1\}$ is a binary variable indicating whether the stylus touches the surface during the movement, as shown in \figref{figure: arch}a.
The corresponding content of the sample ${\bf c} = \left[ c_1, \dots c_M \right]$ is a sequence of characters, where $c_m$ is represented as a one-hot vector of dimension $Q$ (the total number of characters).
Note that in general ${\bf x}$ and ${\bf c}$ have different lengths.  %

We use a subset of a proprietary dataset collected for research and development.
The subset consists of 600k online handwriting samples written by 1,500 people.
Half of the samples (300k) is in English, and the rest is in French, German, Italian, Spanish, and Portuguese.

\subsection{Controllable generative model }

We use the controllable generative model proposed in \cite{chang2021style}.
Given $N$ real training samples, $\mathcal{X} = \left\{  ({\bf x}^i, {\bf c}^i) \right\}_{i=1 \dots N}$, the generative model learns a distribution of handwriting samples, $p({\bf x} | {\bf c}, {\bf z})$, conditioned on the content ${\bf c}$ and the style ${\bf z}$.
In other words, given a target content ${\bf c}$ and a style ${\bf z}$, we can generate new handwriting by sampling from the distribution.
Note that the style ${\bf z}$ is modeled as a latent variable with a (learnable) prior distribution.
Fitting a prior distribution for style is important, because it enables the model to create new styles (\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, ${\bf z}$ not associated with any training sample).

\figref{figure: arch}b shows an overview of the model architecture.
We model in an auto-regressive manner the output distribution $p({\Delta x}_t, {\Delta y}_t, u_t | z_t, {\bf x}_\past, {\bf c})$ as $p({\Delta x}_t, {\Delta y}_t | \cdot) p(u_t | \cdot)$, where $p(x_t, y_t | \cdot)$ is a mixture of 20 bivariate Gaussian distributions modeling the movement of the stylus, and $p(u_t |  \cdot)$ is a Bernoulli distribution modeling whether the stylys touches the surface.
The prior distribution of $z_t$, $p(z_t | {\bf x}_\past, {\bf c})$, is a multivariate Gaussian distribution $\mathcal{N}(z_t | \mbox{\boldmath{$\mu$}}_t, \Sigma_t))$, where $\Sigma_t$ is a diagonal covariance, and $\mbox{\boldmath{$\mu$}}_t$ and $\Sigma_t$ are functions of ${\bf x}_\past$ and ${\bf c}$ (generated by a linear layer).
We refer to \cite{chang2021style} for detailed description of the model.
We trained two generative models: an English model using only the English training data and a multilingual model using training data from all six languages.

One important property of the generative model is that we can control the variation of the synthesized data by scaling the standard deviations of the output distribution $p({\Delta x}_t, {\Delta y}_t, u_t | z_t, {\bf x}_\past, {\bf c})$.
Intuitively, using a small standard deviation generates handwriting close to the mean of the distribution, which results in neat (average-style) handwriting; using a large standard deviation results in more variations in the synthesized handwriting.
When synthesizing, we use \textit{sampling bias} $b \ge 0$ to scale individual standard deviations and the categorical distribution in the mixture of the Gaussian distributions, following the rule in~\cite{graves2013generating}.
A small sampling bias produces large variations in handwriting, and a large sampling bias results in small variations and neat/typical handwriting.

\vspace{-0.3cm}

\subsection{Handwriting recognizer}
\vspace{-0.1cm}

A handwriting recognizer learns a distribution of the content, $p({\bf c} | {\bf x})$, given an input handwriting sequence ${\bf x}$.
As shown in \figref{figure: arch}c, our handwriting recognizer is a 2-layer convolution network followed by a 2-layer bidirectional LSTM and a linear layer.
The input to the recognizer is a sequence containing $u_t, \sin(\theta_t), \cos(\theta_t), \ell_t$ for $t = 1, \dots, T$, where $\theta$ and $\ell$ are the angle and length of the stylus movement at time $t$, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, they are features computed from ${\Delta x}_t$ and ${\Delta y}_t$.
The network outputs the probabilities of each character (including a \textit{blank} symbol) at every time $t$.
We use CTC loss~\cite{graves2006connectionist} to train the model.

\section{Synthesize training data}

How do we efficiently utilize generative models to synthesize training data for a downstream recognizer?
In this section, we propose a systematic framework that identifies beneficial generative settings and underlying issues of the synthetic data.
Our recipe is motivated by the recent evaluation metric widely used to access the generation quality of a generative adversarial network~\cite{shmelkov2018good}.

The proposed framework is based on Table~\ref{table: wer definition}.
Given a synthetic dataset (which is randomly split into training, validation, and test sets) of interest, we train three recognizers --- one using only the real training data, one using only the synthetic training data, and the other using both.
Each of the three models are evaluated by Character Error Rates (CERs) using real and synthetic test sets, as denoted in Table~\ref{table: wer definition}.
Examining these CER values enables us to systematically evaluate the effect of training with the synthetic dataset.

We denote by $\mathcal{S}_e$ the support of the empirical data distribution (formed by the collected training dataset), by $\mathcal{S}_s$ the support of the synthetic data distribution, and by $\mathcal{S}_r$ the support of the real data distribution in-the-wild.
Note that since we have a limited number of collected real data, $\mathcal{S}_e$ only partially covers $\mathcal{S}_r$, \emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot, it lacks certain content and style.
The following conditions can arise:

\paragraph{Case 1: synthetic data contain exploitable artifacts.}
When the synthetic data contains artifacts that correlate with content, the recognizer trained on the synthetic data can exploit these artifacts, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, we have low \WER{tr}{s}{s} and \WER{}{s}{s}.
However, the same recognizer will perform poorly on real data which lack the exploitable artifacts, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, high \WER{}{s}{r}.

\paragraph{Case 2:  $\mathcal{S}_s \subset \mathcal{S}_r$.}
In this case, we have a lower diversity in the synthetic data than in the real data, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, \WER{}{s}{r} > \WER{}{r}{r}.
When the synthetic data only contain typical samples, \emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot, data that appear more frequently, we have \WER{}{r}{r} > \WER{}{r}{s}.
As we have discussed in \secref{sec: models}, when we use a high sampling bias to generate data, the synthetic dataset contains mostly clean and typical handwriting.
Thus, the recognizer trained on synthetic data will perform poorly on real handwriting, which can be messy sometimes.

\paragraph{Case 3: $\mathcal{S}_s \supset \mathcal{S}_e$.}
This is the scenario where the recognizer can benefit from the synthetic data.
By filling in missing content and style, synthetic data increase both the number and the diversity of training data, and thereby improve the performance of downstream recognizers.
In this case, since the synthetic data is more complex than the collected real data, we have \WER{}{r}{s} > \WER{}{r}{r} and \WER{}{s}{s} > \WER{}{r}{r}.
Note that we need to pay attention on the style and content that we introduce to $\mathcal{S}_s$ --- we do not want to overfill the synthetic dataset with words and handwriting styles that rarely appear in the real world.
As discussed before, when we use a small sampling bias to synthesize data, we generate handwriting of a wide variety.
As we will see in the experiments, while it increases the diversity of our training data, it may decrease the performance on real test set.
Using the proposed training procedure, we are able to choose the optimal sampling bias.

\input{table_cer}

\section{Synthesizing training data}
\label{sec: cases}

If data is not synthesized carefully, the domain gap between real and synthetic data can deteriorate the performance of the recognizer on real data. Here, we present a systematic framework to identify problems with synthetic data and optimize  parameters of the generative model to minimize the gap. Our recipe uses a similar procedure as \cite{shmelkov2018good}  to evaluate the generation quality of a generative adversarial network.

Given a synthetic dataset (randomly split into training, validation, and test sets), we train three recognizers: one using only the real training data, one using only the synthetic training data, and one using both.
Each model is evaluated using Character Error Rate (CER) on both the real and synthetic test sets. We use the notation \WER{}{t}{v} to indicate the CER for a model trained on training set $\text{t}$ and evaluated on set $\text{v}$, where $\text{t},\text{v} \in \{\text{r}, \text{s}, \text{b}\}$ indicating \textbf{r}eal, \textbf{s}ynthetic, and \textbf{b}oth data sets, respectively.

Note that the real training set (empirical distribution) is only a finite sample approximation to the true data distribution and might be biased due to the data collection procedure or lack certain content, style, or other variations. Here, we refer to the empirical distribution as $\mathcal{S}_r$, synthetic data distribution as $\mathcal{S}_s$ and the true data distribution as $\mathcal{S}_t$. We assume that the real test set is unbiased. The following are the \emph{common} cases:

\paragraph{Case 1: $\mathcal{S}_s \not \subseteq \mathcal{S}_t$.} Synthetic data contains artifacts or variations that do not exist in the true distribution. These artifacts are equivalent to introducing noisy samples to the training set. In an extreme case, the artifacts may correlate with the content; therefore, the recognizer trained on the synthetic data can exploit these artifacts and will not generalize. The extreme case can be diagnosed by lower \WER{}{s}{s}, but high \WER{}{s}{r} and \WER{}{r}{s}.

\paragraph{Case 2:  $\mathcal{S}_s \subseteq \mathcal{S}_t \ \ \& \ \ \mathcal{S}_s \subset \mathcal{S}_r$.} Synthetic data is a subset of the true distribution, but fails to capture all the variations that exist in the real dataset (empirical distribution). The model trained on synthetic data will fail to generalize well. For example, when both real and synthetic data share the same corpus but the synthetic data only contain printed-style handwriting, the recognizer trained on synthetic data will perform poorly on the real data. Similarly, as we have discussed in \secref{sec: models}, when we use a high sampling bias to generate data, the synthetic dataset contains mostly neat and typical handwriting. Thus, the recognizer will perform poorly on real handwriting, which can be difficult to read sometimes. This case can be diagnosed by lower \WER{}{s}{s} and \WER{}{r}{s}, but high \WER{}{s}{r}.

\paragraph{Case 3:  $\mathcal{S}_s \subseteq \mathcal{S}_t \ \ \& \ \ \mathcal{S}_s \not \subseteq \mathcal{S}_r$.}
Synthetic data is a subset of the true distribution and contains variations (\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot missing style and content) that are not covered in the real dataset. This is the scenario where the recognizer will  benefit  from  the  synthetic  data. By filling in missing data, synthetic data improves the performance of the recognizer leading to lower \WER{}{s}{r} (and \WER{}{s}{s}) than \WER{}{r}{r} and \WER{}{r}{s}.

To mitigate the problems discussed in case 1 and case 2, either the generative model or the sampling process has to be improved accordingly. When the generative model exactly models the true distribution, \WER{}{s}{r} and \WER{}{s}{s} should be identical. In practice, this is difficult to achieve; however, choosing the generative model such that  \WER{}{s}{r} and \WER{}{s}{s} are close is a good strategy to minimize the gap between synthetic and real data. Moreover, in general, neither
case 2 nor case 3 are exact (synthetic and real data contains different variations of the true distribution); therefore, combining real data with synthetic data (\WER{}{b}{r}) improves over using synthetic data (\WER{}{s}{r}) or real data (\WER{}{r}{r}) alone. In the experiments section, we analyze these cases based on the sampling bias of the generative model.

\section{Related work}

Online handwriting recognition is a field of high relevance in mobile computing because
(a) an increasing number of devices are equipped with styluses making handwritten input a natural interface,
(b) many languages are harder to enter with a keyboard than Latin scripts because the size of their alphabet or the use of grapheme clusters makes building intuitive keyboards difficult.
While early work in online handwriting recognition used segment-and-decode methods \cite{newton,pittman:2007,keysers2016multi,jaeger2001online}, recent work trends toward end-to-end approaches utilizing Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), coupled with Connectionist Temporal Classification (CTC) decoding~\cite{graves2006connectionist,plamondon2000online,carbune2020fast,graves-online-nips2008,frinken2015deep}.

Online handwriting synthesis models~\cite{chang2021style,zhang2018separating,bhunia2021handwriting,graves2013generating,aksan2018deepwriting,kotani2020generating} are sequence-to-sequence models that take input text (a sequence of characters) and output handwriting (a sequence of strokes).
To improve downstream recognizers, it is important for the synthesized handwriting to contain few artifacts and a wide range of handwriting styles --- from highly legible, printed-style handwriting to less legible, cursive handwriting.

Synthetic data augmentation for training machine learning models is not new in the literature.
Most of the reference papers focus on image~\cite{xie2020adversarial,pouransari2021extracurricular} and speech~\cite{laptev2020you,li2018training,rosenberg2019speech,zheng2021using,mimura2018leveraging} synthesis, and obtain modest improvements.
To the best of our knowledge, this paper is the first to use a controllable generative model to synthesize online handwriting and explicitly control the content, style, and variation of the generation process to significantly improve the accuracy of downstream handwriting recognizers.


text:
\section{Introduction}
\setcounter{footnote}{0}

At the heart of modern theoretical physics lies the ancient question: "{\it does a fundamental principle, that explains all of reality and that cannot itself be reduced to other, deeper principles, exist?}  This is the search for a {\it final} theory.

With {\bf quantum holonomy theory} we propose a candidate for such a fundamental principle and a candidate for a final theory. Quantum holonomy theory is based on an elementary $\ast$-algebra called the {\it quantum holonomy-diffeomorphism algebra} -- denoted by $\mathbf{Q H D} (M)  $ -- which is generated first by holonomy-diffeomorphisms on a 3-dimensional manifold $M$ and secondly by canonical translation operators on an underlying configuration space of connections -- the space over which the holonomy-diffeomorphisms form a non-commutative $C^*$-algebra of functions \cite{Aastrup:2014ppa}.

Thus, the $\mathbf{QH D} (M)  $ algebra simply encodes how {\it stuff} is moved around in a 3-dimensional manifold. This algebra is so elementary -- and conceptually almost empty -- that it seems like the perfect foundation for a {\it final} theory. The question "{\it what are diffeomorphisms made of?} makes little sense.

The theory, that the $\mathbf{QH D} (M)  $ algebra gives rise to, has several key characteristics of a non-perturbative and background independent theory of quantum gravity coupled to matter and gauge degrees of freedom.
First of all, an infinitesimal version of the central algebraic relation in the $\mathbf{QH D} (M)  $ algebra reproduces the canonical commutation relations of canonical quantum gravity formulated in terms of Ashtekar variables\footnote{In the present setup we use the gauge group $SU(2)$ which in the canonical setup corresponds to a Euclidean signature \cite{AL1}.} \cite{Ashtekar:1986yd,Ashtekar:1987gu}. This means that a Hilbert space representation of the $\mathbf{QH D} (M)  $ algebra will automatically involve information about the kinematics of quantum gravity \cite{Aastrup:2015gba}.
Secondly, the algebra of holonomy-diffeomorphisms produces in a semi-classical limit -- based on a semi-classical state -- an {\it almost commutative algebra} \cite{Aastrup:2012vq}, i.e. the key ingredient in the non-commutative formulation of the standard model of particle physics coupled to general relativity \cite{Connes:1996gi,Chamseddine:2007hz} (see \cite{Boyle:2016cjt} for an interesting recent development). This hints at  a possible connection to the standard model itself.\\

The first question one is faced with when building a theory over the $\mathbf{Q H D} (M)  $ algebra is what Hilbert space representations it has. This question was central to the papers \cite{Aastrup:2014ppa,Aastrup:2015gba,Aastrup:2012vq,Aastrup:2016ytt}. There we first used lattice approximations and later a lattice independent formulation to analyse the question. What we discovered was, however, a simple "no state" argument that seemed to rule out states on $\mathbf{Q H D} (M)  $ -- a finding that made us consider alternative formulations that involved a Dirac type operator and a certain flow-dependent formulation of the $\mathbf{Q H D} (M)  $ algebra \cite{Aastrup:2015gba,Aastrup:2016ytt}.

In this paper we argue, however, that there exist a way around the "no state" argument.  The "no state" argument was based on a functoriality condition with respect to the manifold of the state acting on the translation operators. We argue that it is possible to have states on $\mathbf{Q H D} (M)  $, which are physically realistic and which do not satisfy this functoriality condition. These states will then provide us with a kinematical Hilbert space via the GNS construction.

The outline of this paper is as follows.  In section 2 we introduce the algebra generated by holonomy-diffeomorhisms, denoted $\mathbf{H D} (M)  $, and its extension $\mathbf{QH D} (M)  $ that also include the translation operators over the underlying configuration space of connections. In section 3 we then construct a candidate for a state. Next we link the construction to canonical quantum gravity in section 4 by showing that the canonical commutation relations of canonical quantum gravity formulated in terms of Ashtekar variables emerges from the key algebraic relation of the $\mathbf{QH D} (M)  $ algebra.  Finally,  we end in section 5 with a discussion.

\section{The Quantum holonomy-diffeomorphism algebra}
\label{firsttask}

We start with the holonomy-diffeomorphism algebra $\mathbf{H D} (M)  $, which was first introduced in \cite{Aastrup:2012vq}, and the quantum holonomy-diffeomorphism algebra $\mathbf{QH D} (M) $  as well as its infinitesimal version $\mathbf{dQH D} (M)  $, which  were introduced in \cite{Aastrup:2014ppa} and \cite{Aastrup:2015gba}.

\subsection{The holonomy-diffeomorphism algebra}
\label{beent}

Let $M$ be a compact and connected $3$-dimensional manifold. Consider the vector bundle $S=M\times {\Bbb C}^2$ over $M$ as well as the space of $SU(2)$ connections acting on the bundle. Given a metric $g$ on $M$ we get the Hilbert space $L^2(M,S,dg)$, where we equip $S$ with the standard inner product. Given a diffeomorphism $\phi:M\to M$ we get a unitary operator $\phi^*$ on  $L^2(M,S,dg)$ via
$$( \phi^* (\xi ))(\phi (x) )= (\Delta \phi )(M)  \xi (x) , $$
where  $\Delta \phi (x)$ is the volume of the volume element in $\phi (x)$ induced by a unit volume element in $ x$ under $\phi $.

Let $X$ be a vectorfield on $M$, which can be exponentiated, and let $\nabla$ be a $SU(2)$-connection acting on $S$.  Denote by $t\to \exp_t(X)$ the corresponding flow. Given $x\in M$ let $\gamma$ be the curve
$$\gamma (t)=\exp_{t} (X) (x) $$
running from $x$ to $\exp_1 (X)(x)$. We define the operator
$$e^X_\nabla :L^2 (M , S, dg) \to L^2 (M ,  S , dg)$$
in the following way:
we consider an element $\xi \in L^2 (M ,  S, dg)$ as a ${\Bbb C}^2$-valued function, and define
\begin{equation}
  (e^X_\nabla \xi )(\exp_1(X) (x))=  ((\Delta \exp_1) (x))  \hbox{Hol}(\gamma, \nabla) \xi (x)   ,
  \label{chopin1}
 \end{equation}
where $\hbox{Hol}(\gamma, \nabla)$ denotes the holonomy of $\nabla$ along $\gamma$.
Let ${\cal A}$ be the space of $SU(2)$-connections. We have an operator valued function on ${\cal A}$ defined via
\begin{equation}
{\cal A} \ni \nabla \to e^X_\nabla  .
\nonumber
\end{equation}
We denote this function $e^X$. For a function $f\in C^\infty_c (M)$ we get another operator valued function $fe^X$ on ${\cal A}$. We call this operator a holonomy-diffeomorphisms.
Denote by ${\cal F} ({\cal A} , {\cal B} (L^2(M, S,dg) ))$ the bounded operator valued functions over ${\cal A}$. This forms a $C^*$-algebra with the norm
$$\| \Psi \| =  \sup_{\nabla \in {\cal A}} \{\|  \Psi (\nabla )\| \}, \quad \Psi \in  {\cal F} ({\cal A} , {\cal B} (L^2(M, S,dg )) ). $$

\begin{definition}
Let
$$C =   \hbox{span} \{ fe^X |f\in C^\infty_c(M), \ X \hbox{ exponentiable vectorfield }\}  . $$
The holonomy-diffeomorphism algebra $\mathbf{H D} (M,S,{\cal A})   $ is defined to be the $C^*$-subalgebra of  ${\cal F} ({\cal A} , {\cal B} (L^2(M,S,dg )) )$ generated by $C$.
We will often denote $\mathbf{H D} (M,S,{\cal A})   $ by  $\mathbf{H D}  (M)$ when it is clear which $S$ and ${\cal A}$ is meant.
\end{definition}

It was shown in \cite{AGnew} that  $\mathbf{H D} (M,S,{\cal A})   $ is independent of the metric $g$. \\

\subsection{The quantum holonomy-diffeomorphism algebra}

Let $\mathfrak{su}(2)$ be the Lie-algebra of $SU(2)$.
A section $\omega \in \Omega^1(M,\mathfrak{su}(2))$ induces a transformation of ${\cal A}$, and therefore an operator $U_\omega $ on $\mathcal{F}({\cal A},  {\cal B}(L^2 (M ,  S,g)))$ via
$$U_\omega (\xi )(\nabla) = \xi (\nabla - \omega) ,$$
which satisfy the relation
\begin{equation} \label{konj}
(U_{\omega} fe^X U_{ \omega}^{-1}) (\nabla) = fe^X (\nabla + \omega )  .
\end{equation}
Infinitesimal translations on ${\cal A}$ are given by
\begin{equation}
E_\omega  =\frac{d}{dt}U_{  t  \omega}\Big|_{t=0} \;,
\label{soevnloes}
\end{equation}
where we note that
$$
E_{\omega_1+\omega_2}=E_{\omega_1}+E_{\omega_2\;,}
$$
which follows since the map $\Omega^1 (M,\mathfrak{su}(2))\ni \omega \to U_{ \omega}$ is a group homomorphism, i.e. $U_{(\omega_1+\omega_2 )}=U_{\omega_1}U_{ \omega_2}$.

We define the $\mathbf{QHD}(M)$ as the algebra generated by elements in $\mathbf{HD}(M)$ and by translations $U_{\omega}$. We define the infinitesimal quantum holonomy-diffeomorphism algebra $\mathbf{dQHD}(M)$ as the algebra generated by elements in $\mathbf{HD}(M)$ and by infinitesimal translations $E_{\omega}$.\\

Elements of $\mathbf{QHD}(M)$ can be written in a canonical form. To see this we first define
$$
e^X_\omega := U^{-1}_\omega e^X U_\omega
$$
which permits us to write any combination of $U_\omega$'s and $fe^X$'s in the canonical form
$$
 U_{\omega_1} f_2 e^X_{\omega_2}\ldots f_n e^X_{\omega_n}
$$
for some $\omega_i\in \Omega^1(M,\mathfrak{su}(2))$.
Note that we have the relations
$$
e^X_{\omega_1} U_{\omega_2} = U_{\omega_2} e^X_{\omega_1 + \omega_2}
$$
and
$$
\left( U_{\omega_1} e^X_{\omega_2} \right)^* = U_{-\omega_1} e^{-X}_{-\omega_1 + \omega_2}\;.
$$

\section{States on $\mathbf{QHD}(M)$}

We will now consider states on $\mathbf{QHD}(M)$.
We think of a state as a map
\begin{equation}
\rho: \mathbf{QHD}(M)  \rightarrow M_2(\mathbb{C})\otimes {\cal F}(M\times M)
\label{morgensang}
\end{equation}
where we obtain the actual state by composing $\rho$ with the map
\begin{eqnarray}
&&\Phi_{\psi}: M_2(\mathbb{C})\otimes {\cal F}(M\times M) \rightarrow \mathbb{C} ,
\nonumber\\
&&\hspace{0.8cm} K(x,y)   \rightarrow   \int_{M\times M}     \bar{\psi}(x) K(x,y)  \psi(y)dxdy,
\label{thy}
\end{eqnarray}
where $\psi$ is a $\mathbb{C}^2$-valued half-density on $M$. Alternatively a 'vacuum' state can be obtained via
\begin{equation}
K(x,y)   \rightarrow   \int_{M\times M}   Tr_{M_2}  K(x,y)\delta^{3}(x-y) dxdy,
\label{thedoors}
\end{equation}
where it is implicit understood that we integrate over appropriate half-densities.

Let ${\bf A}$ be a map from $TM$ to $M_2({\Bbb C})$, whose properties will be specified shortly.
We specify the map $\rho_{\bf A}^\kappa$, which is to be a state on $\mathbf{QHD}(M)$, first with
\begin{eqnarray}
\rho_{\bf A}^\kappa( f e^X_\omega) (x,y) = f(x) \hbox{Hol}(\gamma,{\bf A}+\omega) \delta^{3}(y-\exp(X)(x))
\label{hvisker1}
\end{eqnarray}
where $\gamma$ is the curve in $M$ generated by $X$ and which connects $x$ and $y$. The general expression reads
\begin{eqnarray}
\rho_{\bf A}^\kappa(    e^{X_1}_{\omega_1}\ldots  e^{X_n}_{\omega_n}   ) (x,y)
&=&  \hbox{Hol}(\gamma_2,{\bf A}+\omega_2)\ldots \hbox{Hol}(\gamma_n,{\bf A}+\omega_n)
\nonumber\\&&
\cdot\delta^{3}(y-\exp(X)(x))
\end{eqnarray}
where $\gamma_i$ is the path generated by the vector field $X_i$. We write $X$ as a shorthand for the combination of all the involved vector fields.
We then write the state as
\begin{eqnarray}
\rho^\kappa_{(\psi,{\bf A})}=\Phi_{\psi}\circ \rho_{\bf A}^\kappa\;.
\label{tristese}
\end{eqnarray}
Next, we need to specify the state on operators, which have the general structure

 U_{\omega_1} e^{X_2}_{\omega_2}\ldots  e^{X_n}_{\omega_n}
$.
There appears to be two ways to construct such a state. The first is to write
\begin{eqnarray}
\hspace{-1cm}1)\quad\rho_{(\psi,{\bf A})}^\kappa(   U_{\omega_1} e^{X_2}_{\omega_2}\ldots  e^{X_n}_{\omega_n}   )
= \Omega_{\bf A}^\kappa(  \omega_{1})   \rho_{(\psi,{\bf A},{\bf E})}^\kappa  (   e^{X_2}_{\omega_2}\ldots  e^{X_n}_{\omega_n}   )
\label{NOET}
\end{eqnarray}
where $ \Omega_{\bf A}^\kappa(  \omega) $ is a function that satisfies
$$
0\leq\left\vert \Omega_{\bf A}^\kappa(\omega) \right\vert \leq1\;,\quad \Omega_{\bf A}^\kappa(0)=1\;.
$$
The second way to construct a state is to let $\Omega_{\bf A}^\kappa$ be also a function on $M$. In the following we shall use the same symbol $ \Omega_{\bf A}^\kappa $ for both cases and trust that no confusion will arise. For the second option we write
\begin{eqnarray}
\hspace{-1,1cm}2)\quad\rho_{\bf A}^\kappa(   U_{\omega_1} e^{X_2}_{\omega_2}\ldots  e^{X_n}_{\omega_n}   ) (x,y)
&&\nonumber\\
&&\hspace{-2cm}= \Omega_{\bf A}^\kappa(  \omega_{1}) (x)  \rho_{\bf A}^\kappa  (   e^{X_2}_{\omega_2}\ldots  e^{X_n}_{\omega_n}   ) (x,y)
\label{NOTO}
\end{eqnarray}
and then construct the state via (\ref{tristese}). If we employ the latter method it is the integral of $\vert\Omega_{\bf A}^\kappa(\omega)\vert$ that must lie between zero and one and equal one when evaluated on $\omega=0$.

Regardless of which way the candidate for a state is constructed the function $\Omega_{\bf A}^\kappa $ is the key element to understand it. But before we turn our attention hereto let us first consider the element
${\bf A}$.
We write ${\bf A}$ as
$$
{\bf A} = \mathds{A} + \mathds{A}_q
$$
where $\mathds{A}$ is a one-form, which takes values in $\mathfrak{su}(2)$.
$
\mathds{A}_
$
is a map $TM\rightarrow M_2(\mathbb{C})$ that satisfies the following homogeneity condition
$$
\mathds{A}_q(\lambda X) = \vert \lambda \vert \mathds{A}_q( X)\;,\quad \lambda\in  \mathbb{C}
$$
and where $\mathds{A}_q( X) $ is a negative definite element in $M_2(\mathbb{C})$.
Here $\kappa$ enters as a quantization parameter, which separates the classical contribution from its 'quantum' counterparts.

The idea is  to interpret $\mathds{A}$ as an Ashtekar connection. The conjugate Ashtekar variables, the densitised triad field, must then emerge from $\Omega_{\bf A}^\kappa$ in the semi-classical limit.

Let us now return to the function $\Omega_{\bf A}^\kappa$ introduced in (\ref{NOET}) and (\ref{NOTO}). If we first consider the second option, then the function $\Omega_{\bf A}^\kappa$ cannot be a local function on $M$ of $\omega$, since this would bring the state in conflict with the $\mathbf{HD}(M)$ algebra. The problem being the following: an $\omega$ which is zero on  $M$ apart from a  region with small volume, would have $\rho^k_{(\psi,{\bf A})}(U_\omega)$ almost equal to one, i.e. if one considers $\mathbb{A}$ or $\mathbb{A}+\omega$ then they would basically define the same state. However since the holonomy of the flows only involve a one-dimensional integral, these integrals could be "big" running through that region,  i.e. the holonomy on  $\mathbb{A}$ and $\mathbb{A}+\omega$ would be very different contradicting that $\rho^k_{(\psi,{\bf A})}(U_\omega)$ is almost equal to one.
Also when  $\Omega_{\bf A}^\kappa$ were local, it satiesfy properties  similar to the one used in  the argument made in section 6 in \cite{Aastrup:2016ytt}, which lead to the conclusion that under these assumptions there cannot be a non-trivial state on the $\mathbf{QHD}(M)$ algebra. One of the assumptions made in this argument was that $\rho_{(\psi,{\bf A})}^\kappa(   U_{\omega} )$ behaves functorial with respect to restriction of $\omega$ on $M$.
If, however, $\Omega_{\bf A}^\kappa(\omega)$ is not a local function, then this argument does not apply. This observation also applies to the first option (\ref{NOET}), where it implies that $\Omega_{\bf A}^\kappa$ cannot be an integral over $M$ of a local function.

This suggests that $\Omega_{\bf A}^\kappa(\omega)$ involves derivations of $\omega$ and that it will effectively introduce a kind of spatial cut-off that involves a scale. This will then ensure that $\Omega_{\bf A}^\kappa$ does not behave functorially with respect to $M$ on short scales.

Note that the physically interesting state seem to arise only from the second of the above options, where $\Omega_{\bf A}^\kappa$ is a function on $M$ that interacts with the holonomy-diffeomorphisms. The reason for this is that the conjugate variables -- as we shall show in the next section -- i.e. connections and inverse triad fields, come from the holonomy-diffeomorhisms and the translation operators respectively. If these two variables do not interact within the same integral it seems impossible to form operators, which correspond to physical quantities such as the Dirac and gravitational Hamiltonians.

 It is beyond the scope of this paper to analyse what condition $ \Omega_{\bf A}^\kappa$ must satisfy in order to have a state. We suspect that the definition of infinitesimal operators and the computation of the constraint algebra and the requirement to have off-shell closure\footnote{see \cite{Aastrup:2015gba} for computations in the lattice formulation.} will have an impact on what $ \Omega_{\bf A}^\kappa $ may look like. We shall address this key issue in a later publication.

\section{Connection to canonical quantum gravity}
\label{session}

Let us end with a section on the connection between the $\mathbf{QHD}(M)$ algebra and canonical quantum gravity. The following is a rehash of material published in \cite{Aastrup:2014ppa}.

If we combine equation (\ref{konj}) with (\ref{soevnloes}) we obtain
\begin{equation} \label{flowkan}
[ E_\omega , e^X_\nabla ]= \left.\frac{d}{dt}e^X_{\nabla +t\omega}\right\vert_{t=0}  .
\end{equation}
To analyse the righthand side of (\ref{flowkan}) we introduce local coordinates $(x_1 ,x_2,x_3)$ and write $\omega =\omega^i_\mu\sigma_i dx^\mu$.
For a given point $p\in M$ choose the points $$p_0=p,\quad p_1=e^{\frac{1}{n}X}(p),\ldots  ,\quad  p_n= e^{\frac{n}{n}X}(p)$$
on the path
$$t\to e^{tX}(p)  ,t\in [0,1].$$
We write the vectorfield $X=X^\nu\partial_\nu$.
We have
\begin{eqnarray}
\lefteqn{e^X_{\nabla+t\omega}}\nonumber\\
& =&\lim_{n\to \infty } (1+\frac{1}{n}(A(X(p_0))+t \omega^i_\mu \sigma_i X^\mu(p_0) ) (1+\frac{1}{n}(A(X(p_1))+t \omega^i_\mu \sigma_iX^\mu(p_1))\nonumber\\
&& \cdots (1+\frac{1}{n}(A(X(p_n))+t \omega^i_\mu \sigma_i X^\mu(p_n)) ,
\label{vlad}
\end{eqnarray}
where  $\nabla=d+A$, and therefore
\begin{eqnarray}
 \lefteqn{\frac{d}{dt}e^X_{\nabla +t\omega}\Big\vert_{t=0}}
 \nonumber
 \\
 &=& \lim_{n\to \infty }  \Big( \frac{1}{n} \omega^i_\mu\sigma_iX^\mu(p_0)  (1+\frac{1}{n}A(X(p_1)))\cdots (1+\frac{1}{n}A(X(p_n))) \nonumber\\
 &&+ (1+\frac{1}{n}A(X(p_0))) \frac{1}{n}  \omega^i_\mu \sigma_iX^\mu(p_1)  (1+\frac{1}{n}A(X(p_2)))\cdots (1+\frac{1}{n}A(X(p_n))) \nonumber\\
 && + \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \vdots \nonumber\\
 &&+ (1+\frac{1}{n}A(X(p_0)))   (1+\frac{1}{n}A(X(p_2)))\cdots
 \nonumber\\
 &&\hspace{4cm}\cdots
 (1+\frac{1}{n}A(X(p_{n-1}))) \frac{1}{n}  \omega^i_\mu \sigma_i X^\mu(p_n) \Big).
\label{COOM}
\end{eqnarray}
If we restrict this to a path $\gamma(t)$, $t\in [a,b]$, generated by the vector field $X$ equation (\ref{COOM}) gives a line integral of the form
\begin{equation}
\int dt Hol ({\gamma_{<t} },\nabla)  \omega (\dot{\gamma}(t)) Hol( {\gamma_{>t} },\nabla)
\label{omega}
\end{equation}
where $\gamma_{<t}$ is the path $[a,t] \ni \tau\to \gamma(\tau)$ and where $\gamma_{>t}$ is the path $[t,b] \ni \tau\to \gamma(\tau)$.

Equation (\ref{COOM}) and (\ref{omega}) should be compared to the classical setup of Ashtekar variables and holonomies of Ashtekar connections \cite{AL1}. There we have canonically conjugate variables $(\mathds{E}^\mu_i , \mathds{A}_\nu^j)$ where indices $\{i,j,k,...\}$ are $\mathfrak{su}(2)$ indices and $\{\mu,\nu,...\}$ are indices labelling a coordinate system on $M$. $\mathds{E}$ is a densitized inverse triad field $\mathds{E}^\mu_i= e e^\mu_i$ where $e^\mu_i$ is the inverse triad field and $e$ its determinant. $\mathds{A}$ is the Ashtekar connection\footnote{Note again that we here work with $SU(2)$ connections which in a canonical framework correspond to either a Euclidian signature or a Hamiltonian with a comparably more complicated structure, see for instance \cite{AL1}. }. If one considers instead of $\mathds{E}$ its flux over a two-surface $S$
\begin{equation}
F^S_i =\int_S \epsilon_{\mu\nu\rho}  \mathds{E}_i^\mu dx^\nu dx^\rho
\label{flux}
\end{equation}
then the Poisson bracket between the holonomy of $\mathds{A}$ along a curve $\gamma$ and $F^S_i$ reads \cite{AL1}
$$
\left\{  F^S_i, Hol(\gamma,\mathds{A})   \right\}_{PB} = \iota(S,\gamma)  Hol(\gamma_1,\mathds{A}) \sigma_i Hol(\gamma_2,\mathds{A})
$$
where $\gamma = \gamma_1 \cdot \gamma_2$ and where the Pauli matrix is inserted at the point of intersection between $S$ and $\gamma$. $\iota(S,\gamma)=\pm 1$ or $0$ encodes information on the intersection of $S$ and $\gamma$.

We therefore see that before taking the limit $\lim_{n\to \infty}$ of (\ref{COOM}) we have simply the commutator of the sum of the flux operators $\sum_k \frac{1}{n}X^\mu (p_k) F^{S_k}_i $, where $S_k$ is the plane orthogonal to the $x_\mu$-axis intersecting $p_k$, and the holonomy operator of the path $$t\to e^{tX}(p)  ,t\in [0,1].$$
It follows that $E_{\sigma_i dx^\mu}$ is  a series of flux-operators $F^S_i$ sitting along the path $$t\to e^{tX}(p)  ,t\in [0,1],$$
where the surfaces $S$ are just the planes othogonal to the $x_\mu$ direction.
But since there are infinitely many of them, they have been weighted with the infinitesimal length, i.e. with a $dx^\mu$
We can formally write this as
$$
E_{\sigma_i dx^\mu} = \int_M \hat{F}^S_i dx^\mu  \;,
$$
where $ \hat{F}^S_i$ is an operator, which corresponds to a quantization of the flux operator (\ref{flux}).
This provides us with a solid interpretation of the $\mathbf{QHD}(M)$ algebra in terms of canonical quantum gravity, where the operator $E_\omega$ is a global flux operator.

Note, however, that $E_\omega$ transforms as a one-form. This means that $E_\omega$ will {\it not}, when we incorporate a classical correspondence via a state, correspond to the Ashtekar variable $\mathds{E}^\mu_i$ but rather to the inverse triad field $e^\mu_i$. The reason for this is that $\mathds{E}$ is a densitised field and thus does not transform as a one-form. This corresponds to the fact that the determinant $e$ emerges in our framework via the state, which is based on half-densities on $M$. This, in turn, implies that we are dealing with a non-canonical framework.

Finally, we can also make the holonomies infinitesimal in order to see the canonical commutation relations between the Ashtekar variables directly. This was done in \cite{Aastrup:2015gba} and shall not be done here.

\section{Discussion}

We have presented an alternative formulation of {\it quantum holonomy theory} based on a Hilbert space representation of the $\mathbf{QHD}(M)$ algebra. In previous publications we believed that such representations were ruled out by a certain functoriality condition, but in these pages we argue that there exist a way to construct physically interesting states, that do not satisfy this condition.
The result will be a more natural formulation of the theory, where it is the $\mathbf{QHD}(M)$ algebra itself -- and not an infinitesimal and flow-dependent variant thereof \cite{Aastrup:2016ytt} -- that provides the quantum variables.

It remains, however, to analyse whether such a state in fact exist. This comes down to analysing what conditions $ \Omega_{\bf A}^\kappa$ must satisfy and whether such conditions can be met. We expect that the required non-locality of $ \Omega_{\bf A}^\kappa$ will introduce a scale and possible a whole sequence of free parameters. We also expect that the formulation of infinitesimal operators with a correct semi-classical limit will further restrict the form of $ \Omega_{\bf A}^\kappa$.

Note the difference between the $\mathbf{HD}(M)$ and $\mathbf{QHD}(M)$ algebra with respect to gauge transformations. The spectrum of the $\mathbf{HD}(M)$ algebra is given by certain connections modulus unitary transformations, i.e. gauge transformations. The $\mathbf{QHD}(M)$ algebra, on the other hand, involves gauge transformations. We suspect that this difference will play a role when analysing whether a state exist on $\mathbf{QHD}(M)$.

This key issue aside, an interesting aspect of quantum holonomy theory is the way it deals with the square root of the determinant of the metric. The conjugate Ashtekar variable is an inverse {\it densitized} triad field, which means -- when incorporated in a quantum theory -- that one has to deal with various powers of an operator that corresponds to the square root of the determinant of the metric, something that causes serious difficulties. In the present approach the square root of the determinant of the metric arises not from the quantum operators but from the state via integrals over half-densities. This implies first of all that quantum holonomy theory departs from a canonical quantization scheme. It is something very similar, but nevertheless different. Secondly it means that the construction of operators, that correspond for example to the Hamilton and diffeomorphism constraints, should be much simpler.

Finally, a key aspect of the theory, which we would like to emphasize, is that the state, over which the kinematical Hilbert space is build, is semi-classical. This means that this approach to a theory of quantum gravity {\it automatically} includes a semi-classical limit. This aspect is in great contrast to other approaches to non-perturbative approaches to quantum gravity \cite{AL1}, where the semi-classical limit is a serious challenge.

redpajama_set_name: RedPajamaArXiv