INTELLIGIBLE LANGUAGE MODELING WITH INPUT SWITCHED AFFINE NETWORKS
The computational mechanisms by which nonlinear recurrent neural networks (RNNs) achieve their goals remains an open question. There exist many problem domains where intelligibility of the network model is crucial for deployment. Here we introduce a recurrent architecture composed of input-switched affine transformations, in other words an RNN without any nonlinearity and with one set of weights per input. We show that this architecture achieves near identical performance to traditional architectures on language modeling of Wikipedia text, for the same number of model parameters. It can obtain this performance with the potential for computational speedup compared to existing methods, by precomputing the composed affine transformations corresponding to longer input sequences. As our architecture is affine, we are able to understand the mechanisms by which it functions using linear methods. For example, we show how the network linearly combines contributions from the past to make predictions at the current time step. We show how representations for words can be combined in order to understand how context is transferred across word boundaries. Finally, we demonstrate how the system can be executed and analyzed in arbitrary bases to aid understanding.
1 INTRODUCTION
Neural networks and the general field of deep learning have made remarkable progress over the last few years in fields such as object recognition (Krizhevsky et al., 2012), language translation (Sutskever et al., 2014), and speech recognition (Graves et al., 2013). For all of the success of the deep learning approach however, there are certain application domains in which intelligibility of the system is an essential design requirement. One commonly used example is the necessity to understand the decisions that a self-driving vehicle makes when avoiding various obstacles in its path. Another example is the application of neural network methodologies to scientific discovery (Mante et al., 2013). Even where intelligibility is not an overt design requirement, it is fair to say that most users of neural networks would like to better understand the models they deploy.
There are at least two approaches to creating intelligible network models. One approach is to build networks as normal, and then apply analysis techniques after training. Often this approach yields systems that perform extremely well, and whose intelligibility is limited. A second approach is to build a neural network where intelligibility is an explicit design constraint. In this case, the typical result is a system that can be understood reasonably well, but may underperform. In this work we follow this second approach and build intelligibility into our network model, yet without sacrificing performance for the task we studied.
Designing intelligibility into neural networks for all application domains is a worthy, but daunting goal. Here we contribute to that larger goal by focusing on a commonly studied task, that of character based
∗This work was performed as an intern at Google Brain. †Work done as a member of the Google Brain Residency program (g.co/brainresidency) ‡Work performed when author was a visiting faculty at Google Brain.
language modeling. We develop and analyze a model trained on a one-step-ahead prediction task of the Text8 dataset, which is 10 million characters of Wikipedia text (Mahoney, 2011). The model we use is a switched affine system, where the input determines the switching behavior by selecting a transition matrix and bias as a function of that input, and there is no nonlinearity. Surprisingly, we find that this simple architecture performs as well as a vanilla RNN, Gated Recurrent Unit (GRU) (Cho et al., 2014), IRNN (Le et al., 2015), or Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) in this task, despite being a simpler and potentially far more computationally efficient architecture.
In what follows, we discuss related work, define our Input Switched Affine Network (ISAN) model, demonstrate its performance on the one-step-ahead prediction task, and then analyze the model in a multitude of ways, most of which would be currently difficult or impossible to accomplish with modern nonlinear recurrent architectures.
2 RELATED WORK
Work by the authors of (Karpathy et al., 2015) attempted to use character-based language modeling to begin to understand how the LSTM (Hochreiter & Schmidhuber, 1997) functions. In it, they employ n-gram word models to highlight what the LSTM has – and has not – learned about the text corpus. They were able to break down LSTM language model errors into classes, such as e.g., "rare word" errors. The authors of (Greff et al., 2015) engaged in a large study to understand the relative importance of the various components of an LSTM. The authors of (Collins et al., 2016) performed an enormous hyperparameter study to disentangle the effects of capacity and trainability in a number of RNN architectures.
Attempts to understand networks in more general contexts include the use of linearization and nonlinear dynamical systems theory to understand RNNs in (Sussillo & Barak, 2013). In feedforward networks the use of linear probes has been suggested by (Alain & Bengio, 2016), and there exist a host of back-propagation techniques used to infer the most important input to drive various components of the feed-forward network, e.g. (Le et al., 2012).
The ISAN uses an input-switched affine model. The highly related linear time-varying systems are standard material in undergraduate electrical engineering text books. Probabilistic versions of switching linear models with discrete latent variables have a history in the context of probabilistic graphical models. A recent example is the switched linear dynamical system in (Linderman et al., 2016). Focusing on language modeling, (Belanger & Kakade, 2015) defined a probabilistic linear dynamical system as a generative language model for creating context-dependent token embeddings and then used steady-state Kalman filtering for inference over token sequences. They used singular value decomposition and discovered that the right and left singular vectors were semantically and syntactically related. One difference between the ISAN and the LDS is that the weight matrices of the ISAN are input token dependent (while the biases of both models are input dependent). Finally, multiplicative neural networks (MRNNs) were proposed precisely for character based language modeling in (Sutskever et al., 2011; Martens & Sutskever, 2011). The MRNN architecture is similar to our own, in that the dynamics matrix switches as a function of the input character. However, the MRNN relied on a tanh nonlinearity, while our model is explicitly linear. It is this property of our model which makes it both amenable to analysis, and computationally efficient.
The Observable Operator Model (OOM) (Jaeger, 2000) is similar to the ISAN in that the OOM updates a latent state using a separate transition matrix for each input symbol and performs probabilistic sequence modeling. Unlike the ISAN, the OOM requires that a linear projection of the hidden state corresponds to a normalized sequence probability. This imposes strong constraints on both the model parameters and the model dynamics, and restricts the choice of training algorithms. In contrast, the ISAN applies an affine readout to the hidden state to obtain logits, which are then pushed through a SoftMax to obtain probabilities. Therefore no constraints need to be imposed on the ISAN’s parameters and training is easy using backprop. Lastly, the ISAN is formulated as an affine, rather than linear model. While this doesn’t change the class of processes that can be modeled, it enhances the stability of training and greatly enhances interpretability. We elaborate upon these ideas in Section 6.1.
3 METHODS
3.1 MODEL DEFINITION
In what follows Wx and bx respectively denote a transition matrix and a bias vector for a specific input x, the symbol xt is the input at time t, and ht is the hidden state at time t. Our ISAN model is defined as
ht = Wxt ht−1 + bxt . (1)
The network also learns an initial hidden state h0. We emphasize the intentional absence of any nonlinear activation function.
3.2 CHARACTER LEVEL LANGUAGE MODELLING WITH RNNS
The RNNs are trained on the Text8 Wikipedia dataset, for one-step-ahead character prediction. The Text8 dataset consists only of the 27 characters ‘a’-‘z’ and ‘_’ (space). Given a character sequence of x1, ...,xt, the RNNs are trained to minimize the cross-entropy between the true next character, and the output prediction. We map from the hidden state, ht, into a logit space via an affine map. The probabilities are computed as
p (xt+1) = softmax (lt) (2) lt = Wro ht + bro, (3)
where Wro and bro are the readout weights and biases, and lt is the logit vector. In line with (Collins et al., 2016) we split the training data into 80%, 10%, and 10% for train, test, and evaluation set respectively. The network was trained with the same hyperparameter tuning infrastructure as in (Collins et al., 2016). Analysis in this paper is carried out on the best-performing ISAN model, which has 1, 271, 619 parameters, corresponding to 216 hidden units, and 27 dynamics matrices Wx and biases bx.
4 RESULTS AND ANALYSIS
4.1 ISAN PERFORMANCE ON THE TEXT8 TASK
The results on Text8 are shown in Figure 1a. For the largest parameter count, the ISAN matches almost exactly the performance of all other nonlinear models with the same number of maximum parameters: RNN, IRNN, GRU, LSTM. However, we note that for small numbers of parameters the ISAN performs considerably worse than other architectures. All analyses use ISAN trained with 1.28e6 maximum parameters (1.58 bpc cross entropy). Samples of generated text from this model are relatively coherent. We show two examples, after priming with "annual reve", at inverse temperature of 1.5, and 2.0, respectively:
• “annual revenue and producer of the telecommunications and former communist action and saving its new state house of replicas and many practical persons”
• “annual revenue seven five three million one nine nine eight the rest of the country in the united states and south africa new”.
As a preliminary, comparative analysis, we performed PCA on the state sequence over a large set of sequences for the vanilla RNN, GRU of varying sizes, and ISAN. This is shown in Figure 1b. The eigenvalue spectra, in log of variance explained, was significantly flatter for the ISAN than the other architectures.
We also compare the ISAN performance to a fully linear RNN without input switched dynamics. This achieves a cross-entropy of 3.1 bits / char, independent of network size. This perplexity is only slightly better than that of a Naive Bayes model on the task, at 3.3 bits / char. The output probability of the fully linear network is a product of contributions from each previous character, as in Naive Bayes. Those factorial contributions are learned however, giving ISAN a slight advantage. We also run a comparison to a fully linear network with a non-linear readout. This achieves 2.15 bits /
char, independent of network size. Both of these comparisons illustrate the importance of the input switched dynamics for achieving good results in the absence of non-linear hidden state dynamics.
Lastly we also test to what extent the ISAN can deal with large dictionaries by running it on a byte-pair encoding of the text8 task, where the input dictionary consists of the 272 different possible character combinations. We find that in this setup the LSTM consistently outperforms the ISAN for the same number of parameters. At 1.3m parameters the LSTM achieves a cross entropy of 3.4 bits / char-pair, while ISAN achieves 3.55. One explanation for this finding is that the matrices in ISAN are a factor of 27 smaller than the matrices of the LSTMs. For very large numbers of parameters the performance of any architecture saturates in the number of parameters, at which point the ISAN can ‘catch-up’ with more parameter efficient architectures like LSTMs.
4.2 DECOMPOSITION OF CURRENT PREDICTIONS BASED ON PREVIOUS TIME STEPS
Taking advantage of the linearity of the hidden state dynamics for any sequence of inputs, we can decompose the current latent state ht into contributions originating from different timepoints s in the history of the input:
ht = t∑ s=0
( t∏
s′=s+1
Wxs′ ) bxs , (4)
where the empty product when s + 1 > t is 1 by convention, and bx0 = h0 is the learned initial hidden state. This is useful because we can analyze which factors were important in the past, for determining the current character prediction.
Using this decomposition and the linearity of matrix multiplication we can also write the unnormalized logit-vector, lt, as a sum of terms linear in the biases,
lt = bro + t∑ s=0 κts (5)
κts = Wro
( t∏
s′=s+1
Wxs′ ) bxs , (6)
where κts is the contribution from timestep s to the logits at timestep t, and κ t t = bxt . For notational convenience we will sometimes replace the subscript s with the corresponding input character xs at step s when referring to κts – e.g. κ t ‘q’ to refer to the contribution from the character ‘q’ in a string.
Similarly, when discussing the summed contributions from a word or substring we will sometimes write κtword to mean the summed contributions of all the κ t s from that source word, ∑ s∈word κ t s – e.g. κt‘the’ to refer to the total logit contribution from the word ‘the’.
While in standard RNNs the nonlinearity causes interdependence of the bias terms across time steps, in the ISAN the bias terms can be interpreted as independent linear contributions to the state that are propagated and transformed through time. We emphasize that κts includes the multiplicative contributions from the Wxs′ for s < s
′ ≤ t. It is however independent of prior inputs, xs′ for s′ < s. This is the main difference between the analysis we can carry out with the ISAN compared to a non-linear RNN. In general the contribution of a specific character sequence will depend on the hidden state at the start of the sequence. Due to the linearity of the dynamics, this dependency does not exist in the ISAN.
In Figure 2 we show an example of how this decomposition allows us to understand why a particular prediction is made at a given point in time, and how previous characters influence the decoding. For example, the sequence ‘_annual_revenue_’ is processed by the ISAN: Starting with an all-zero hidden state, we use equation (6) to accumulate a sequence of κt‘_′ ,κ t ‘a′ ,κ t ‘n′ ,κ t ‘n′ , .... These values can then be used to understand the prediction of the network at some time t, by simple addition across the s index, which is shown in Figure 2.
In Figure 3 we provide a detailed view of how past characters contribute to the logits predicting the next character. There are two competing options for the next letter in the word stem ‘reve’: either ‘revenue’ or ‘reverse’. We show that without the contributions from ‘_annual’ the most likely decoding of the character after the second ‘e’ is ‘r’ (to form ‘reverse’), while the contributions from ‘_annual’ tip the balance in favor of ‘n’, decoding to ‘revenue’. In a standard RNN a similar analysis could be carried out by comparing the prediction given an artificially limited history.
Using the decomposition of current step predictions in to κts, we can also investigate how quickly the contributions of κts decay as a function of t− s. In Figure 4a we can see that this contribution decays on two different exponential timescales. We hypothesize that the first time scale corresponds to the decay within a word, while the next corresponds to the decay of information across words and sentences. This effect is also visible in Figure 5. We note that it would be difficult to carry out this analysis in a non-linear RNN.
We can also show the relevance of the κts contributions to the decoding of characters at different positions in the word. For examples, we observe that κt‘_’ makes important contributions to the prediction of the next character at time t. We show that using only the κt‘_’, the model can achieve
a cross entropy of < 1 / char when the position of the character is more than 3 letters from the beginning of the word.
Furthermore we can link back from the norm-decay to the importance of past characters for the decoding quality. By artificially limiting the number of past κ available for prediction, Figure 4c, we show that the prediction quality improves rapidly when extending the history from 0 to 10 characters and then saturates. This rapid improvement aligns with the range of faster decay in Figure 4a.
4.3 FROM CHARACTERS TO WORDS
The ISAN provides a natural means of moving from character level representation to word level. Using the linearity of the hidden state dynamics we can aggregate all of the κts belonging to a given
word and visualize them as a single contribution to the prediction of the letters in the next word. This allows us to understand how each preceding word impacts the decoding for the letters of later words. In Figure 5 we show that the words ‘higher’ and ‘than’ make large contributions to the prediction of the characters ‘h’ and ‘n’ in ‘tevenue’, as measured by the norm of the κt‘_the’ and κt‘_annual’.
In Figure 6 we show that these κtword are more than a mathematical convenience and even capture word-level semantic information. Shown is a t-SNE embedding of the κtword for the most common 4000 words in the data-set, with examples of the kind of clusters that arise.
4.4 CHANGE OF BASIS
We are free to perform a change of basis on the hidden state, and then to run the affine ISAN dynamics in that new basis. Note that this change of basis is not possible for other RNN architectures, since the action of the nonlinearity depends on the choice of basis.
In particular we can construct a ‘readout basis’ that explicitly divides the latent space into a subspace Pro‖ spanned by the rows of the readout matrix Wro, and its orthogonal complement P ro ⊥ . This representation explicitly divides the hidden state dynamics into a 27-dimensional ‘readout’ subspace that is accessed by the readout matrix to make predictions, and a ‘computational’ subspace comprising the remaining 216− 27 dimensions that are orthogonal to the readout matrix. We apply this change of basis to analyze an intriguing observation about the hidden offsets bx: As shown in Figure 7, the norm of the bx is strongly correlated to the log-probability of the unigram x in the training data. Re-expressing network parameters using the ‘readout basis’ shows that this correlation is not related to reading out the next-step prediction. This is because the norm of the projection of bx into Pro⊥ remains strongly correlated with character frequency, while the projections into Pro‖ have norms that show little correlation. This indicates that the information content or surprise of a letter is encoded through the norm of the component of bx in the computational space, rather than in the readout space.
Similarly, in Figure 8 we illustrate that the structure in the correlations between the bx is due to their components in Pro‖ , while the correlation in P ro ⊥ is relatively uniform. We can clearly see two blocks of high correlations between the vowels and consonants respectively, while b‘_’ is uncorrelated to either.
4.5 COMPARISON WITH n-GRAM MODEL WITH BACKOFF
We compared the computation performed by n-gram language models and those performed by the ISAN. An n-gram model with back-off weights expresses the conditional probability p (xt|x1...xt−1) as a sum of smoothed count ratios of n-grams of different lengths, with the contribution of shorter n-grams down-weighted by backoff weights. On the other hand, the computations performed by the ISAN start with the contribution of bro to the logits, which as shown in Figure 9a) corresponds to the unigram log-probabilities. The logits are then additively updated with contributions from longer n-grams, represented by κts. This additive contribution to the logits corresponds to a multiplicative modification of the emission probabilities from histories of different length. For long time lags, the additive correction to log-probabilities becomes small (Figure 2), which corresponds to multiplication by a uniform distribution. Despite these differences in how n-gram history is incorporated, we
nevertheless observe an agreement between empirical models estimated on the training set and model predictions for unigrams and bigrams. Figure 9 shows that the bias term bro gives the unigram probabilities of letters, while the addition of the offset terms bx accurately predict the bigram distribution of P (xt+1|xt). Shown are both an example, P (x|‘_′), and a summary plot for all 27 letters.
We further explore the n-gram comparison by artificially limiting the length of the character history that is available to the ISAN for making predictions, as shown in Figure 4c).
5 ANALYSES OF PARENTHESES COUNTING TASK
To show the interpretability of the ISAN we train a model on the parenthesis counting task. Bringing together ideas from sections 4.4 and 6.1 we re-express the transition dynamics in a new basis that fully reveals computations performed by the ISAN.
We analyze the simple task of parentheses counting, which was defined in (Collins et al., 2016). Briefly, the RNN is required keep track of the nesting level of 3 different types of parentheses
independently. The inputs are the one-hot encoding of the different opening and closing parentheses (e.g. ‘(’, ‘)’, ‘{’, ‘}’) as well as a noise character (‘a’). The output is the one-hot encoding of the nesting level between (0-5), one set of counts for each parentheses task. One change from (Collins et al., 2016) is that we slightly simplify the problem by exchanging the cross-entropy error with an L2 error and linear readout (this change leads to slightly cleaner figures, but does not qualitatively change the results).
We first re-express the transition dynamics in terms of linear, rather than affine operations. Consider the matrix W′ ∈ R(n+1)×(n+1):
W′ = [ W b 0 1 ] , (7)
where 0 is a row vector of zeros. The matrix W′ emulates the affine transition for any hidden state h ∈ Rn×1: W′ [ h 1 ] = [ Wh+ b 1 ] . (8)
The matrices W and W′ are closely connected. Each eigenvalue of W is also an eigenvalue of W′. Moreover, eigenvectors of W become the eigenvectors of W′ when expanded with a zero dimension. In fact, W′ only has one extra eigenvalue of exactly 1 that is necessary to preserve the last dimension of the expanded hidden state.
To analyze the the parentheses task we analyze W′. The key to understanding how the network solves the parentheses task is to find a change of bases that clarifies the dynamics necessary to count 3 sets of independent parentheses nesting levels. In this case we use a matrix composed of the readout matrix, modified by adding a set of vectors that spans the null space of the readout (including the additional bias dimension).
W′ro =
[ Wro bro
O
] (9)
where O is the orthogonal complement of the subspace spanned by the row vectors of [Wro bro] in an N + 1-dimensional space. We perform the following change of basis of the dynamics matrices,
Wx(ro) ′ = Wro ′Wx ′ (W′ro) −1 (10)
and visualize the results in Fig. 10. Figure 10 shows that this system created delay lines which count the nesting level, with fixed point dynamics at the 0 count (5 count), so that the system stays at both numbers when the input would otherwise increment (decrement) the count. The matrices also implement fixed point dynamics, as implemented via an identity submatrix to preserve the memory of the parenthesis nesting counts when an unrelated symbol enters the system (e.g. The ‘{}’ count is preserved by the ‘(’ matrix when a ‘(’ symbol enters the system).
6 DISCUSSION
In this paper we motivated an input-switched affine recurrent network for the purpose of intelligibility. We showed that a switched affine architecture achieves the same performance, for the same number of maximum parameters, on a language modeling task as do more common RNN architectures, including GRUs and LSTMs. We performed a series of analyses, demonstrating that the simplicity of the latent dynamics makes the trained RNN far easier to understand and interpret.
6.1 BENEFITS OF AFFINE TRANSITIONS OVER LINEAR
ISAN uses affine operators to model state transitions assigned to each input symbol. Following eq. (1) each transition consists of matrix multiplication and bias vector addition. An important question is whether the biases are needed and how the ISAN would be impacted if linear transition operators were used instead of affine ones. The answer is two-fold. First, affine dynamics can be exactly implemented using linear operators in a hidden space expanded by one additional dimension. Therefore, the expressivity of ISAN does not depend on choosing a linear or affine formulation. However, we found that the affine parametrization of transitions is much easier to train. We attempted to train models using only linear transitions, but achieved a loss of only 4.1 bits per character, which
corresponds to the performance of a unigram character model. Second, affine operators are easier to interpret because they permit easy visualization of contributions of each input token on the final network’s prediction, as demonstrated in Section 4.2.
6.2 COMPUTATIONAL BENEFITS
Switched affine networks hold the potential to be massively more computationally and memory efficient for text processing than standard RNNs, as explained in the next two subsections.
6.2.1 SPARSE PARAMETER ACCESS
As shown in Figure 1a, the performance for fixed parameter count is nearly identical between the ISAN and other recurrent networks. However, at each time step, only the parameters associated with a single input are used. For K possible inputs and N parameters, the computational cost per update
step is O ( N K ) , a factor of K speedup over non-switched architectures. Similarly, the number of
hidden units is O (√
N K
) , a factor of K 1 2 memory improvement for storage of the latent state.
6.2.2 COMPOSITION OF AFFINE UPDATES
The memory and computational benefits in Section 6.2.1 are shared by other switched networks. However, ISAN is unique in its ability to precompute affine transformations corresponding to input strings. This is possible because the composition of affine transformations is also an affine transformation. This property is used in Section 4.3 to evaluate the linear contributions of words, rather than characters. This means that the hidden state update corresponding to an entire input sequence can be computed with identical cost to the update for a single character (plus the dictionary lookup cost for the composed transformation). ISAN can therefore achieve very large speedups on input processing, at the cost of increased memory use, by accumulating large lookup tables of the Wx and bx corresponding to common input sequences. Of course, practical implementations will have to incorporate complexities of memory management, batching, etc.
6.3 FUTURE WORK
There are some obvious future directions to this work. Currently, we define switching behavior using an input set with finite and manageable cardinality. Studying word-level language models with enormous vocabularies may require some additional logic to scale. Adapting this model to continuous-valued inputs is another important direction. One approach is to use a tensor factorization similar to that employed by the MRNN (Sutskever et al., 2014). Another is to build a language model which switches on bigrams or trigrams, rather than characters or words, targeting an intermediate number of affine transformations.
Training very large switched linear models has the potential to be extremely fruitful, due both to their improved computational efficiency, and our ability to better understand and manipulate their behavior.
7 ACKNOWLEDGEMENTS
We would like to thank Jasmine Collins for her help and advice, and Quoc Le, David Ha and Mohammad Norouzi for helpful discussions.

Summary:  The authors present a simple RNN with linear dynamics for language modeling. The linear dynamics greatly enhance the interpretability of the model, as well as provide the potential to improve performance by caching the dynamics for common sub-sequences. Overall, the quantitative comparison on a benchmark task is underwhelming. It’s unclear why the authors didn’t consider a more common dataset, and they only considered a single dataset. On the other hand, they present a number of well-executed techniques for analyzing the behavior of the model, many of which would be impossible to do for a non-linear RNN.
Overall, I recommend that the paper is accepted, despite the results. It provides an interesting read and an important contribution to the research dialogue.
Feedback
The paper could be improved by shortening the number of analysis experiments and increasing the discussion of related sequence models. Some of the experiments were very compelling, whereas some of them (eg. 4.6) sort of feels like you’re just showing the reader that the model fits the data well, not that the model has any particularly important property. We trust that the model fits the data well, since you get reasonable perplexity results.
LSTMS/GRUs are great for for language modeling for data with rigid combinatorial structure, such as nested parenthesis. It would have been nice if you compared your model to non-linear methods on this sort of data. Don’t be scared of negative results! It would be interesting if the non-linear methods were substantially better on these tasks.
You should definitely add a discussion of Belanger and Kakade 2015 to the related work. They have different motivations (fast, scalable learning algorithms) rather than you (interpretable latent state dynamics and simple credit assignment for future predictions given past). On the other hand, they also have linear dynamics, and look at the singular vectors of the transition matrix to analyze the model.
More broadly, it would be useful for readers if you discussed LDS more directly. A lot of this comparison came up in the openreview discussion, and I recommend folding this into the paper. For example, it would be useful to emphasize that the bias vectors correspond to columns of the Kalman gain matrix.
One last thing regarding LDS: your model corresponds to Kalman filtering but in an LDS you can also do Kalman smoothing, where state vectors are inferred using the future in addition to the past observations. Could you do something similar in your model?
What if you said that each matrix is a sparse/convex combination of a set of dictionary matrices? This parameter sharing could provide even more interpretability, since the characters are then represented by the low-dimensional weights used to combine the dictionary elements. This could also provide more scalability to word-level problems.


SEQUENCE GENERATION WITH A PHYSIOLOGICALLY PLAUSIBLE MODEL OF HANDWRITING AND RECURRENT MIXTURE DENSITY NETWORKS
The purpose of this study is to explore the feasibility and potential benefits of using a physiological plausible model of handwriting as a feature representation for sequence generation with recurrent mixture density networks. We build on recent results in handwriting prediction developed by Graves (2013), and we focus on generating sequences that possess the statistical and dynamic qualities of handwriting and calligraphic art forms. Rather than model raw sequence data, we first preprocess and reconstruct the input training data with a concise representation given by a motor plan (in the form of a coarse sequence of ‘ballistic’ targets) and corresponding dynamic parameters (which define the velocity and curvature of the pen-tip trajectory). This representation provides a number of advantages, such as enabling the system to learn from very few examples by introducing artificial variability in the training data, and mixing of visual and dynamic qualities learned from different datasets.
1 INTRODUCTION
Recent results (Graves, 2013) have demonstrated that, given a sufficiently large training data-set, Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) Recurrent Mixture Density Networks (RMDNs) (Schuster, 1999) are capable of learning and generating convincing synthetic handwriting sequences. In this study we explore a similar network architecture combined with an intermediate feature representation, given by the parameters of a physiologically plausible model of handwriting: the Sigma Lognormal model (Plamondon, 1995; Plamondon et al., 2014).
In the work by Graves (2013) and subsequent derivations, the RMDN operates on raw sequences of points recorded with a digitizing device. In our approach we preprocess the training data using an intermediate representation that describes a form of “motor program” coupled with a sequence of dynamic parameters that describe the evolution of the pen tip. By doing so, we use a representation that is more concise (i.e. lower in dimensionality), meaningful (i.e. every data point is a high level segment descriptor of the trajectory), and is resolution independent.
This project stems from the observation that human handwriting results from the orchestration of a large number of motor and neural subsystems, and is ultimately produced with the execution of complex and skillful motions. As such we seek a representation that abstracts the complex task of trajectory formation from the neural network, which is then rather focused on a higher level task of movement planning. Note that for the scope of this study, we do not implement text-to-handwriting synthesis (Graves, 2013), but rather focus on the task of generating sequences that possess the statistical and dynamic qualities of handwriting, which can be expanded to calligraphy, asemic handwriting, drawings and graffiti (Berio & Leymarie, 2015; Berio et al., 2016)). In particular, we focus on two distinct tasks: (1) learning and generating motor plans and (2) given a motor plan,
1Goldsmiths College, University of London. Department of Computing. 2École Polytechnique de Montréal, Canada.
predicting the corresponding dynamic parameters that determine the visual and dynamic qualities of the pen trace. We then go on to show that this modular workflow can be exploited in ways such as: mixing of dynamic qualities between data-sets (a form of handwriting “style transfer” ) as well as learning from small datasets (a form of “one shot learning”).
The remainder of this paper is organised as follows: in Section 2, after briefly summarising the background context, we then briefly describe the Sigma Lognormal model and RMDNs; in Section 3 we present the data preprocessing step and the RMDN models that build up our system; in Section 4 we propose various applications of the system, including learning handwriting representations from small datasets and mixing styles.
2 BACKGROUND
Our study is grounded on a number of notions and principles that have been observed in the general study of human movement as well as in the handwriting synthesis/analysis field (known as Graphonomics (Kao et al., 1986)). The speed profile of aiming movements is typically characterised by a “bell shape” that is variably skewed depending on the rapidity of the movement (Lestienne, 1979; Nagasaki, 1989; Plamondon et al., 2013). Complex movements can be described by the superimposition of a discrete number of “ballistic” units of motion, which in turn can each be represented by the classic bell shaped velocity profile and are often referred to as strokes. A number of methods synthesise handwriting through the temporal superimposition of strokes, the velocity profile of which is modelled with a variety of functions including sinusoidal functions (Morasso & Mussa Ivaldi, 1982; Maarse, 1987; Rosenbaum et al., 1995), Beta functions (Lee & Cho, 1998; Bezine et al., 2004), and lognormals (Plamondon et al., 2009). In this study we rely on a family of models known as the Kinematic Theory of Rapid Human Movements, that has been developed by Plamondon et al. in an extensive body of work since the 1990’s (Plamondon, 1995; Plamondon et al., 2014). Plamondon et al. (2003) show that if we consider that a movement is the result of the parallel and hierarchical interaction of a large number of coupled linear systems, the impulse response of such a system to a centrally generated command asymptotically converges to a lognormal function. This assumption is attractive from a modelling perspective because it abstracts the high complexity of the neuromuscular system in charge of generating movements with a relatively simple mathematical model which further provides state of the art reconstruction of human velocity data (Rohrer & Hogan, 2006; Plamondon et al., 2013).
A number of methods have used neural inspired approaches for the task of handwriting trajectory formation (Schomaker, 1992; Bullock et al., 1993; Wada & Kawato, 1993). Similarly to our proposed method, Ltaief et al. (2012) train a neural on a preprocessed dataset where the raw input data is reconstructed in the form of handwriting model parameters. Nair & Hinton (2005) use a sequence of neural networks to learn the motion of two orthogonal mass spring systems from images of handwritten digits for classification purposes. With a similar motivation to ours, Plamondon & Privitera (1996) use a Self Organising Map (SOM) to learn a sequence of ballistic targets, which describe a coarse motor plan of handwriting trajectories. Our method builds in particular on the work of Graves (2013), who describes a system that uses a recurrent mixture density networks (RMDNs) (Bishop, 1994) extended with a LSTM architecture (Hochreiter & Schmidhuber, 1997), to generate synthetic handwriting in a variety of styles.
2.1 SIGMA LOGNORMAL MODEL
On the basis of Plamondon’s Kinematic Theory (Plamondon, 1995), the Sigma Lognormal (ΣΛ) model (Plamondon & Djioua, 2006) describes complex handwriting trajectories via the vectorial superimposition of a discrete number of strokes. With the assumption that curved handwriting movements are done by rotating the wrist, the curvilinear evolution of strokes is described with a circular arc shape. Each stroke is charactersied by a variably assymmetric "bell shape" speed profile, which is described with a (3 parameter) lognormal function. The planar evolution of a trajectory is then described by a sequence of virtual targets {vi}i=mi=1 , which define “imaginary” (i.e. not necessarily located along the generated trajectory) loci at which each consecutive stroke is aimed. The virtual targets provide a low level description of the motor plan for the handwriting trajectory. A smooth trajectory is then generated by integrating the velocity of each stroke over time. The trajectory smoothness can be defined by adjusting the activation-time offset of a given stroke with respect to the
previous stroke, which is denoted with ∆t0i; a smaller time offset (i.e. a greater overlap between lognormal components) will result in a smoother trajectory (Fig. 1, c). The curvature of the trajectory can be varied by adjusting the central angle of each circular arc, which is denoted with θi. Equations and further details for the ΣΛ model can be found in Appendix A.
A sequence of virtual targets provides a very sparse spatial description or “motor plan” for the trajectory evolution. The remaining stroke parameters, ∆t0i and θi, define the temporal, dynamic and geometric features of the trajectory and we refer to those as dynamic parameters.
2.2 RECURRENT MIXTURE DENSITY NETWORKS
Mixture Density Networks (MDN) were introduced by Bishop (1994) in order to model and predict the parameters of a Gaussian Mixture Model (GMM), i.e. a set of means, covariances and mixture weights. Schuster (1999) showed that MDNs could be to model temporal data using RNNs. The author used Recurrent Mixture Density Networks (RMDN) to model the statistical properties of speech, and they were found to be more successful than traditional GMMs. Graves (2013) used LSTM RMDNs to model and synthesise online handwriting, providing the basis for extensions to the method, also used in Ha et al. (2016); Zhang et al. (2016). Note that in the case of a sequential model, the RMDN outputs a unique set of GMM parameters for each timestep t, allowing the probability distribution to change with time as the input sequence develops. Further details can be found in Appendix C.1.
3 METHOD
We operate on discrete and temporally ordered sequences of planar coordinates. Similarly to Graves (2013), most of our results come from experiments made on the IAM online handwriting database (Marti & Bunke, 2002). However, we have made preliminary experiments with other datasets, such as the Graffiti Analysis Database (Lab, 2009) as well as limited samples collected in our laboratory from a user with a digitiser tablet.
As a first step, we preprocess the raw data and reconstruct it in the form of ΣΛ model parameters Section 3.1. We then train and evaluate a number of RMDN models for two distinct tasks:
1. Virtual target prediction. We use the V2V-model for this task. Given a sequence of virtual targets, this model predicts the next virtual target.
2. Dynamic parameter prediction. For this task we trained and compared two model architectures. Given a sequence of virtual targets, the task of these models is to predict the corresponding dynamic parameters. The V2D-model is condititioned only on the previous virtual targets, whereas the A2D-model is conditioned on both the previous virtual targets and dynamic parameters.
We then exploit the modularity of this system to conduct various experiments, details of which can found in Section 4.
3.1 PREPROCESSING: RECONSTRUCTING ΣΛ PARAMETERS
A number of methods have been developed by Plamondon et. al in order to reconstruct ΣΛ-model parameters from digitised pen input data (O’Reilly & Plamondon, 2008; Plamondon et al., 2014; Fischer et al., 2014). These methods provide the ideal reconstruction of model parameters, given a high resolution digitised pen trace. While such methods are superior for handwriting analysis and biometric purposes, we opt for a less precise method (Berio & Leymarie, 2015) that is less sensitive to sampling quality and is aimed at generating virtual target sequences that remain perceptually similar to the original trace. We purposely choose to ignore the original dynamics of the input, and base the method on a geometric input data only. This is done in order to work with training sequences that are independent of sampling rate, and in sight of future developments in which we intend to extract handwriting traces from bitmaps, inferring causal/dynamic information from a static input as humans are capable of (Edelman & Flash, 1987; Freedberg & Gallese, 2007).
Our method operates on a uniformly sampled input contour, which is then segmented in correspondence with perceptually salient key points: loci of curvature extrema modulated by neighbouring contour segments (Brault & Plamondon, 1993; Berio & Leymarie, 2015), which gives an initial estimate of each virtual target vi. We then (i) fit a circular arc to each contour segment in order to estimate the θi parameters and (ii) estimate the ∆t0i parameters by analysing the contour curvature in the region of each key point. Finally, (iii) we iteratively adjust the virtual target positions to minimise the error between the original trajectory and the one generated by the corresponding ΣΛ parameters. For Further details on the ΣΛ parameter reconstruction method, the reader is referred to Appendix B.
3.2 DATA AUGMENTATION
We can exploit the ΣΛ parameterisation to generate many variations over a single trajectory, which are visually consistent with the original, and with a variability that is similar to the one that would be seen in multiple instances of handwriting made by the same writer (Fig. 3) (Djioua & Plamondon, 2008a; Fischer et al., 2014; Berio & Leymarie, 2015). Given a dataset of n training samples, we randomly perturb the virtual target positions and dynamic parameters of each sample np times, which results in a new augmented dataset of size n+ n× np where legibility and trajectory smoothness is maintained across samples. This would not be possible on the raw online dataset, as perturbations for each data-point would eventually result in a noisy trajectory.
3.3 PREDICTING VIRTUAL TARGETS WITH THE V2V-MODEL
The V2V-model is conditioned on a history of virtual targets and given a new virtual target it predicts the next virtual target (hence the name V2V). Note that each virtual target includes the corresponding
pen state — up (not touching the paper) or down (touching the paper). Repeatedly feeding the predicted virtual target back into the model at every timestep allows the model to synthesise sequences of arbitrary length. The implementation of this model is very similar to the handwriting prediction demonstrated by Graves (2013), although instead of operating directly on the digitised pen positions, we operate on the much coarser virtual target sequences which are extracted during the preprocessing step. The details of this model can be found in Appendix C.3
3.4 PREDICTING DYNAMIC PARAMETERS WITH THE V2D AND A2D MODELS
The goal of these models is to predict the corresponding dynamic parameters (∆t0i, θi) for a given sequence of virtual targets. We train and compare two model architectures for this task. The V2Dmodel is conditioned on the history of virtual targets, and given a new virtual target, this model predicts the corresponding dynamic parameters (∆t0i, θi) for the current stroke (hence the name V2D). Running this model incrementally for every stroke of a given virtual target sequence allows us to predict dynamic parameters for each stroke. The implementation of this model is very similar to the V2V-model, and details can be found in Appendix C.4.
At each timestep, the V2D model outputs and maintains internal memory of a probability distribution for the predicted dynamic parameters. However, the network has no knowledge of the parameters that are sampled and used. Hence, dynamic parameters might not be consistent across timesteps. This problem can be overcome by feeding the sampled dynamic parameters back into the model at the next timestep. From a human motor planning perspective this makes sense as, for a given drawing style, when we decide the curvature and smoothness of a stroke we will take into consideration the choices made in previously executed strokes.
The A2D model predicts the corresponding dynamic parameters (∆t0i, θi) for the current stroke conditioned on a history of both virtual targets and dynamic parameters (i.e. all ΣΛ parameters - hence the name A2D). We use this model in a similar way to the V2D model, whereby we run it incrementally for every stroke of a given virtual target sequence. However, internally, at every timestep the predicted dynamic parameters are fed back into the model at the next timestep along with the virtual target from the given sequence. The details of this implementation can be found in Appendix C.5.
4 EXPERIMENTS AND RESULTS
Predicting Virtual Targets. In a first experiment we use the V2V model, trained on the preprocessed IAM dataset, to predict sequences of virtual targets. We prime the network by first feeding it a sequence from the test dataset. This conditions the network to predict sequences that are similar to the prime. We can see from the results (Fig. 4) that the network is indeed able to produce sequences that capture the statistical qualities of the priming sequence, such as overall incline, proportions, and oscillation frequency. On the other hand, we observe that amongst the generated sequences, there are often patterns which do not represent recognisable letters or words. This can be explained by the high variability of samples contained in the IAM dataset, and by the fact that our representation is very concise, with each data-point containing high significance. As a result, the slightest variation in a prediction is likely to cause a large error in the next. To overcome this problem, we train a new model with a dataset augmented with 10× variations as described in Section 3.2. Due to our limited computing resources 1, we test this method on 1/10th of the dataset, which results in a new dataset with the same size as the original, but with a lower number of handwriting specimens with a number of subtle variations per specimen. With this approach, the network predictions maintain statistical similarity with the priming sequences, and patterns emerge that are more evocative of letters of the alphabet or whole words, with fewer unrecognizable patterns (Fig. 4). To validate this result, we also test the model’s performance training it on 1/10th of the dataset, without data augmentation, and the results are clearly inferior to the previous two models. This suggests that the data augmentation step is highly beneficial to the performance of the network.
1We are thus not able to thoroughly test the large network architectures that would be necessary to train on the whole augmented dataset.
Predicting Dynamic Parameters. We first evaluate the performance of both the V2D and A2D models on virtual targets extracted from the test set. Remarkably, although the networks have not been trained on these sequences, both models predict dynamic parameters that result in trajectories that are readable, and are often similar to the target sample. We settle on the A2D model trained on a 3× augmented dataset, which we qualitatively assess to produce the best results (Fig. 5).
We then proceed with applying the same A2D model on virtual targets generated by the V2V models primed on the test set. We observe that the predictions on sequences generated with the augmented dataset are highly evocative of handwriting and are clearly different depending on the priming sequence (Fig. 6, c), while the predictions made with the non-augmented dataset are more likely to resemble random scribbles rather than human readable handwriting (Fig. 6, b). This further confirms the utility of the data augmentation step.
User defined virtual targets. The dynamic parameter prediction models can also be used in combination with user defined virtual target sequences (Fig. 7). Such a method can be used to quickly and interactively generate handwriting trajectories in a given style, by a simple point and click procedure. The style (in terms of curvature and dynamics) of the generated trajectory is determined by the data used to train the A2D model, and by priming the A2D model with different samples, we can apply different styles to the user defined virtual targets.
One shot learning. In a subsequent experiment, we apply the data augmentaion method described in Section 3.2 to enable both virtual target and dynamic prediction models to learn from a small dataset of calligraphic samples recorded by a user using a digitiser tablet. We observe that with a low number of augmentations (50×) the models generate quasi-random outputs, and seem to learn only the left to right trend of the input. With higher augmentation (700×), the system generates outputs that are consistent to the human eye with the input data (Fig. 8). We also train our models using only a single sample (augmented 7000×) and again observe that the model is able to reproduce novel sequences that are similar to the input sample (Fig. 9). Naturally, the output is a form of recombination of the input, but this is sufficient to synthesise novel outputs that are qualitatively similar to the input. It should be noted that we are judging the performance of the one-shot learned models qualitatively, and we may not be testing the full limits of how well the models are able to generalise. On the other hand, these results, as well as the “style transfer” capabilities exposed in following section suggest a certain degree of generalisation.
Style Transfer. Here, with a slight abuse of terminology, we utilise the term "style" to refer to the dynamic and geometric features (such as pen-tip acceleration and curvature) that determine the visual qualities of a handwriting trajectory. Given a sequence of virtual targets generated with the V2V model trained on one dataset, we can also predict the corresponding dynamic parameters with the A2D model trained on another. The result is an output that is similar to one dataset in lettering structure, but possesses the fine dynamic and geometric features of the other. If we visually inspect Fig. 10, we can see that both the sequence of virtual targets reconstructed by the dataset preprocessing method, and the trajectory generated over the same sequence of virtual targets with dynamic parameters learned from a different datasets, are both readable. This emphasises the importance of using perceptually salient points along the input for estimating key-points in the data-set preprocessing step (Section 3.1).
Furthermore, we can perform the same type of operation within a single dataset, by priming the A2D model with the dynamic parameters of a particular training example, while feeding it with the virtual targets of another. To test this we train both (V2V, A2D) models on a corpus containing 5 samples of the same sentence written in different styles and then augmented 1400× (Fig. 11). We envision the utility of such as system in combination with virtual targets interactively specified by a user.
5 CONCLUSIONS AND FUTURE WORK
We have presented a system that is able to learn the parameters for a physiologically plausible model of handwriting from an online dataset. We hypothesise that such a movement centric approach is advantageous as a feature representation for a number of reasons. Using such a representation provides a performance that is similar to the handwriting prediction demonstrated by Graves (2013) and Ha et al. (2016), with a number of additional benefits. These include the ability to: (i) capture both the geometry and dynamics of a hand drawn/written trace with a single representation, (ii) express the variability of different types of movement concisely at the feature level, (iii) demonstrate greater flexibility for procedural manipulations of the output, (iv) mix “styles” (applying curvature and dynamic properties from one example, to the motor plan of another), (v) learn a generative model from a small number of samples (n < 5), (vi) generate resolution independent outputs.
The reported work provides a solid basis for a number of different future research avenues. As a first extension, we plan to implement the label/text input alignment method described in Graves’ original work that should allow us to synthesise readable handwritten text and also to provide a more thorough comparison of the two methods. Our method strongly relies on an accurate reconstruction of the input in the preprocessing step. Improvements should target especially parts of the latter method that depend on user tuned parameters, such as the identification of salient points along the input (which requires a final peak detection pass), and measuring the sharpness of the input in correspondence with salient points.
ACKNOWLEDGMENTS
The system takes as a starting point the original work developed by (Graves, 2013). We use Tensorflow, the open-source software library for numerical computation and deep learning (Abadi et al., 2015), and a rapid implementation was possible thanks to a public domain implementation developed by (Ha, 2015).
A SIGMA LOGNORMAL MODEL
The Sigma Lognormal model (Plamondon & Djioua, 2006) describes complex handwriting trajectories via the vectorial superimposition of lognormal strokes. The corresponding speed profile Λi(t) assumes a variably asymmetric "bell shape" which is described with a 3 parameter lognormal function
Λi(t) = − 1
σi √ 2π(t− t0i) exp
( (ln(t− t0i)− µi)2
2σi2
) (1)
where t0i defines the activation time of a stroke and the parameters µi and σi determine the shape of the lognormal function. µi is referred to as log-time delay and is biologically interpreted as the rapidity of the neuromuscular system to react to an impulse generated by the central nervous system (Plamondon et al., 2003); σi is referred to as log-response time and determines the spread and asymmetry of the lognormal.
The curvilinear evolution of strokes is described with a circular arc shape, which results in
φi(t) = θi + θi
[ 1 + erf ( ln(t− t0i)− µi
σi √ 2
)] , (2)
where θi is the central angle of the circular arc that defines the shape of the ith stroke.
The planar evolution of a trajectory is defined by a sequence of virtual targets {vi}i=mi=1 , where a trajectory with m virtual targets will be characterised by m− 1 circular arc strokes. A ΣΛ trajectory, parameterised by the virtual target positions, is given by
ξ(t) = v1 + ∫ t 0 dτΛi(τ) m−1∑ i=1 Φi(τ) (vi+1 − vi), (3)
with Φi(t) = [ h(θi)cosφi(t) −h(θi)sinφi(t) h(θi)sinφi(t) −h(θi)cosφi(t) ] , and h(θi) = { 2θi 2sinθi if |sinθi| > 0,
1 otherwise, (4)
which scales the extent of the stroke based on the ratio between the perimeter and the chord length of the circular arc.
Intermediate parameterisation. In order to facilitate the precise specification of timing and profile shape of each stroke, we recur to an intermediate parametrisation that takes advantage of a few known properties of the lognormal (Djioua & Plamondon, 2008b) in order to define each stroke with (i) a time offset ∆ti with respect to the previous stroke, (ii) a stroke duration Ti and (iii) a shape parameter αi, which defines the skewedness of the lognormal. The corresponding ΣΛ parameters {t0i, µi, σi} can be then computed with:
σi = ln(1 + αi), (5)
µi = −ln ( −e
−3σi − e3σi Ti
) , (6)
and t0i = t1i − eµ−3σ t1i = t1(i−1) + ∆ti t1(0) = 0, (7)
where t1i is the onset time of the lognormal stroke profile. As α approaches 0, the shape of the lognormal converges to a Gaussian, with mean t1 + eµ−σ 2
(the mode of the lognormal) and standard deviation d6 .
B RECONSTRUCTING ΣΛ PARAMETERS FROM AN ONLINE DATASET
The ΣΛ parameter reconstruction method operates on a input contour uniformly sampled at a fixed distance which is defined depending on the extent of the input, where we denote the kth sampled point along the input with p[k]. The input contour is then segmented in correspondence with perceptually salient key points, which correspond with loci of curvature extrema modulated by neighbouring contour segments (Brault & Plamondon, 1993; Berio & Leymarie, 2015). The proposed approach shares strong similarities with previous work done for (i) compressing online handwriting data with a circular-arc based segmentation (Li et al., 1998) and (ii) for generating synthetic data for handwriting recognisers (Varga et al., 2005). The parameter reconstruction algorithm can be summarised with the following steps:
• Find m key-points in the input contour. • Fit a circular arc to each contour segment defined between two consecutive key-points
(defining individual strokes), and obtain an estimate of each curvature parameter θi.
• For each stroke compute the corresponding ∆ti parameter by analysing the curvature signal in the region of the corresponding key-point.
• Define an initial sequence of virtual targets with m positions corresponding with each input key-point.
• Repeat the following until convergence or until a maximum number of iterations is reached Berio & Leymarie (2015):
– Integrate the ΣΛ trajectory with the current parameter estimate. – Identify m key-points in the generated trajectory. – Move the virtual target positions to minimise the distance between the key-points of
the generated trajectory and the key-points on the input contour.
The details for each step are highlighted in the following paragraphs.
Estimating input key-points. Finding significant curvature extrema (which can be counted as convex and concave features for a closed/solid shape) is an active area of research, as relying on discrete curvature measurements remains challenging. We currently rely on a method described by Feldman & Singh (2005), and supported experimentally by De Winter & Wagemans (2008): first we measure the turning angle at each position of the input p[k] and then compute a smooth version of the signal by convolving it with a Hanning window. We assume that the turning angles have
been generated by a random process with a Von Mises distribution with mean at 0 degrees, which corresponds with giving maximum probability to a straight line. We then measure the surprisal (i.e. the negative logarithm of the probability) for each sample as defined by Feldman & Singh (2005), which normalised to the [0, 1] range simplifies to:
1− cos(θ[k]), (8)
where θ[k] is the (smoothed) turning angle. The first and last sample indices of the surprisal signal together with its local maxima results in m key-point indices {ẑi}. The corresponding key-points along the input contour are then given by {p [ẑi]}.
Estimating stroke curvature parameters. For each section of the input contour defined between two consecutive key-points, we estimate the corresponding stroke curvature parameter θi by first computing a least square fit of a circle to the contour section. We then compute the internal angle of the arc supported between the two key-points, which is equal 2θi, i.e. two times the corresponding curvature parameter θi.
Estimating stroke time-overlap parameters. This step is based on the observation that a smaller values of ∆t0i, i.e. a greater time overlap between strokes, result in smoother trajectories. On the contrary, a sufficiently large value of ∆t0i will result in a sharp corner in proximity of the corresponding virtual target. We exploit this notion, and compute an estimate of the ∆t0i parameters by examining the sharpness of the input contour in the region of each key-point.
To do so we examine the previously computed turning angle surprisal signal, in which we can observe that sharp corners in the contour correspond with sharper peaks, while smoother corners correspond with smooth peaks with a larger spread. By treating the surprisal signal as a probability density function, we can then use statistical methods to measure the shape of each peak with a mixture of parametric distributions, and examine the shape of each mixture component in order to get an estimate of the corresponding sharpness along the input contour. To do so we employ a variant of Expectation Maximisation (EM) (Dempster et al., 1977) in which we treat the distance along the contour as a random variable weighted by the corresponding signal amplitude normalised to the [0, 1] range. Once the EM algorithm has converged, we treat each mixture component as a radial basis function (RBF) centred at the corresponding mean, and use linear regression as in Radial Basis Function Networks (Stulp & Sigaud, 2015) to fit the mixture parameters to the original signal (Calinon, 2016). Finally we generate an estimate of sharpness λi (bounded in the [0, 1] range) for each key point using as a logarithmic function of the mixture parameters and weights. The corresponding ∆t0i parameters are then given by
∆ti = ∆tmin + (∆tmax −∆tmin)λi , (9)
where ∆tmin and ∆tmax are user specified parameters that determine the range of the ∆t0i estimates.
Note that we currently utilise an empirically defined function for this task. But in future steps, we intend to learn the mapping between sharpness and mixture component parameters from synthetically samples generated with the ΣΛ model (for which ∆t0i, and consequently λi, are known).
Iteratively estimating virtual target positions. The loci along the input contour corresponding with the estimated key-points provide an initial estimate for a sequence of virtual targets, where each virtual target position is given by vi = p[ẑi]. Due to the trajectory-smoothing effect produced by the time overlaps, the initial estimate will result in a generated trajectory that is likely to have a reduced scale with respect to the input we wish to reconstruct (Varga et al., 2005). In order to produce a more accurate reconstruction, we use an iterative method that shifts each virtual target towards a position that will minimise the error between the generated trajectory and the reconstructed input. To do so, we compute an estimate of m output key-points {ξ (zi)} in the generated trajectory, where z2, ..., zm are the time occurrences at which the influence of one stroke exceeds the previous. These will correspond with salient points along the trajectory (extrema of curvature) and can be easily computed by finding the time occurrence at which two consecutive lognormals intersect. Similarly to the input key-point case, ξ(z1) and ξ(zm) respectively denote the first and last points of the generated trajectory. We then iteratively adjust the virtual target positions in order to move each generated
key-point ξ(zi) towards the corresponding input key-point p[ẑi] with:
vi ← vi + p [ẑi]− ξ (zi) , (10)
The iteration continues until the Mean Square Error (MSE) of the distances between every pair p [ẑi] and ξ(zi) is less than an experimentally set threshold or until a maximum number of iterations is reached (Fig. 16). This method usually converges to a good reconstruction of the input within few iterations (usually < 5). Interestingly, even though the dynamic information of the input is discarded, the reconstructed velocity profile is often similar to the original (in number of peaks and shape), which can be explained by the extensively studied relationships between geometry and dynamics of movement trajectories (Viviani & Terzuolo, 1982; Lacquaniti et al., 1983; Viviani & Schneider, 1991; Flash & Handzel, 2007).
C RMDN MODEL DETAILS
In order to increase the expressive generative capabilities of our networks, we train them to model parametric probability distributions. Specifically, we use Recurrent Mixture Density Networks that output the parameters of a bivariate Gaussian Mixture Model.
C.1 BIVARIATE RECURRENT MIXTURE DENSITY NETWORK
If a target variable zt can be expressed as a bivariate GMM, then for K Gaussians we can use a network architecture with output dimensions of 6K. This output vector would then consist of (µ̂t ∈ IR2K , σ̂t ∈ IR2K , ρ̂t ∈ IRK , π̂t ∈ IRK), which we use to calculate the parameters of the
GMM via (Graves, 2013)
µkt = µ̂ k t : means for k’th Gaussian, µ k t ∈ IR 2
σkt = exp(σ̂ k t ) : standard deviations for k’th Gaussian, σ k t ∈ IR 2
ρkt = tanh(ρ̂ k t ) : correlations for k’th Gaussian, ρ k t ∈ (−1, 1)
πkt = softmax(π̂ k t ) : mixture weight for k’th Gaussian , K∑ k πkt = 1
(11)
We can then formulate the probability distribution function Pt at timestep t as
Pt = K∑ k πktN(zt | µkt ,σkt , ρkt ), where (12)
N (x | µ,σ, ρ) = 1 2πσ1σ2 √ 1− ρ2 exp
[ − Z
2(1− ρ2)
] , and (13)
Z = (x1 − µ1)2
σ21 +
(x2 − µ2)2 σ22 − 2ρ(x1 − µ2)(x2 − µ2) σ1σ2 (14)
C.2 TRAINING OBJECTIVE
If we let θ denote the parameters of a network, and given a training set S of input-target pairs (x ∈X, ŷ ∈ Ŷ ), our training objective is to find the set of parameters θML which has the maximum likelihood (ML). This is the θ that maximises the probability of training set S and is formulated as (Graves, 2008)
θML = arg max θ
Pr(S | θ) (15)
= arg max θ S∏ (x,ŷ) Pr(ŷ | x,θ). (16)
Since the logarithm is a monotonic function, a common method for maximizing this likelihood is minimizing its negative logarithm, also known as the Negative Log Likelihood (NLL), Hamiltonian or surprisal (Lin & Tegmark, 2016). We can then define our cost function J as
J = − ln S∏
(x,ŷ)
Pr(ŷ | x,θ) (17)
= − S∑
(x,ŷ)
ln Pr(ŷ | x,θ). (18)
For a bivariate RMDN, the objective function can be formulated by substituting eqn. (12) in place of Pr(ŷ | x,θ) in eqn. (18).
C.3 V2V MODEL
Input At each timestep i, the input to the V2V model is xi ∈ IR3, where the first two elements are given by ∆vi (the relative position displacement for the i’th stroke, i.e. between the i’th virtual target and the next), and the last element is ui ∈ {0, 1} (the pen-up state during the same stroke). Given input xi and its current internal state (ci,hi), the network learns to predict xi+1, by learning the parameters for the Probability Density Function (PDF) : Pr(xi+1 |xi, ci,hi). With a slight abuse of notation, this can be expressed more intuitively as Pr(xi+1 | xi,xi−1, ...,xi−n) where n is the maximum sequence length.
Output We express the predicted probability of ∆vi as a bivariate GMM as described in Section C.1, and ui as a Bernoulli distribution. Thus for K Gaussians the network has output dimensions of (6K + 1) which, in addition to eqn. (11), contains êi which we use to calculate the pen state probability via (Graves, 2013)
ei = 1
1 + exp(êi) , ei ∈ (0, 1) (19)
Architecture We use Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) networks with input, output and forget gates (Gers et al., 2000), and we use Dropout regularization as described by Pham et al. (2014). We employ both a grid search and a random search (Bergstra & Bengio, 2012) on various hyperparameters in the ranges: sequence length {64, 128}, number of hidden recurrent layers {1, 2, 3}, dimensions per hidden layer {64, 128, 256, 400, 512, 900, 1024}, number of Gaussians {5, 10, 20}, dropout keep probability {50%, 70%, 80%, 90%, 95%} and peepholes {with, without}.
For comparison we also tried a deterministic architecture whereby instead of outputing a probability distribution, the network outputs a direct prediction for xi+1. As expected, the network was unable to learn this function, and all sequence of virtual targets synthesized with this method simply travel in a repeating zig-zag line.
Training We use a form of Truncated Backpropagation Through Time (BPTT) (Sutskever, 2013) whereby we segment long sequences into overlapping segments of maximum length n. In this case long-term dependencies greater than length n are lost, however with enough overlap the network can effectively learn a sliding window of length n timesteps. We shuffle our training data and reset the internal state after each sequence. We empirically found an overlap factor of 50% to perform well, though further studies are needed to confirm the sensitivity of this figure.
We use dynamic unrolling of the RNN, whereby the number of timesteps to unroll to is not set at compile time, in the architecture of the network, but unrolled dynamically while training, allowing variable length sequences. We also experimented with repeating sequences which were shorter than the maximum sequence length n, to complete them to length n. We found that for our case they produced desirable results, with some side-effects which we discuss in later sections.
We split our dataset into training: 70%, validation: 20% and test:10% and use the Adam optimizer (Kingma & Ba, 2014) with the recommended hyperparameters. To prevent exploding gradients we clip gradients by their global L2 norm as described in (Pascanu et al., 2013). We tried thresholds of both 5 and 10, and found 5 to provide more stability.
We formulate the loss function J to minimise the Negative Log Likelihood as described in Section C.2 using the probability density functions described in eqn. (12) and eqn. (19).
C.4 V2D MODEL
Input The input to this network at each timestep i is identical to that of the V2V-model, xi ∈ IR3, where the first two elements are ∆vi (normalised relative position displacement for the i’th stroke), and ui ∈ {0, 1} (the pen state during the same stroke). Given input xi and its current internal state (ci,hi), the network learns to predict the dynamic parameters (∆t0i, θi) for the current stroke i, by learning the parameters for Pr(∆t0i, θi | xi, ci,hi). Again with an abuse of notation, this can be expressed more intuitively as Pr(∆t0i, θi | xi,xi−1, ...,xi−n) where n is the maximum sequence length.
Output We express the predicted probability of the dynamic parameters (∆t0i, θi) as a bivariate GMM as described in Section C.1.
Architecture We explored very similar architecture and hyperparamereters as the V2V-model, but found that we achieved much better results with a shorter maximum sequence length. We trained a number of models with a variety of sequence lengths {3, ..., 8, 13, 16, 21, 32}.
Training We use the same procedure for training as the V2V-model.
C.5 A2D MODEL
Input The input to this network xi ∈ IR5 at each timestep i is slightly different to the V2V and V2D models. Similar to the V2V and V2D models, the first two elements are ∆vi (normalised relative position displacement for the i’th stroke), and the third element is ui ∈ {0, 1} (the pen state during the same stroke). However in this case the final two elements are the dynamic parameters for the previous stroke (∆t0i−1, θi−1), normalized to zero mean and unit standard deviation.
Given input xi and its current internal state (ci,hi), the network learns to predict the dynamic parameters (∆t0i, θi) for the current stroke i, by learning the parameters for Pr(∆t0i, θi |xi, ci,hi). Again with an abuse of notation, this can be expressed more intuitively as Pr(∆t0i, θi | xi,xi−1, ...,xi−n) where n is the maximum sequence length.
Output The output of this network is identical to that of the V2D model.
Architecture We explored very similar architecture and hyperparamereters as the V2D model.
Training We use the same procedure for training as the V2V-model.
C.6 MODEL SELECTION
We evaluated and batch rendered the outputs of many different architectures and models at different training epochs, and settled on models which were amongst those with the lowest validation error, but also produced visibily more desirable results. Once we picked the models, the results displayed are not cherry picked.
The preprocessed IAM dataset contains 12087 samples (8460 in the training set) with maximum sequence length 305, minimum 6, median 103 and mean 103.9. For the V2V/V2D/A2V models trained on the IAM database we settle on an architecture of 3 recurrent layers, each with size 512, a maximum sequence length of 128, 20 Gaussians, dropout keep probability of 80% and no peepholes.
For the augmented one-shot learning models we used similar architectures, but found that 2 recurrent layers each with size 256 was able to generalise better and produce more interesting results that both captured the prime inputs without overfitting.
For V2V we used L2 normalisation on ∆vi input, and for A2D/V2D we used
We also tried a number of different methods for normalising and representing ∆vi on the input to the models. We first tried normalising the components individually to have zero mean and unit standard deviation. We also tried normalising uniformly on L2 norm again to have zero mean and unit standard deviation. Finally, we tried normalised polar coordinates, both absolute and relative.

The paper presents a method for sequence generation with a known method applied to feature extracted from another existing method. The paper is heavily oriented towards to chosen technologies and lacks in literature on sequence generation. In principle, rich literature on motion prediction for various applications could be relevant here. Recent models exist for sequence prediction (from primed inputs) for various applications, e.g. for skeleton data. These models learn complex motion w/o any pre-processing.
Evaluation is a big concern. There is no quantitative evaluation. There is no comparision with other methods.
I still wonder whether the intermediate representation (developed by Plamondon et al.) is useful in this context of a fully trained sequence generation model and whether the model could pick up the necessary transformations itself. This should be evaluated.
Details:
There are several typos and word omissions, which can be found by carefully rereading the paper.
At the beginning of section 3, it is still unclear what the application is. Prediction of dynamic parameters? What for? Section 3 should give a better motivation of the work.
Concerning the following paragraph
"While such methods are superior for handwriting analysis and biometric purposes, we opt for a less precise method (Berio & Leymarie, 2015) that is less sensitive to sampling quality and is aimed at generating virtual target sequences that remain perceptually similar to the original trace.
"
This method has not been explained. A paper should be self-contained.
The authors mentioned that the "V2V-model is conditioned on (...)"; but not enough details are given.
Generally speaking, more efforts could be made to make the paper more self-contained.