## Neural Architecture based on Fuzzy Perceptual Representation For Online Multilingual Handwriting Recognition

Hanen Akouaydi, Student Member, IEEE, Sourour Njah, Member, IEEE, Wael Ouarda, Member, IEEE, Anis Samet, Professional tutor, Thameur Dhieb, Member, IEEE, Mourad Zaied,Senior Member, IEEE, and Adel M. Alimi, Senior Member, IEEE

Abstract -Due to the omnipresence of mobile devices, online handwritten scripts have become the most important feeding input to smartphones and tablet devices. To increase online handwriting recognition performance, deeper neural networks have extensively been used. In this context, our paper handles the problem of online handwritten script recognition based on extraction features system and deep approach system for sequences classification. Many solutions have appeared in order to facilitate the recognition of handwriting. Accordingly, we used an existent method and combined with new classifiers in order to get a flexible system. Good results are achieved compared to online characters and words recognition system on Latin and Arabic scripts. The performance of our two proposed systems is assessed by using five databases. Indeed, the recognition rate exceeds 98 % .

Index Terms -Online handwriting recognition, Fuzzy Perceptual Code, Beta-elliptic model, Fuzzy logic, Deep Fuzzy Neural Network, Convolutional LSTM, Fuzzy Ground Truth Training, CTC, Noise.

F

## 1 INTRODUCTION

H ANDWRITING recognition system is the tool whereby a computer system can recognize characters and other symbols written by hand. It recognizes individual characters and an entire word, just as the human eye and mind would. Humans can understand any type of writing in different constraints. Thus, many researches are conducted to analyze handwriting and recognize it automatically. In this context, we develop, in our work, two systems for online handwritten script. In fact, the first system uses the perceptual theory, inspired from the human perceptual system, transforming language from graphical marks into symbolic representation. It proceeds by the detection of common properties, and then it gathers them according to some perceptual laws proximity, similarity...,etc. Finally, it attempts to build a representation of the handwritten form based on the assumption that the perception of form is the identification of basic features that are arranged till we identify an object. Therefore, the representation of handwriting is a combination of primitive strokes. Handwriting is a sequence of basic codes which are grouped together to define a character or a shape. This perceptual theory segments an handwritten script into elliptic strokes by Betaelliptic model of handwriting generation. These strokes will be classified into primitives codes then we apply LSTM classifier in order to recognize online handwritten script. We choose multilingual scripts : Latin and Arabic.

In the second system, we apply convolutional LSTM on (x, y,

- · H.Akouaydi, S.Njah, T.Dhieb, M.Zaied and A.M.Alimi, REGIM-Lab.: REsearch Groups in Intelligent Machines, University of Sfax, National Engineering School of Sfax (ENIS), BP 1173, Sfax, 3038, Tunisia.

· Anis Samet, Professional tutor, Sifast. E-mail: anis.samet@sifast.com Manuscript received August 01, 2019;

z) collected from Arabic/Latin datasets and also from data acquired from Android application.

The structure of this paper is as follows: Section 2 presents an overview of systems for handwriting recognition in literature based on extraction features and deep learning approaches. Section 3 illustrates our two systems for online handwritten script recognition. Section 4 reports the rate recognition performed on online handwriting and section 5 draws concluding remarks of future works.

## 2 RELATED WORK

Handwriting recognition is transforming graphical marks into symbolic representation. Although humans are similar and have a common education, they produce significantly different handwritten styles. The factors of the variability are related to the writer, emotional factors, language of writing, the writing with different size, styles and speed...etc. We found two main approaches for Handwriting Recognition : The first approach is based on segmentation and classification (or segment-and-decode) and the second is related to time-sequence interpretation.

## 2.1 Handcrafted Approaches

Those approaches segment then decode the script in order to identify it. Traditional classification are based on extraction features that are engineered and extracted from kinetic signals.

Google online handwriting system is one of the most powerful systems that presently supports 22 scripts and 97 languages. The same architecture of its system is used for mobile devices with more limited computational power by means of changing some of the settings of the system [3]. We

TABLE 1 Summary of systems for handwriting recognition.

| Approaches               | Authors             | Database                                                                                                                      | Method                                                              | Accuracy                                |
|--------------------------|---------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|-----------------------------------------|
| Handcrafted Approaches   | [1] [2] [3] [4] [5] | LMCA + manually Segmented characters form ADAB LMCA + manually Segmented characters form ADAB UNIPEN+ Google cloud Farsi ADAB | DNN MLP+HMM system Language Model NN+GA Grapheme segmentation + MLP | 82.12 % 96.45 % 99.02 % 94.20 % 93.50 % |
| Deep Learning Approaches | [6] [7] [8] [9]     | Chinese Mathematical Expressions Chinese Drawing & Recognition Chinese characters                                             | CNN GRU CNN+Drop Weight RNN                                         | 97.20 % 52.43 % 96.88 % 93.98 %         |

also find another approach for online Arabic handwriting recognition, based on neural networks approach. This work suggests a method based on TDNN classifier and multilayer perceptron,it achieved 98.50% for characters, 96.90% for words [4]. There is another work which recognizes mathematical expression by using HMM and the system was conducted by covering two things: feature modification experiment and code words number experiments. The best result is gained with respect to four combination features [10]. We also notice a system for online Farsi characters recognition. The proposed system is tested on TMU database and 94.2% obtained as recognition rate for the recognition of the group and the character, respectively [4]. Hamdi et al. [5] suggest a hybridization of neural networks and genetic algorithm for online Arabic handwriting recognition. In fact, based on Beta-elliptic model and baseline detection. The proposed system reached 93.5% as average of recognition. Another work deals with Online Arabic Handwritten Character Recognition based on a Deep Neural Networks DNN in which we attempt to optimize the training process by a smooth construct of the deep architecture. This system uses LMCA database for training and testing data. Tagougui et al. gained 96.16% as testing rate [1]. There is a new work which proposes a model based on Bzier curve interpolation and recurrent neural networks for online handwriting recognition and that able to support 102 languages [11]. Besides, another work describes a hybrid model based on bi-directional LSTM and Connectionist Temporal Classifier (CTC) for handwriting recognition on mobile devices. [12]

The success of deep neural networks approaches has been achieved based on handwriting recognition. However, recognition is not only a way to understand a language, but also a challenging task to teach machine how to write automatically. Most research works did not propose universal methods to recognize special languages as Farsi, Chinese or Mathematical Expressions [13]. The major motivation of this paper consists in dealing with online script by using some novel techniques in training deep recurrent neural networks. In fact,our proposed models for handwriting recognition applied various lexicons by using perceptual codes, LSTM and convolutional LSTM.

## 2.2 Deep Learning Approaches

Generative approaches based on Convolutional Neural Networks (CNN) and LSTM to recognize handwritten script. The extraction features process requires a deep knowledge of application domain, or human experience or hardware memory to store script in order to identify script from user. Accordingly, automatic and deep methods are attracting in the field of handwriting Recognition. By adoption data-driven approaches for signal (x, y, z) script classification, the process of selecting meaningful features from the data is deferred to the learning model. CNNs are capable of detecting both spatial and temporal dependencies among signals, and can effectively model scale invariant features. It is noteworthy that two main advantages can be considered, when applying deep neural networks to handwriting recognition which are as follows:

- · Local Dependency:New Deep Neural Networks capture local dependencies of handwriting signals. In image recognition tasks, the nearby pixels typically have strong relationship with each other. Similarly, in handwritten script(online (x, y, z)) given an script (signal) the nearby acceleration readings are likely to be correlated.
- · Scale Invariance: Deep neural Networks preserve feature scale invariant. In image recognition, the training images could probably different scales. In handwritten script , a person may write with different styles (e.g., with different motion intensity).

Those approaches used are: hidden Markov models (HMMs) [14], time-delay neural networks (TDNN) [15], and recurrent neural networks (RNN) [16], of which longshort-term-memory networks (LSTM) [17] are a specific case, presently receiving significant attention with regards to machine learning.

## 3 PROPOSED SYSTEM

Online handwritten scripts are always stored as form of sequence points. Divers features can be extracted from them such as stroke coordinates, times stamp, writing angle and curvature, etc. The aim of the turn-out of such system is to recognize the sequence points. In this paper, two models are suggested, both of them are based on deep neural networks first corresponds to Online Handwriting Segmentation &amp; Recognition using LSTM, OnHSR-LSTM , which requires a step of segmentation, and the second, consists in Online Handwriting Recognition using convLSTM, OnHRconvLSTM . The two models will be described below:

Fig. 1. Architecture OnHS-LSTM.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-66763b73-e7a8-4666-b9bd-f9738180cd2c.jpg)

TABLE 2 Basic Perceptual Codes

|   Number | Basic Perceptual Codes   | Abbreviation   | Shape   |
|----------|--------------------------|----------------|---------|
|        1 | Valley                   | V              |         |
|        2 | Left-oblique-shaft       | L-O-S          |         |
|        3 | Shaft                    | S              |         |
|        4 | Right-oblique-shaft      | R-O-S          |         |
|        5 | Right-Half-Occlusion     | R-H-O          |         |
|        6 | Left-Half-Occlusion      | L-H-O          |         |
|        7 | Up-Half-Occlusion        | U-H-O          |         |
|        8 | Down-Half-Occlusion      | D-H-O          |         |
|        9 | Occlusion                | Occ            |         |

## 3.1 Online Handwriting Segmentation &amp; Recognition using LSTM, OnHSR-LSTM

Our first model OnHSR-LSTM, is based on the fact that handwriting is a form created by hand movement and all we perceive is only a form [18], [19], [20] It proposes that handwriting is a concatenation of perceptual codes grouped together, so as to form a character, a digit or a word. Each online script can be composed of elementary basic shapes. With (x, y, z) coordinates of online trace and the beta-elliptic model we detect basic shapes forming the script. Then, we classify them relying on LSTM in order to recognize the character or the written digit. Figure2 represents the architecture of LSTM-Character-recognizer, which recognizes Online Handwritten letters. Our first model is endowed with crucial phases:the phase of segmentation and the phase of classification and recognition.

## 3.1.1 Segmentation Step

The segmentation step is concerned with the regeneration of the online handwritten script by means of strokes or by a set of control points that describe the handwritten trace.

3.1.1.1 Preprocessing:

Preprocessing is a key step with respect to recognition in

Fig. 2. Illustrated example of composed forms.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-8e3e7725-bdb4-41e2-befe-ad126a90d4ef.jpg)

order to obtain better recognition rate. The principles of preprocessing steps are used to eliminate trembles in writing , to reduce noise and to remove the hardware imperfections. The preprocessing operations used in our system are:

- · Interpolation : adding missing points caused by the variation of writing velocity.
- · De-hooking : eliminating trembles in writing due to inaccuracies in rapid pen-down/up detection and trembles in writing;
- · Removing noise : we apply a Chebyshev low pass filer to eliminate the noise generated by spatial and temporal sampling;

## 3.1.1.2 Beta-elliptic Features:

Handwriting is a message which can be decoded into basic shapes. The main assumption of our proposed model consists in the fact that handwriting is a concatenation of visual codes grouped together so as to get a shape, a character or a digit. The principal basic shapes to write are: -, / , | \ , . [21], [22]. The basic shape which can compose any script. Indeed, the basic shapes are illustrated in Table 2. The analysis of any language has demonstrated that a script is an arrangement of vertical and diagonal lines as shown Figure 1, which demonstrates that a letter is a combination of two or more basic shapes. In fact we have 9 basic perceptual shapes that can be simple or complex. These basic shapes have been identified as Global Perceptual Codes with GPC 'Ain' which only characterize the Arabic script. [18], [19], [23], [24], [25] Thus, in this paper we use only 9 basic perceptual shapes which can form any script. In Figure 1, we notice that handwriting is an arrangement of perceptual codes in a special order, and we can from different scripts with the same group of basic shape such as the two Arabic letter GLYPH&lt;5&gt;« hamza Z . In this step, we extract the basic shape that gather together to identify a script. Hence, we use the beta-elliptic model as a model of segmentation. Basic shape are not directly detectable. Therefore, we use in the step of segmentation the beta-elliptic model.

Beta-elliptic model propose that handwriting can be reconstructed by the sum of impulse signals by the curvilinear velocity and can be also approximated by a sequence of elliptic shapes. [25], [26], [27] So, handwriting can be segmented into simple movements called strokes, which are the result of superposition of time overlapped velocity profiles. Consequently, each stroke with its curvilinear velocity obeys the Beta-elliptic approach and online script can be segmented into different ones by tracting Beta-elliptic parameters. [19],

## [20], [24], [28]

The Beta function is presented in the equation(1)

$$\beta ( t, p, q, t _ { 0 }, t _ { 1 } ) & = ( \stackrel { \mu } { \frac { t - t _ { 0 } } { t _ { c } - t _ { 1 } } } ) ^ { p } ( \stackrel { t _ { 1 } - t } { \frac { t } { t _ { 1 } - t _ { c } } } ) ^ { q }, I F t \in ] t _ { 0 }, t _ { 1 } [ \quad ( 1 ) \\ & = 0, I F n o t$$

where t 0 : starting time of Beta function.

t c : is the instant when the curvilinear velocity reaches the amplitude of the inflexion point.

t 1 : the ending time of Beta function.

With p, q, t ( 0 &lt; t 1 ) ∈ R,and

$$t _ { c } = \frac { p * t _ { 1 } + q * t _ { 0 } } { p + q }, p = q * \frac { t - t _ { 0 } } { t _ { c } - t _ { 1 } }$$

p , q intermediate parameters which have an influence on the symmetry and the width of Beta shape.

As a result, a stroke is characterized by 10 parameters.The following beta-elliptic parameters a, b, x 0 , y 0 and θ reflect the geometric properties of the set of muscles and joints used in a particular handwriting movement and describe the static aspect of handwriting. These parameters are:

- · x 0 , x 1 :the coordinates of the ellipse center,
- · a: big axe of the ellipse
- · b: small axe of the ellipse
- · θ : the angle of the deviation of the elliptic arc and the horizontal which is obtained by the following equation 2:

$$\theta = \arctan ( \frac { Y _ { 1 } - Y _ { 0 } } { X _ { 1 } - X _ { 0 } } )$$

The Beta elliptic was performed in systems for writer identification and verification. [29], [30], [31] An online handwriting script can be segmented into segments and each segment is a group of strokes. In fact, the number of strokes is identified automatically from the curvilinear velocity representation. In the trajectory domain, each stroke is limited by both M1 and M3 that correspond to the maximum curvature and minimum curvilinear velocity. M2 is defined by the maximum tangential velocity.

3.1.1.3 Elementary Perceptual Codes (EPC) Detection with Beta-points:

The main idea of this part is to associate an EPC (Elementary Perceptual Code) to each stroke. It means that we classify strokes of the handwritten script generated by the betaelliptic model into EPCs.This classification is performed by using the parameter, corresponding to the deviation angle of each ellipse ' θ ', by the horizontal axis. An EPC is limited by beta control points M1, M3 and goes through H which the orthogonal projection of M2. The number of strokes is identified automatically by beta-elliptic model. This number is equal to the number of EPCs. The EPCs are similar to Freeman's code, which are frequently used for online handwriting recognition, being able to express stroke directions. Based on this technique, features are extracted from each online trajectory segment. By means of analogy with Freemans codes, an EPC is a whole region incorporating a group of directions as depicted in Figure 2.(a) showing the 8 directions of Freeman's codes and 2.(b) illustrating all EPCs in their corresponding parts. [19]

Different factors(style, speed, position, size, orientation )

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-25f6e136-6888-487c-9e43-f8726416fecc.jpg)

Fig. 3. Segmentation Step of Arabic Letter 'yaa'.

Fig. 4. Segmentation and Detection of Beta-points of Arabic word 'Kalaa' .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-23197bdf-afbe-4319-9a10-a06480639552.jpg)

Fig. 5. Directions: (a): Freeman ′ s chain code, (b): the proposed regions, (c):Deviation angles regions and EPCs on the trigonometric circle,(d):Elementary Perceptual Codes

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-8156708b-1a4e-4b35-99eb-00f54470a37b.jpg)

## Membership degre

Fig. 6. Example of perceptual problem decision using fuzzy logic.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-ffdd776f-ab25-43de-817a-086300d10c66.jpg)

can affect the handwritten script, thus making its treatment and analysis difficult. Figure 6 presents some regions of trigonometric circle, we consider an overlapped interval value noted cst equal to /16 where there is a problem of indecision pertaining to the corresponding EPC. This indicates an example of the perceptual problem decision related to the belongings of elliptic stroke. To prevent some perception problem, we associate an EPC to each stroke with a certain membership degree. Thus, we opt for employing fuzzy logic representation to resolve the problems of fuzzy perception and the uncertainty of the assessed EPC [18], [19], [23]. Correspondingly, for each stroke a certain belongingness degree will be allowed.

## 3.1.2 LSTM FOR CLASSIFICATION

After extracting the Basic Perceptual Codes (BPC) of online script, a classifier is required to identify it. We opt to use LSTM, Long Short-Term Memory as classifier. LSTM is specially constructed RNNs, Reccurent Neural Networks, node to preserve long lasting dependencies by avoiding the vanishing gradient problem. It is explicitly conceived to prevent the long-term dependency problem, a tool of sequence classification in order to identify time-sequences. In fact, their default behavior of LSTM cells lies mainly in remembering information forlong periods of time. It provides a memory of the previous network internal state. Long Short-Term memory (LSTM) layer: The LSTM network nodes posses a specific architecture, referred to as memory block. Each memory block incorporates a memory cell, and its interaction with the rest of the network is controlled by three gates, i.e: an input gate, an output gate and a forget gate. This permits the memory cell to preserve its state for a long period. The following illustrates the architecture of LSTM To identify the online script or to identify basic perceptual codes, we use LSTM. The handwritten script is segmented into strokes limited by beta-points. We apply fuzzy logic in order to obtain the membership of one stroke

Fig. 7. Architecture of LSTM cells.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-27b50352-fbc2-4d54-993d-b16bf61c166b.jpg)

to the 4 EPCs. A segment is a group of strokes, and stroke is composed of 4 EPCs with degree of membership. Each stroke is classified into 4 EPCs, and a segment is a group of n strokes. A BPC is a group of n segments or a script(character, digit ,word) is a group of n EPCs. Handwriting is a group of perceptual codes which are the bases pattern to read, write, identify, segment and recognize a script. In this section, we detail how from these perceptual codes, we can form a character, a shape or a digit. Handwriting has an equation as our perceptual theory which is:

$$\underset { \text{ons of} } { \text{s.s.s.s.h} } \quad h a n d w r i t i n g = \left \{ S e g m e n t _ { 1 } i, S e g m e n t _ { 2 } i, S e g m e n t _ { n } i \right \} \ ( 3 )$$

n:the total number of segment s composing handwriting.

$$\ h a n d w r i t i n g = \{ B P C _ { 1 } i, B P C _ { 2 } i, \dots, B C P _ { n } i \} \quad ( 4 )$$

i:1,9 correspond to BPC s. (see Table II)

$$B P C = \{ E P C _ { 1 } j, E P C _ { 2 } j, \dots, E P C _ { p } j \} \quad \ \ ( 5 )$$

$$\ B P C = \{ S t r o k e _ { 1 } j, S t r o k e _ { 2 } j, \dots, S t r o k e _ { p } j \} \quad ( 6 )$$

p:Number of stroke s identified in the trace.

j:1,2,3,4 correspond to EPC s. (see Table I) With an adequate arrangement of elementary perceptual codes (EPCs),

$$\begin{smallmatrix} & S t r o k e / E P C = b e t a { - p o i n t 1 } { + b e t a { - p o i n t 2 } { + b e t a { - p o i n t 3 } } } \\ \text{if online} & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & && & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &  & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &
 & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & \\ & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & \ & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & _ & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &. & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & - & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & ; & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & } & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & = & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & + & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &, & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &

 & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & b & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & ^ & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & | & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &$$

The beta-points are three ,anchorage perceptual points [18], [19], [25],forming a stroke. The stroke goes from M1 to M2, which are the maximum curvilinear velocity and the minimum curvilinear velocity, respectively. It passes through H which is the orthogonal projection of M2, standing for the maximum tangential velocity or the inflexion point. These Basic perceptual Codes (BPCs)are divided into two classes:

- · A simple class contains: Valley, Left oblique shaft, Shaft, Right oblique shaft.
- · Acomplex class is composed of: Right half occlusion, Left half occlusion, Up half occlusion, Down half occlusion, Occlusion.

As already mentioned, handwriting is a sequence of basic perceptual codes and by the combination of simple ones, we obtain global ones. Numerous possible combination of EPCs can form a BPC. In fact, there is a large number of combinations that forms a BPC. Therefore, a large set of BPCs features is assumed to be available in order to recognize the corresponding BPCs of the initial script. As mentioned above, the Basic Perceptual Codes are a set of

sequence of Elementary perceptual Codes. The same BPC can be obtained by different sets of EPCs. The BPC 'Valley' can be obtained by a minimum set of three successive EPCs, corresponding to Valley and at a maximum set of six Valley EPCs. Those difficulties which are the sequences of EPCs, can vary in length and be comprised of a very large combination of one BPC, and may require a model to learn the long-term context or dependencies between BPCs in the input sequence of initial script. Thus, the problem of the suitable choice of corresponding BPC can be consequently regarded as related to sequence classification. Sequence classification is a predictive modeling problem, there is some sequence of inputs over space or time and the task consist in predicting a category for the sequence. To improve the

Fig. 8. Some samples of BPC's combination.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-3da127cb-25d8-4010-8b4c-0b09600c3b63.jpg)

performance of our network, we use the CTC as output layer after softmax layer in system 1 which pertains to Perceptual Codes-LSTM for Online handwriting recognition. In fact, it is a layer of labelling. The CTC algorithm employs a many-to-one sequence-to-sequence mapping function that converts the sequence of labeling with timing information into the shorter sequence of labels, by means of removing timing and alignment information [32]. Indeed, the main point is to introduce the additional CTC blank label for the sequence that has timing information, then remove the blank labels and merge repeating labels do as to obtain the unique corresponding sequence. Based on our OnHSRLSTM system,the output of the final LSTM layer is passed through a softmax layer that yields the predictable characters for each time step. The output of the LSTM layers at each time step is located into a softmax layer in order to obtain a probability distribution words, digits or characters, as well as a blank label added by CTC.

## 3.2 Convolutional LSTM for Handwriting Recognition

Our second proposed system OnHR-convLSTM, is presented in the Figure 9 with the architecture of convLSTM cell. It is a generative model that has as input the online signal (x, y, z) of the script and predicts characters as well as words from trained model. This generative system excels with sequence learning tasks and can align unsegmented data with their corresponding ground truth. LSTM network models are a type of recurrent neural network capable of learning and remembering over long sequences of input data. In fact, they may be convenient for such problem. The model can support multiple parallel sequences of input data, such as each axis of the (x, y, z) data. The model learns to extract features from sequences of observations and how to map the internal features to various types of activity. The merit of using LSTMs for sequence classification lies in the fact that they can learn from the raw time series data, directly, and in turn, do not necessitate domain expertise to manually engineer input features. The model can learn an internal representation of the time series data, thus ideally achieving comparable performance to models fit on a version of the data-set with engineered features. ConvolutionalLSTM model is a different approach, wherein the image passes through the convolution layers and the result consists in a set flattened to a 1D array with the obtained features. Repeating this process to all images in the time set, give rise to a set of features over time, which is, in fact, the LSTM layer input. The ConvLSTM layer output is a combination of a Convolution and LSTM output. The other ConvLSTM parameters are derived from the Convolution and the LSTM layer. The The Convolutional LSTM architecture includes using Convolutional Neural Network (CNN) layers for feature extraction on input data combined with LSTMs, in order to support sequence prediction. It was developed for visual time series prediction problems and the application of generating textual descriptions from sequences of images (e.g. videos). A Convolutional LSTM can be defined by adding CNN layers on the front end followed by LSTM layers with a Dense layer on the output. This architecture is considered helpful by defining two sub-models: the CNN Model for feature extraction and the LSTM Model for interpreting the features across time steps. Unlike an LSTM that reads the data directly in order to calculate internal state and state transitions, and unlike the CNN LSTM that is interpreting the output from CNN models, the ConvLSTM uses convolutions directly as part of reading input into the LSTM units themselves. Both Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) have shown improvements over Deep Neural Networks (DNNs) across a wide range of speech recognition tasks. CNNs, LSTMs and DNNs are complementary with respect to their modeling capabilities. Indeed, CNNs are efficient at reducing frequency variations, LSTMs are good at temporal modeling, and CNNs are appropriate for mapping features to a more separable space. Therefore, we combine them into one unified architecture.

## 4 REGULARIZATION AND DATA AUGMENTATION

To improve the performance of our system, we used some techniques, as illustrated above.

## 4.1 Dropout

The main goal when using dropout is to regularize the neural network that, we train. The technique involves of dropping neurons randomly with some probability p. Those random modifications of the networks structure are believed to avoid co-adaptation of neurons by making it impossible for two subsequent neurons to rely solely on each other. In fact, dropout is implicitly bagging, at test time, a large number of neural networks that share parameters. Besides, dropout is used it to prevent the over-fitting of our network from and to upgrade the latter's performance, dropout is applied. This technique consists in temporarily removing a

Fig. 9. (a)Architecture of OnHR-convLSTM, (b) convLSTM cell.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-b489fede-5789-4ec6-b694-49084e5ba307.jpg)

unit from the network. The selection of such removed unit is randomly occurring only during the training stage.

## 4.2 Data Augmentation

The use of deformation technology, in the context of deep neural networks, aims chiefly providing shape variation and generating numerous online training data. Deep LSTM extends the data-set by applying affine transformations including scaling, rotations, and translations. Besides, stroke jiggling is also used to generate local distortions to enrich our data with local diversity. Data augmentation is added in order to prevent over-fitting, and to make the learning more uniform. The neural network training is at the risk of overfitting and hurting the classification robustness. One way to deal with this problem is data augmentation where the training set is artificially augmented by adding replicas of the training samples under certain types of transformations

Fig. 10. Data Augmentation of Arabic letter 'haa' h .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-420b3a77-5a6c-4e8f-b94c-b82ea2d064bd.jpg)

that preserve the class labels. Data generated under such label-preserving transformations will ameliorate the prediction invariance and generalization capacity pertaining to the

neural networks. Data augmentation using label-preserving transformations proved to be effective for neural network training in terms of making invariant predictions. The variation in handwriting style represents a potential problem with regards to online character. To resolve such problem, the concept of character distortion is introduced in order to generate numerous training samples. In fact, character distortion is produced via applying an affine transformation and its variants to the training data. The distortion is applied by performing flipping and rotation (changing the angle of orientation). Data augmentation maintained by different scripts emanating from different writers who do not the same style of writing.

Fig. 11. Distortion of digit 7.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-3ffbb8f6-365d-4628-8b7b-c27ab818123f.jpg)

## 4.3 Training Network

To improve the performance of two our proposed systems, various techniques are used that we will illustrate below.

## 4.3.1 Framewise/Crisper Training

We opt this technique of training with two proposed systems OnHRS-LSTM and OnHR-convLSTM which each segmented trace or not segmented trace are belong only to one class(one character,one word). Each data belongs to one

## Algorithm 1 Crisper Training Algorithm

## repeat

for all

do online script data (x, y, z) not segmented or segmented into basic perceptual codes: each data belongs to one class

Use LSTM for training

until data=0

Use the trained network to predict the labels

label. It means that Framewise method computes the forced alignments of the input sequence with target output of the correct word/char sequence. We train the network to classify each sequence individually, based on cross-entropy cost function. [33] Algorithm 1 describes our method of training, which consists in the fact that one sequence corresponds only to one predicted class.

## 4.3.2 Training with Fuzzy Ground Truth

To perform the network performance of our first proposed model, OnHRS-LSTM, each segmented data belongs to all classes with certain membership. To define the value of membership, we calculate the segmented data which has the longest series of elliptic-strokes { l i } . Then, we calculate the Euclidian distance between the length of class of reference and the actual data segmented { l r } and { l i } .

## Algorithm 2 Training with Fuzzy Ground Truth Algorithm

## repeat

for all

do online script data (x, y, z) segmented into BE- parameters, define { l r }

Euclidian distance between { l r } and { l i }

Use LSTM for training

until data=0

Use the trained network to predict the labels

TABLE 3 Common Characters Confusions in Latin/Arabic Script.

| Latin Script                              | Latin Script   | Latin Script   | Latin Script                   | Latin Script                    | Latin Script     |
|-------------------------------------------|----------------|----------------|--------------------------------|---------------------------------|------------------|
| P                                         | m              | a              | n                              | p b                             | i e l c          |
| p                                         | n              | c              | u                              |                                 |                  |
| Arabic Script                             | Arabic Script  | Arabic Script  | Arabic Script                  | Arabic Script                   | Arabic Script    |
| GLYPH<128> GLYPH<17> GLYPH<128> GLYPH<11> | ¤ GLYPH<9> ¤   | X GLYPH<9> X   | GLYPH<144> GLYPH<9> GLYPH<144> | GLYPH<9> ‹ GLYPH<16> GLYPH<134> | p P h GLYPH<240> |

## 5 SINGLE CHARACTER EXPERIMENTS

The pipeline of our system 1 , OnHSR-LSTM, consists in obtaining the online script, segmenting it into strokes, and classifying them into EPCs. After acquiring EPCs, we apply LSTM in order to obtain BPCs, and then eventually we apply LSTM in order to identify the character or shape or word corresponding of the initial script. The evaluation of the performance pertaining to our two proposed systems, OnHSR-LSTM &amp; OnHR-convLSTM, relies to 5 multilingual databases.

## 5.1 Arabic Database

Factors do not generalize very well to a one , thus making the task of recognition difficult. Although Arabic is ranked the fourth most spoken language in the world after Chinese, English and Spanish, to the best of our knowledge, there is works in the field of Arabic text recognition, which pave the way for exploration of such favorable area of research. Arabic script, written from right to left, is a cursive script. Arabic script, written from right to left is a cursive script in both printed and handwritten versions. Comparing to Latin scripts, it involves of strokes and dots above and under letters, along with, the connectivity of characters. There are 28 letters and the shape of those letters change depending on their position in the word: preceded and/or followed by other letters or isolated. Arabic letters can take four different shapes depending on their position in the word (first, middle, last and isolated) as illustrated in the Figure 1. Some letters can take different forms, for example the letter Ghayen (GLYPH&lt;5&gt;« GLYPH&lt;9&gt; , GLYPH&lt;5&gt;'GLYPH&lt;7&gt; GLYPH&lt;9&gt; , ¤ GLYPH&lt;9&gt; , 'GLYPH&lt;7&gt;) GLYPH&lt;9&gt; . We find that some Arabic letters sets share the same shapes and only a dot makes a difference between them such as or (GLYPH&lt;128&gt; GLYPH&lt;17&gt; , GLYPH&lt;128&gt;) or (h . , p , h) . Arabic

TABLE 4

Comparison of recognition rate of System Perceptual Codes-LSTM with other systems.

| System       | Database                                      | Methods            | RR Framewise   | RR Fuzzy Ground Truth   |
|--------------|-----------------------------------------------|--------------------|----------------|-------------------------|
| Tagougui2014 | LMCA+ manually segmented characters form ADAB | DNN                | 82.12 %        | -                       |
| Tagougui2014 | LMCA+ manually Segmented characters form ADAB | MLP NN /HMM system | 96.45 %        | -                       |
| OnHSR-LSTM   | LMCA+ manually segmented characters form ADAB | BCP+LSTM           | 91.50 %        | 97.50 %                 |
| OnHSR-LSTM   | MAYASTROUN                                    | BCP+LSTM           | 94.50 %        | 98.5 %                  |
| OnHSR-LSTM   | ADAB                                          | BCP+LSTM           | 90.00 %        | 96.00 %                 |

writing recognition, it is a hard task since numerous letters posses different numbers of positions and dots. In most cases, Arabic writing is devoid of vowels. The meaning of the word is frequently determined by the context of the sentence. Thus, all vowels are not considered in our work. However, Latin script, is considered as the easiest script, it serves the largest family of a total of 59 languages [3]. It composes of 26 characters. Our work based on the theory of perception i.e.: all we perceive is a form including handwriting scripts. Handwriting Recognition is feasible, even if their constitutions is missing based on the RBC approach [25], [34]. Handwriting is a visual scene perceived in order to decode the contained message as any visual scene presenting illusions. The existence of perceptual illusions in handwriting makes its recognition, identification and understanding difficult. The main assumption of the proposed theory is that handwriting involves a concatenation of perceptual codes grouped together so as to obtain a shape, a character or a digit. Hence, the perceptual segmentation2 makes recognition easier by perceiving a character as a group of BPC. In order to compare our proposed works,

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-7f74153b-c5cb-4e56-9f85-5e0fc8345ce6.jpg)

segmented from ADAB Database display noticeable performances as compared to other already developed methods. Indeed, LMCA (Lettres Mots Chiffres Arabes) database [19], [20] contains 30.000 shapes for ten digits, 100.000 shapes for 56 Arabic letters and 500 Arabic words. 55 respondents were hosted to participate in the development of the handwritten LMCA.Weused54shapes instead of 56 Shapes. Some letters are manually segmented Letters and ligatures, extracted from the four sets (a,b,c and d) of ADAB database [36] to recognize isolated online Arabic handwritten characters. To test our system OnHSR-LSTM, we use 24861 samples for training and test, such as digits and Arabic letters (beginning, middle and end of the word) incorporating various styles of handwriting. The over-fitting can be reduced considerably by randomly omitting half of the hidden units on each training case by using Dropout. Random dropout yields remarkable improvements to our proposed. We perform our system 1 with Fuzzy Ground Truth in the range 94 % to 98.2 % as recognition rate. We attempt to use different combinations of one BPC or character in order to prevent the distortion of the model during the training step. Table 6 reveals that OnHSR-LSTM, describes both static and dynamic aspects of handwriting and by means of perceptual codes we codify the scripts into sequence of codes, in the form of an equation. Moreover, it demonstrates that the method of training fuzzy Ground Truth improved the recognition rate RR, yielding competitive results compared to similar works. Then, those results indicate the fact that the results increase intensively by applying data augmentation and dropout mechanisms. As shown in Table 7, our Perceptual Codes-LSTM system achieves the best recognition rate, especially when we use CTC and Fuzzy Ground Truth methods during training. Table 7 boosts the fact that Fuzzy LSTM is endowed with a more efficient performance,compared to framewise proposed approaches in terms of recognition rate.

Fig. 12. Arabic Alphabets and their forms at different positions.

we need to use the same lexicon which is Arabic in this section, extracted from MAYSTROUN [34] , LMCA and manually segmented from ADAB Database. MAYSTROUN, is composed of 400 letters, 300 digits, 200 words and 200 Arabic texts written by 15 writers [28]. Kherallah et al. [35] used Circular trajectory modeling and NN for Arabic digits recognition, wherein each digit is repeated 1000 times, and they obtained 95% as recognition rate. Compared to the results of [12], we obtained better recognition by achieving 98.5% of Arabic digit recognition. In Table 2 we compare our results with the results achieved by Tagougui et al. [2], [1]. In fact, with reference to Table 2, LMCA and manually

## 5.2 Latin Database

The problem with comparing online handwritten character recognition systems lies to the few number of available databases. Accordingly, we find an old database UNIPEN [37] which is used frequently, however,its evaluation is restricted to single Latin character recognition. It is used in [3]. Thus, we attempt to compare it with our recognizer system 1. The results are summarized in Table 8. We observe that the recognition rate is very competitive, especially with set 1(c), due to CTC and Fuzzy Ground Truth as well as for the prepossessing applying in online script.

So far, based on the aforementioned two precedent tables, using fuzzy training gives rise to an enhancement in terms of classification accuracy. In order to further ensure

TABLE 5 Comparison of recognition rate with Google system with UNIPEN set 1(c).

| System     | Methods        | RR with Framewise   | RR with Fuzzy Ground Truth   |
|------------|----------------|---------------------|------------------------------|
| [3]        | Language model | 94.9 %              | -                            |
| OnHSR-LSTM | BCP+LSTM       | 90.50 %             | 97.5 %                       |

the strength of the proposed approaches, additional tests are performed on another database IRONOFF [38]. The testing results of recognition of digits are illustrated in Table 8. In fact, this table demonstrates that our system 2, OnHR-convLSTM, achieves efficient results compared to our system 1, OnHSR-LSTM. To allow a fair comparison

TABLE 6 Comparison of recognition rate with IRONOFF Database.

| System        | Methods   | Recognition Rate   |
|---------------|-----------|--------------------|
| OnHSR-LSTM    | BCP+LSTM  | 97.5 %             |
| OnHR-convLSTM | convLSTM  | 98.00 %            |

between framewise and fuzzy training, the performances of the two proposed recognizers is shown in the next figure which describes the evolution of predicted and test fuzzy ground truth,illustrating that the two curves are close to each other.

Fig. 13. Classes Predicted and Correct ones.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-8e3a1a08-c786-4457-bd0e-9326ae332206.jpg)

## 5.3 Our Mobile Database

Based on mobile application of [7], we created a mobile database for acquired data from different persons with different styles of writing in order to examine the performance of our system. The (x, y, z) coordinates are stored in text file of online written script.

The online script can be presented as a variable length sequence.

$$i n p u t = [ ( x _ { 1 }, y _ { 1 }, z _ { 1 } ), ( x _ { 2 }, y _ { 2 }, z _ { 3 } ), \dots, ( x _ { n }, y _ { n }, z _ { n } ) ] \quad ( 8 ) \quad 7 \quad \text{N}.$$

where { x i } and { y i } are the xy-coordinates of the pen movements and { z i } is 0 when the pen is up and is 1 when the pen is down.

Moreover, experiments were conducted on this mobile database. Correspondingly, we remark that the results are very competitive with reference to previously published results, which presented in Table 8 [23] [39].

TABLE 7 Comparison of recognition rate with previous results system with

Mobile Database .

| System        | Methods   | Recognition Rate   |
|---------------|-----------|--------------------|
| OnHR-convLSTM | ConvLSTM  | 96.50 %            |
| OnHSR-LSTM    | BCP+LSTM  | 98.40 %            |

## 6 WORD RECOGNITION EXPERIMENTS

## 6.1 Word Recognition Experiments on ADAB

ADAB is a database of Arabic online handwritten words [40]. To evaluate our two systems we used it and good results are achieved as describes table 10 below.Table 10 shows that our system 2, OnHR-convLSTM, achieves best results compared to our system 1,OnHSR-LSTM .

TABLE 8 Comparison of recognition rate with ADAB Database.

| System        | Methods   | Recognition Rate   |
|---------------|-----------|--------------------|
| OnHSR-LSTM    | BCP+LSTM  | 96.50 %            |
| OnHR-convLSTM | ConvLSTM  | 97.0 %             |

## 6.2 Word Recognition Experiments on IRONOFF

The IRONOFF database [38] contains 31346 isolated word from a 196 word lexicon (French and English). Although the database contains both online and offlifne information of the handwriting signals, only the online information is used for our experiments. Table 9 reports the performance of our recognition systems Again, we achieve a very competitive rate with respect to other published results we are aware of.

TABLE 9 Comparison of recognition rate with IRONOFF Database.

| System        | Methods   | Recognition Rate   |
|---------------|-----------|--------------------|
| OnHSR-LSTM    | BCP+LSTM  | 93 %               |
| OnHR-convLSTM | ConvLSTM  | 98 %               |

## 7 NOISE

In order to study the robustness of the proposed recognizer system, a randomly noise signal is applied in both training and testing databases. Noise in handwriting can be ascribed to the support of writing, the person who write, and Parkinson person who produces trembled script. Figure 15 illustrates how our Beta-elliptic segmentation method eliminates trembling from script. Table 10 demonstrates that LSTM

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-12e7fd7e-6374-4b39-ad59-12d88f246d4f.jpg)

Fig. 14. Interfaces of Mobile Database.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-5f712ba6-9487-45b8-9694-bf6e716503fb.jpg)

TABLE 10 Recognition Rate with LMCA and manually segmented from ADAB Database with noise .

| System        | Methods                       | Recognition Rate   |
|---------------|-------------------------------|--------------------|
| OnHSR-LSTM    | BCP+LSTM                      | 96.00 %            |
| OnHR-convLSTM | (x,y,z)+ConvLSTM              | 50.45 %            |
| OnHR-convLSTM | (x,y,z)+ConvLSTM +teta+4 EPCs | 74.41 %            |

Fig. 15. Sample of input noise &amp; output

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-d7484b93-e33d-4a5b-9a4b-c276bb859339.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-2437cd9d-9b5e-458e-8d6d-d79eca6b1867.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-2a3cef9e-33ff-4a98-951d-93883d24f163.jpg)

with Beta-elliptic features yield best accuracy compared to (x,y,z), only because the Beta-elliptic method can improve the online script and eliminate, displaying its robustness with respect to noise. Therefore, our perceptual method, based on its preprocessing step can improve the quality of online script via removing noise and applying sampling, which is not available with OnHR-convLSTM system that neglects the preprocessing and improving step of the signal of the input handwritten script. However, OnHR-convLSTM with teta+4 EPCs provides result compared to only (x,y,z). Thus, the elliptic model is not only a segmentation model but is also with its methods of preprocessing, capable of improving the quality of acquired script, thus yielding competitive recognition rate. Beta-elliptic model can eliminate trembling from script. But also can detect with its parameters velocity and produce a different shape then a normal one So, Beta-elliptic a good solution to detect some assumptions of trembling in handwriting and can assists in the decision process leading to the diagnosis of some diseases related to trembles of handwriting or movements. Figure 16 illustrate the difference beta-elliptic points between normal script and trembled script.

Fig. 16. Sample of input noise &amp; output word dhahab I . ºX GLYPH&lt;9&gt;

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-55d71c2e-0f2f-447d-9769-49c6035036a1.jpg)

## 8 PERCEPTUAL ERRORS

The existence of perceptual illusions in handwriting makes its recognition, identification and understanding difficult. Figure 16 presents some examples of handwriting perceptual illusions. Those errors can be performed by our recognition language system. In Figure 16.(a) what do you see? : This handwriting shape can be identified as a letter 'O' or a digit '0', according to the context it appears in. The presented handwriting perceptual illusions proved

Fig. 17. Handwriting perceptual illusions : (a): the word he or be, (b): the letter B or the number 13, (e): the two Arabic words aalimon: GLYPH&lt;213&gt;GLYPH&lt;230&gt;˚« GLYPH&lt;10&gt; or alimon : GLYPH&lt;213&gt;GLYPH&lt;230&gt;¸ GLYPH&lt;10&gt; @ .

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-ae375913-c3a0-4373-bb18-f82c3376fec7.jpg)

that finding evidence, whether or not humans go through an identification process as a part of perception, has been difficult.

## 9 CONCLUSION

Two systems for online handwriting recognition are presented in this paper. The first is based on the fact that handwriting is composed of a group of Perceptual Codes and the second is generative system based on convolutional LSTM. We validate our system with three databases and mobile database, containing digits, letters and words in Arabic language. Good results are obtained almost identical to those produced by the human perceptual system. Encouraged obtained recognition rate is equal to 98.5%. We plan to build and test our proposed systems recognition in a large-scale database and other scripts.The perspective of improvement of our method is located in order to reach the stage of a sentence. Thanks to beta-elliptic model for handwriting strokes generation, we obtain supplementary information about dynamic domain. Furthermore, through this segmentation model, we can improve the quality of acquired trace.

## ACKNOWLEDGMENTS

The research leading to these results received funding from the Ministry of Higher Education and Scientific Research of Tunisia under the grant agreement number LR11ES48. Moreover, this project was carried out under the MOBIDOC scheme, funded by the EU through the EMORI program and managed by the ANPR.

## REFERENCES

- [1] N. Tagougui and M. Kherallah, 'Recognizing online arabic handwritten characters using a deep architecture,' Proc.SPIE , vol. 10341, pp. 15, 2017. [Online]. Available: https://doi.org/10. 1117/12.2268419
- [2] N. Tagougui, H. Boubaker, M. Kherallah, and A. M. Alimi, 'A hybrid NN/HMM modeling technique for online arabic handwriting recognition,' CoRR , vol. abs/1401.0486, 2014. [Online]. Available: http://arxiv.org/abs/1401.0486
- [3] D. Keysers, T. Deselaers, H. A. Rowley, L. L. Wang, and V. Carbune, 'Multi-language online handwriting recognition,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 39, no. 6, pp. 1180-1194, June 2017.
- [4] V. Ghods and M. K. Sohrabi, 'Online farsi handwritten character recognition using hidden markov model,' JCP , vol. 11, pp. 169175, 2016.
- [5] Y. Hamdi, A. Chaabouni, B. Houcine, and A. M. Alimi, 'Offlexicon online arabic handwriting recognition using neural network,' pp. 1-5, 2016.
- [6] W. Yang, L. Jin, Z. Xie, and Z. Feng, 'Improved deep convolutional neural network for online handwritten chinese character recognition using domain-specific knowledge,' pp. 551-555, Aug 2015.
- [7] J. Zhang, J. Du, and L. Dai, 'A gru-based encoder-decoder approach with attention for online handwritten mathematical expression recognition,' 12 2017.
- [8] X. Xiao, Y. Yang, T. Ahmad, L. Jin, and T. Chang, 'Design of a very compact cnn classifier for online handwritten chinese character recognition using dropweight and global pooling,' 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) , vol. 01, pp. 891-895, 2017.
- [9] X. Zhang, F. Yin, Y. Zhang, C. Liu, and Y. Bengio, 'Drawing and recognizing chinese characters with recurrent neural network,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 40, no. 4, pp. 849-862, April 2018.
- [10] Y. M.Pranoto, E. Setyati, E. Pramana, Y. Kristian, and R. Budiman, 'Real time handwriting recognition for mathematic expressions using hidden markov model,' pp. 1-6, 07 2016.
- [11] V. Carbune, P. Gonnet, T. Deselaers, H. A. Rowley, A. N. Daryin, M. Calvo, L. Wang, D. Keysers, S. Feuz, and P. Gervais, 'Fast multi-language lstm-based online handwriting recognition,' CoRR , vol. abs/1902.10525, 2019. [Online]. Available: http://arxiv.org/abs/1902.10525
- [12] G. Ren and V. Ganapathy, 'Recognition of online handwriting with variability on smart devices,' pp. 7605-7609, May 2019.
- [13] N. Tagougui, M. Kherallah, and A. M. Alimi, 'Online arabic handwriting recognition: A survey,' Int. J. Doc. Anal. Recognit. , vol. 16, no. 3, pp. 209-226, Sep. 2013. [Online]. Available: http://dx.doi.org/10.1007/s10032-012-0186-8
- [14] C. C. Tappert, C. Y. Suen, , and T. Wakahara, 'The state of the art in online handwriting recognition,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 12, p. 787808, August 1990.
- [15] S. Jaeger, S. Manke, J. Reichert, and A. Waibel, 'Online handwriting recognition: The npen++ recognizer,' Int. J. Doc. Anal. Recognit. , vol. 3, p. 169180, 2001.
- [16] J. Schenk and G. Rigoll, 'Novel hybrid nn/hmm modelling techniques for on-line handwriting recognition,' in Proc. Int. Workshop Frontiers Handwriting Recognit. , p. 619623, 2006.
- [17] A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, and H. B. andJ. Schmidhuber, 'A novel connectionist system for unconstrained handwriting recognition,' IEEE Trans. Pattern Anal. Mach. Intell. , vol. 31, p. 855868, May 2009.
- [18] S. Njah, H. Bezine, and A. M. Alimi, 'A new encoding system: Application to on-line arabic handwriting,' pp. 451-456, 2010.
- [19] S. Njah, M. Ltaief, H. Bezine, and A. M. Alimi, 'The pertohs theory for on-line handwriting segmentation,' IJCSI , vol. 9, pp. 142-151, 2012.
- [20] S. Njah, H. Bezine, and A. M. Alimi, 'On-line arabic handwriting segmentation via perceptual codes: application to mayastroun database,' pp. 1-5, March 2011.
- [21] J.R.Pinales, 'Reconnaissance hors-ligne de lcriture cursive par lutilisation de modles perceptifs et neuronaux,' PhD, University of Paris , 2002.
- [22] A.Malaviya, L.Peters, and P.Camposano, 'A fuzzy online handwriting recognition system: Fohres,' Proceedings of the International Conference on fuzzy theory and technology , pp. 1-15, 1993.
- [23] H. Akouaydi, S. Abdelhedi, S. Njah, M. Zaied, and A. M. Alimi, 'Decision trees based on perceptual codes for on-line arabic character recognition,' no. 1-5, pp. 153-157, April 2017.
- [24] M. Ltaief, S. Njah, H. Bezine, and A. M. Alimi, 'Genetic algorithms for perceptual codes extraction,' International Journal of Intelligent Learning Systems and Applications, JILSA , pp. 256-265, 2012.
- [25] S. Njah, H. Bezine, and A. M. Alimi, 'A new approach for the extraction of handwriting perceptual codes using fuzzy logic,' pp. 302-307, 2008.
- [26] H.Bezine, A.M.Alimi, and N.Derbel, 'A developmental theory of the relation between geometry and kinematics in handwriting,' IEEE International Conference Systems, Man and Cybernetics , p. 5, 2002.
- [27] M.Ltaief, H.Bezine, and A.M.Alimi, 'A spiking neural network model for complex handwriting movements generation,' Interna-

tional Journal of Computer Science and Information Security 14 , p. 7, 2016.

[28] S. Njah, H. Bezine, and A. M. Alimi, 'A fuzzy genetic system for segmentation of on-line handwriting: Application to adab database,' pp. 95-102, April 2011.

[29] T.Dhieb, W.Ouarda, H.Boubaker, M.B.Hlima, and A. M. Alimi, 'Online arabic writer identification based on beta-elliptic model,' pp. 74-79, 2015.

[30] T.Dhieb, W.Ouarda, H.Boubaker, and A. M. Alimi, 'Beta-elliptic model for writer identification from online arabic handwriting,' vol. 11, p. 6, 2016.

[31] T.Dhieb, S.Njah, W.Ouarda, H.Boubaker, M.B.Ayed, and A. M. Alimi, 'An online writer identification system based on betaelliptic model and fuzzy elementary perceptual codes,' p. 12, 2018.

[32] G. Alex, F. Santiago, G. Faustino, and S. Jurgen, 'Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,' New York, NY, USA, pp. 369-376, 2006. [Online]. Available: http://doi.acm.org/10.1145/ 1143844.1143891

[33] T. Bluche, H. Ney, J. Louradour, and C. Kermorvant, 'Framewise and ctc training of neural networks for handwriting recognition,' pp. 81-85, Aug 2015.

[34] S. Njah, B. B. Nouma, H. Bezine, and A. M. Alimi, 'Mayastroun: A multilanguage handwriting database,' pp. 308-312, Sept 2012.

[35] M. Kherallah, S. Njah, A. M. Alimi, and N. Derbel, 'Recognition of on-line handwritten digits by neural networks using circular and beta approaches,' vol. 2, pp. 164-169, Oct 2002.

[36] M.Kherallah, A.Elbaati, H.ElAbed, and A.M.Alimi, 'The on/off (lmca) dual arabic handwriting database,' 2008.

[37] I. Guyon, L. Schomaker, R. Plamondon, M. Liberman, and S. Janet, 'Unipen project of on-line data exchange and recognizer benchmarks,' vol. 2, pp. 29-33 vol.2, Oct 1994.

[38] V. Gaudin, P. Lallican, S. Knerr, and P. Binter, 'The ireste on/off (ironoff) dual handwriting database,' p. 455 458, 1999.

[39] H.Akouaydi, S.Njah, and A. M. Alimi, 'Android application for handwriting segmentation using pertohs theory,' Proc.SPIE , vol. 10341, pp. 10 341 -10 341 -5, 2016. [Online]. Available: https://doi.org/10.1117/12.2269059

[40] H. E. Abed, M. Kherallah, V. Mrgner, and A. M. Alimi, 'On-line arabic handwriting recognition competition, adab database and participating systems,' International Journal on Document Analysis and Recognition , vol. 14, pp. 15-23, 2011.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-5a892a60-382a-4261-9bbf-b75c74b8e756.jpg)

Hanen Akouaydi currently a PhD student in computer science at the National School of Engineers of Sfax. She obtained her engineering degree in Computer Science from Higher Institute of Multimedia Arts of Manouba in 2012. Her research interests include online handwriting recognition and deep learning. She is a member of the IEEE Student and the IEEE Computer Society and affiliate to REGIM Laboratory.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-04ed44e5-cc85-4c42-a48f-48d3eb05bddb.jpg)

Sourour Njah currently an Assistant Professor at the High Institute of Commerce of Sfax. She earned the M.S. degree in Computer Science from Faculty of Economics and Management of Sfax in 2003 and the Ph.D. degree in Computer Engineering from the National School of Engineers of Sfax in 2013. Her research domains are cursive handwriting analysis and fuzzy logic. She is a member of the IEEE and affiliate to REGIM Laboratory.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-f41ce57b-cccc-43f6-9b6e-133c63e1d05f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-d94b899b-7db1-437c-835f-0ccef68093f1.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-eb78b5c2-501b-4cb4-aad1-e720bfa84580.jpg)

Wael Ouarda is currently a Postdoctoral Researcher at the National School of Engineers of Sfax, Tunisia. He received earned the his M.S. degree in Computer Science from the INSA Lyon, France in 2010 and the Ph.D. degree in Computer Engineering from the National School of Engineers of Sfax in 2016. His research focused on Bio- metric systems, Information Fusion and Pattern Recognition. He is a member of the IEEE and he is affiliate to REGIM Laboratory.

Anis Samet the director of the site SIFAST in Tunisia since 2013. He received his M.S. in IT applied to management from the Faculty of Management of Sfax in 2004. His work focused on Team management, E-learning, Quality and process management,Business Intelligence,Piloting of teams,Business Unit, site, Piloting projects, etc.

Thameur Dhieb presently a PhD student of computer science at the Higher Institute of Computer Sciences and Communication Techniques of Hammam Sousse. He obtained his M.S. degree in Computer Science from the Faculty of Economics and Management of Sfax in 2012. His research interests include writer identification and signature verification. He is a member of the IEEE Student and the IEEE Computer Society and affiliate to REGIM Laboratory.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-b8a70081-e539-42fd-a1e2-fd0b8af9025f.jpg)

Mourad Zaied actually a Professor at the National School of Engineers of Gabes. He earned the Ph.D. degree and then the HDR both in Electrical and Computer Engineering in 2013. He is IEEE Senior member since 2000. He received his HDR, and his Ph.D degrees in Computer Engineering and the Master of science from the National Engineering School of Sfax respectively in 2013,2008 and in 2003. He obtained the degree of Computer Engineer from the National Engineering School of Monastir in 1995. Since

1997 he served in several institutes and faculties in the university of Gabes as teaching assistant. His research interests include Computer Vision and Image and video analysis. These research activities are centered on Wavelets and Wavelet networks and their applications to data classification and approximation, pattern recognition and image, audio and video coding and indexing.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00634v1-figure-056a70ec-4ad9-44fc-8188-e4382ad0ca87.jpg)

Adel M. Alimi currently a Professor at the National School of Engineers of Sfax. He obtained his Ph.D. degree and then his HDR both in Electrical and Computer Engineering in 1995 and 2000 respectively. His research interest includes applications of intelligent methods and vision systems. He is a guest editor of several special issues of international journals like Soft Computing. He is IEEE Senior member since 2000 and he is the director of REGIM Laboratory.
Hanen Akouaydi Sourour Njah Wael Ouarda Anis Samet Thameur Dhieb Mourad Zaied Adel M. Alimi
Neural Architecture based on Fuzzy Perceptual Representation For Online Multilingual Handwriting Recognition
Due to the omnipresence of mobile devices, online handwritten scripts have
become the most important feeding input to smartphones and tablet devices. To
increase online handwriting recognition performance, deeper neural networks
have extensively been used. In this context, our paper handles the problem of
online handwritten script recognition based on extraction features system and
deep approach system for sequences classification. Many solutions have appeared
in order to facilitate the recognition of handwriting. Accordingly, we used an
existent method and combined with new classifiers in order to get a flexible
system. Good results are achieved compared to online characters and words
recognition system on Latin and Arabic scripts. The performance of our two
proposed systems is assessed by using five databases. Indeed, the recognition
rate exceeds 98%.