extfieldsofstudy:
Computer Science
Engineering

text:
ZoomCount: A Zooming Mechanism for Crowd Counting in Static Images

This paper proposes a novel approach for crowd counting in low to high density scenarios in static images. Current approaches cannot handle huge crowd diversity well and thus perform poorly in extreme cases, where the crowd density in different regions of an image is either too low or too high, leading to crowd underestimation or overestimation. The proposed solution is based on the observation that detecting and handling such extreme cases in a specialized way leads to better crowd estimation. Additionally, existing methods find it hard to differentiate between the actual crowd and the cluttered background regions, resulting in further count overestimation. To address these issues, we propose a simple yet effective modular approach, where an input image is first subdivided into fixed-size patches and then fed to a four-way classification module labeling each image patch as low, medium, high-dense or no-crowd. This module also provides a count for each label, which is then analyzed via a specifically devised novel decision module to decide whether the image belongs to any of the two extreme cases (very low or very high density) or a normal case. Images, specified as high- or low-density extreme or a normal case, pass through dedicated zooming or normal patch-making blocks respectively before routing to the regressor in the form of fixed-size patches for crowd estimate. Extensive experimental evaluations demonstrate that the proposed approach outperforms the state-of-the-art methods on four benchmarks under most of the evaluation criteria.

I. INTRODUCTION
I N recent years, convolutional neural networks have attracted a lot of attention and been successfully applied to various computer vision problems, such as object detection [24], [29], [46], face recognition [8], depth estimation [16], [17], image classification [7], [50], image-to-image translation [44], [45], and crowd counting [34]. Crowd counting is an integral part of crowd analysis. It plays an important role in event management of huge gatherings like Hajj, sporting, and musical events or political rallies. Automated crowd count can lead to better and effective management of such events and prevent any unwanted incident [18]. Crowd counting is an active research problem due to different challenges pertaining to large perspective, huge variance in scale and image resolution, The work was supported in part by NSF NRI and USDA NIFA under the award no. 2019-67021-28996, and the Nvidia GPU grant.
U. Sajid [3] methods overestimate in case of crowd-like cluttered background patches as in (a) and (b), where there is no crowd at all. Similarly, these methods highly underestimate or overestimate in two extreme cases, where most crowd patches belong to either high or low-crowd count as in (c) and (d) respectively, as compared to the ground truth (GT).
severe occlusions and dense crowd-like cluttered background regions. Also, manual crowd counting subjects to very slow and inaccurate results due to the complex issues as mentioned above.
To obtain accurate, fast and automated results, CNN-based approaches have been proposed that achieve superior performance over traditional approaches [9], [10], [43]. CNNbased methods can be broadly classified into three categories; regression-based, detection-based, and density map estimation methods. Regression-based methods [41] directly regress the count from the input image. However, these CNN regressors alone cannot handle huge diversity in the crowd images varying from very low to very high. CNN detection-based methods [13], [32] first detect persons in the image and then sum all detection results to yield the final crowd count estimate. Detection-based methods perform well in low crowd images but could not be generalized well to high-density crowd images as detection fails miserably in such cases due to very few pixels per head or person. Density map estimation methods [22], [39], [40]   Left and right graphs compare ten cases each, belonging to high dense and low dense extreme respectively for Density map [3], DenseNet [20] based direct regression and our method. As shown, other models either highly underestimate or overestimate, whereas the proposed method remains the closest to the ground truth (GT) bar in most cases.
for each image pixel. The final estimate is then calculated by summing all density map values. These methods do not rely on localizing crowd but rather on estimating crowd density in each region of the crowd image. Density map estimation methods outperform other approaches and recent state-of-theart methods mostly belong to this category. However, density per pixel estimation remains a huge challenge as indicated in [31] due to large variations in the crowd density across different images. This naturally leads to a question: In which scenarios these methods may fail and why?
One key issue with regression and density map methods is that they only rely on direct count estimate and density map estimation per pixel for the input image respectively, thus, they may get subjected to large crowd count for cluttered background image patches. As shown in Fig. 1, models [22] based on these methods consider this 224 × 224 image patch as a crowd patch and make false estimates, making the system unreliable as similar patterns are bound to occur in many practical scenarios.
In addition, we observe that both types of methods perform well for images which contain most crowd patches with neither low nor high crowd density. Problem arises when images have most crowd patches with either high or low-density crowd. Due to the limitation in handling such practical diversity in crowd density, these methods may either highly underestimate or overestimate the crowd count in these two extreme cases, as shown in Fig. 1. To further explain this phenomenon, we analyze ten such cases for both extremes separately from very recent UCF-QNRF dataset [22] on the state-of-the-art density map method [3], [22] and direct regression-based method as shown in Fig. 2. It can be observed that, in both extreme cases, the crowd estimates are either highly overestimated or underestimated due to the limitations as discussed above.
To solve these fundamental problems, we propose a modular approach as shown in Fig. 3. It comprises of a Crowd Density Classifier (CDC), a novel Decision Module (DM), and a Count Regressor Module (CRM). The input image is first sub-divided into fixed-size patches (224×224) and fed to the CDC module that contains a deep CNN classifier to perform a four-way classification (low, medium, high-density, and no-crowd) on each patch. The classification module eliminates any crowd- Fig. 3. ZoomCount architecture. The test image X, divided into 224 × 224 patches, first passes through the Crowd Density Classifier (CDC) module, which discards no-crowd patches as classified by the robust 4-way DenseNet classifier. The accumulator stores patch count per class (P CC X ) of this image. Decision Module (DM), based on CDC module output and using either autonomous RFDB or heuristic-based RSE module, decides whether this image should be divided into all normal (N ormal) patches or make all either zoom-in (Z in ) or zoom-out (Zout) based patches before proceeding to the patch-based regressor (COU N T ER) for each patch crowd count. Image final crowd estimate is then obtained by summing all patches count. like background patches (no-crowd) from the test image and feeds the information about the number of patches belonging to each of the no-crowd, low, medium and high-density classes to the Decision Module (DM) using an accumulator. DM uses either the machine learning based RFDB module or heuristicbased Rule-Set Engine (RSE) module to determine if the image belongs to a case of low, normal, or high density. Based on the DM decision, this image is then divided into fixed-size patches using one of three independent image patch-making modules (Z in , N ormal, Z out ). The image that belongs to lowdensity extreme case is divided into patches using the zoomout (Z out ) patch-maker; the image that belongs to high-density extreme case is divided via zoom-in (Z in ) patch-making block; and the normal case image is split into patches using normal (N ormal) patch-maker. These patches are then routed one by one to the patch-wise count regressor (COU N T ER) for crowd estimate and the image total crowd count is obtained by summing all patches count.
The Z in block divides each input patch into four 112 × 112 patches, and then up-scales each patch by 2× before routing each patch to the count regressor. Intuitively, this module is further zooming-in into the image and looking in-detail all patches by using 1/2 input patch size instead of the original 224 × 224 patches. Similarly, zoom-out patch-maker divides the input image into 448 × 448 patches, and down-scales each patch by 2× as it is dealing with the image containing lowdensity crowd patches mostly. The normal case image directly employs the original 224 × 224 patch size with no up-scaling or down-scaling. The main contributions of this work include: • The paper reveals and analyzes the fact that extremely high and low dense crowd images greatly influence the performance of the state-of-the-art regression and density map based methods for crowd counting.
• A novel strategy is proposed to address the problem of counting in highly varying crowd density images by first classifying the images into either one of the extreme cases (of very low or very high density) or a normal case, and then feeding them to specifically designed patch-makers and crowd regressor for counting.
• A novel rule-set engine is developed to determine whether the image belongs to an extreme case. For images of extremely high density, a zoom-in strategy is developed to look into more details of the image; while for images of low-density extreme, a zoom-out based regression is employed to avoid overestimate.
• We created four new datasets, each from the corresponding crowd counting benchmark, for the training and testing of different machine learning algorithms to classify an image as normal, high or low-dense extreme case using its patches classification count. These manually verified datasets will facilitate the researchers in analyzing complex crowd diversity, which is at the core of the crowd analysis. The proposed ZoomCount scheme is thoroughly evaluated on four benchmarks: UCF-QNRF [22], ShanghaiTech [49], WorldExpo'10 [48], and AHU-Crowd [19]. The experimental results demonstrate the effectiveness and generality of the proposed strategy and rule-sets, which are never realized for crowd counting. The overall performance of the proposed model outperforms the state-of-the-art approaches on most of the evaluation criteria. The proposed models and source code, as well as the created datasets, will be available on the author's website.

II. RELATED WORK
Crowd counting is an active research area in computer vision with different challenges related to large perspective, occlusion, cluttered background regions and high variance in crowd density across different images. Earlier work [42], [43] focused on the head or full-body detection for counting using handcrafted features for detectors learning. These methods failed in case of high dense images, where it is hard to find such handcrafted features. The approaches were shifted towards regression based counting [9], [10], [33], where a mapping function was learned to directly regress count from local patches of an image. These methods improved the counting process, however, they could not handle huge crowd diversity and also lack awareness about crowd density across all parts of the image.
Recently, CNN-based approaches have been widely used [22], [26], [27], [41]. They are broadly categorized into three classes; Counting by detection, counting by direct regression, and counting using density map estimation. CNN-based object detectors [13], [32] detect each person in the image, and the final count is then calculated by summing all detections. These methods [25], [36] deteriorate in high density and severe occlusion cases, where each head only occupies a few pixels. Counting by direct regression methods [41] directly regress count by learning feature maps from the input image patch. Wang et al. [41] proposed an end-to-end AlexNet [23] based regressor for crowd count. These methods alone cannot handle huge diversity in different crowd images.
Density map estimation methods [6], [22], [26], [27], [39] learn to map crowd density per pixel of an image without localizing the counts. The final estimate is calculated by summing all density estimations. Zhang et al. [49] proposed a three-column CNN architecture (MCNN) to handle crowd diversity across images. Each column is designed to handle different scales using different receptive field sizes. Sindagi et al. [40] extended the idea of MCNN to incorporate contextual information for high-quality density maps generation. Recently, Sam. et al. [35] proposed SwitchCNN which routes each input patch to one of three independent CNN regressors using a switch CNN classifier. Based on the classification and regression idea, Sindagi et al. [39] designed a Cascaded-MTL that estimates count for the whole image by using cascaded 10-way classification prior and final density map estimation.
Crowd counting models, based on whole image estimation and training from the scratch, are subjected to over-fitting due to limited dataset availability (only a few hundred training images). Thus, patch-based models are widely used nowadays. The final sum is computed by adding up all patch count estimates. Liu et al. [27] proposed a hybrid approach by incorporating both regression and detection blocks using an attention-guided mechanism to handle low and high-density cases simultaneously. Li et al. [26] designed a CSRNet to get multi-scale contextual information by incorporating dilationbased convolutional layers. Idress et al. [22] proposed a composition loss based model for simultaneous crowd counting and localization. Existing methods perform worse in extreme cases where most crowd patches belong to either high density or low density. Moreover, these methods lack the ability to fully discard any cluttered background regions in the image, thus resulting in overestimate.

III. PROPOSED APPROACH
The proposed framework is shown in Fig. 3, which is composed of three modules namely Crowd Density Classifier (CDC), Decision Module (DM) and Count Regressor Module (CRM). The input image is first sub-divided into 224 × 224 size patches and each patch then passes through the CDC module for 4-way classification (low, medium, high-density or no-crowd). The accumulator gathers and feeds patch count per class information to the Decision Module. Based on accumulator information and utilizing either Random Forest based Decision Block (RFDB) or heuristic-based Rule-Set Engine (RSE), DM routes this image to one of three specialized patch-making blocks (Z in , N ormal, Z out ) of CRM where the input image is divided into corresponding patches, followed by the crowd estimate for each patch via crowd regressor (COU N T ER). Finally, the image crowd count is calculated by summing all patches count. Below we will discuss the details of each module, as well as the rules defined for the two possible extremes.

A. Crowd Density Classifier (CDC) Module
The CDC module is composed of a deep CNN 4-way classifier that specializes in making a distinction between nocrowd (NC), low-density (LC), medium-density crowd (MC), and high-density crowd (HC) for each input patch. Let X be a test image sub-divided into N patches [x 1 , x 2 , ...x N ], each with a size of 224 × 224. The accumulator gathers each patch classification result for the input image X as follows: for i = 1, 2, ....N and y belongs to either NC, LC, MC or HC class label. In the end, the accumulator passes the patch count per class (P CC X ) of this image to the decision module (DM) as: where P N C , P LC , P M C and P HC denote the total number of patches being classified as NC, LC, MC and HC respectively of the image X. Patches being classified as NC are discarded, and thus remaining {N − P N C } crowd patches are going to be used for final crowd estimate. As a result, the crowd-like cluttered background regions (such as the tree leaves shown in Fig. 1), which may result in overestimation otherwise, will be eliminated.
Definitions of NC, LC, MC and HC class labels. During experiments for each crowd counting benchmark dataset, we randomly extract patches from its training images for the CDC classifier training and assign a ground truth class label (NC,LC,MC,HC) to each extracted patch. Since these datasets also contain the localization of people, so we generate the ground truth class label for each patch using this information Blue blocks represent Dense Blocks 1, 2, 3 and 4 from left to right, followed by a fully connected layer and final softmax 4-way classification. 224 × 224 size input patch is expected. and the maximum people count possible in any image patch of the corresponding dataset. LC class label is assigned to a patch if the ground truth people count for that patch is less than or equal to 5% of the maximum possible count but greater than zero as zero crowd means NC class patch. Similarly, patches with ground truth people count between 5% to 20% of the maximum possible count are assigned the MC class label, while patches containing more than 20% of the maximum people count are labeled as HC category patches. In the end, a total of 90, 000 patches, with an equal amount per class label, are generated for the CDC classifier training in each benchmark setting. Example patches for each class label are shown in Fig. 4.
Classifier Details. We use DenseNet-201 [20] as our 4-way classifier, as shown in Fig. 5. It has four dense blocks with transition layers (convolution and pooling) in between them to adjust feature maps size accordingly. The DenseNet-201 has consecutive 1 × 1 and 3 × 3 convolutional layers in each dense block in {6, 12, 48, 32} sets respectively. At the end of the last dense block, a classification layer is composed of 7 × 7 global average pooling, followed by 1000 − D fully connected layer and the final 4-way softmax classification with cross-entropy loss.

B. Decision Module (DM)
The decision module, based on the CDC module output P CC X , decides if the test image should be treated as a normal image or a low or a high-density extreme case image. DM makes this decision by utilizing one of the two separate and independent decision-making blocks, namely Rule-Set Engine (RSE) and Random Forest based Decision Block (RFDB). RSE is a novel heuristic-based approach which employs the rule-sets to detect if the test image is either an extreme or a normal case, while RFDB is an automated decision-making block based on Random Forest algorithm that learns to map the test image features (P N C , P LC , P M C , P HC ) to the respective class label (Z in , N ormal, Z out ). We also create new RFDB training datasets, each from corresponding crowd counting benchmark, for the training of RFDB module as explained in Sec. III-B3.
1) Rule-Set Engine (RSE): The accumulated patch count per class (P CC X ) from CDC module is tested against two different rule-sets to determine if an input image is a case of low or a high density extreme or a normal one so that

Algorithm 1 Rule-Set Engine Algorithm
Input: P CC X (Patch Count per Class for Test Image X)= {P NC , P LC , P MC , P HC } Output: Normal or Z in or Z out Let P all = P N C + P LC + P M C + P HC if input patch count satisfies any of following rules then if P N C > 0 and P HC <= (P all * 0.05) end else if input patch count satisfies any of following rules then Output = Z in Rule 5: if P LC + P M C == 0 Rule 6: if P LC + P HC == 0 Rule 7: if P HC > (P all * 0.50) Rule 8: if P N C > 0 and P M C >= (P all * 0.33) and P HC >= (P all * 0.33) end else Output = N ormal it can be divided into patches using the most suitable patchmaking block (Z in , N ormal, Z out ). The overall goal of RSE is to encourage an image with more number of high-density patches to pass through zoom-in patch-making block (Z in ), whereas the image with more number of low-density patches goes through a zoom-out patch-making block (Z out ). If the image does not belong to any of the two extreme cases, it will be treated as a normal case that uses the normal patch-maker (N ormal).
Rules. The RSE module consists of two generalized rulesets, aiming to detect the images belonging to any of the two extreme cases: the low-density extreme (Rules 1-4) and the high-density extreme (Rules 5-8). As illustrated in Algorithm 1, if no rule applies to the test image X, it will use N ormal patch-maker, whereas the image satisfying any rule from (1 − 4) or (5 − 8) will generate its patches using Z out or Z in patch-making blocks respectively. Each rule is explained in detail in Table I. This table also shows the most affected images by a specific rule in terms of resolution. For example, Rule 7 is highly applicable on relatively lower resolution (LR) images, whereas Rule 2 can affect images of any resolution equally. It is important to note that these rule-sets are used consistently and evaluated across all four publicly available datasets in the experiments, thus demonstrating the generality and efficacy of such rule-sets. In addition, the current rule sets are extendable by adding more rules to refine the classification/decision process. Please note that all parameters in Table  I are chosen empirically. 2) Random Forest based Decision Block (RFDB): The scalable rule-sets based decision process yields promising results as demonstrated throughout the experiments in Sec. V. Nevertheless, there are many heuristics to handle and it requires manual input and special attention while inducting new rules. To address this issue, we propose an automated machine learning based approach that learns the decision process by mapping the four features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) to respective class label (Z in , N ormal or Z out ) for each image, where the features denote percentages instead of total image patches belonging to NC, LC, MC and HC classes respectively and labels represent zoom-in, normal and zoomout patch-making blocks required to generate the patches from the particular input image before proceeding to the count regressor. We employ percentages for features because of the huge variance in resolution across different images in a dataset, which directly influences the features and hence training quality. In addition, since there is no such dataset available for the crowd counting problem to-date that can help in learning this mapping, thus we generate a new RFDB training dataset from each corresponding benchmark as explained in detail in next subsection. To automate the decision block process, we explored different machine learning classification models and found the random forest-based model to be the most effective as demonstrated in the experiments in Sec. V. Thus, we choose the random forest algorithm and hence named this module as Random Forest based Decision Block.
Random Forest (RF), being a bootstrap aggregation or bagging based ensemble method, can be used both for classifi-cation and regression. We employ the RF algorithm to classify the four features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) to a class label of (Z in , Normal or Z out ) by building, training and tuning a large collection of de-correlated binary decision trees. Each tree then casts a vote for class prediction for the test sample. Finally, the class label with a majority vote is assigned to that test sample i.e., the input image.
Each RF decision tree t k is built using a bootstrap sample BS(t k ) which is generated from the training data. Such bootstrap sample is given as: for K = 0, 1, 2, ...N − 1, where N denotes the total number of RF trees. Each row represents one training sample for the tree t k with the class label as the last entry. We use N = 100, which is set empirically as no significant improvement has been observed in performance beyond this number. The trees are grown using the classification and regression (CART) algorithm, where the nodes get split until all leaves become unmixed or contain less than m min samples [1]. We use m min = 2 throughout our experiments, thus splitting nodes until they contain either only one sample or become pure. To quantify the quality of a tree node split, Gini Impurity has been used as: where L denotes the total unique class labels and F i denotes the frequency of class label i at node n. During testing, each RF tree gives its class prediction for test image X. Final class label is obtained by the majority vote criterion [14] as follows: where C k (X) represents the class prediction by the k th RF tree. Feature Importance (FI) depicts the role of each feature in determining the node split and eventually the quality of the RF decision trees building. Features with much lesser FI value can be easily discarded as they do not play any significant role in decreasing the node impurity. As shown in the left graph in Fig. 6, all four features have approximately the same FI values in each RFDB dataset. Thus, we keep and use all four available features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) in all four newly generated RFDB datasets.
3) Dataset generation for RFDB: The RFDB module learns to map the image extracted features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) to the respective class label (Z in , N ormal or Z out ) using training dataset with the corresponding mapping. No such dataset has been created to-date. Thus, for each benchmark (ShanghaiTech [49], UCF-QNRF [22], AHU [19]), we created a new respective RFDB dataset which contains this mapping. To create the new RFDB dataset, each training image's required features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) are extracted using ground truth crowd localization information and definitions of class labels (N C, LC, M C, HC) as stated in III-A, followed by manual verification and ground truth (GT) class label assignment. To ensure the quality of the generated dataset, each sample entry was then double checked for any inconsistency, duplicates, missing and erroneous cases. For the extracted features, we use percentages instead of the actual number of patches (P N C , P LC , P M C , P HC ) belonging to each category because of the huge resolution difference across the images within each benchmark dataset.
Statistics. For each of the four crowd counting benchmarks, we create the corresponding RFDB dataset using its corresponding training images. For instance, in the case of ShanghaiTech dataset (300 training images), we generate the new 300 samples RFDB dataset with each entry being created using one of the respective training image, followed by manual verification that also includes removal or modification of inconsistent entries. In total, 220, 2830 and 812 samples are finalized for the three RFDB datasets based on ShanghaiTech [49], WorldExpo'10 [48] and UCF-QNRF [22] benchmarks, respectively. For AHU [19] based RFDB dataset, 90 out of 96 available entries are kept on average with 5-fold crossvalidation. The lower graph in Fig. 6 shows the percentage of each class label in all four newly created RFDB datasets.

C. Count Regressor Module (CRM)
The CRM module comprises of three independent patch-making blocks and a deep CNN count regressor (COU N T ER). The decision module routes the test image to one of these patch-makers for dividing it into 224 × 224 patches after required up-scaling or down-scaling, followed by the crowd count for each image patch via the count regressor (COU N T ER). The regressor employs DenseNet-201 [20] inspired architecture with a single neuron after the fully connected layer to directly regress the crowd count. Mean squared error (MSE), as defined below, has been employed as the loss function for the count regressor c : where N is the number of training patches per batch, Y i is the ground truth crowd count for the input patch X i , and F is the function that maps the input patch X i to the crowd count with learnable parameters Θ. Zoom-in based Patch Maker (Z in ): Ideally, the decision module (DM ) routes the image, with most crowd patches being classified as high-density crowd, to this patch-maker. The image, using this patch-maker, is further sub-divided into equal 112 × 112 patches, and then up-scaled by 2× before proceeding to the count regressor for each patch crowd count. Intuitively, it looks into each patch in detail by estimating the count on smaller zoomed-in highly crowded patches. In this way, it greatly stabilizes and improves the count estimate for high-density images, where other methods may either underestimate or overestimate too much due to fixed patch sizes, as demonstrated in the experiments Sec. V.
Zoom-out based Patch Maker (Z out ): This block is responsible for handling the low-density extreme case images as detected and routed by the decision module. Z out takes 448 × 448 original patches of the test image X, down-scales them by 2 times, and feeds each resultant patch to the CDC classifier to eliminate any no-crowd patches, as shown in Fig.  7. The count estimate for each crowd patch is then computed through CRM count regressor (COU N T ER) followed by the image total count estimate, which is the sum of all patches crowd counts. In other words, it assists the count regressor by using larger area per input patch (448 × 448 down-scaled to 224 × 224) which alleviates the overestimation problem.
Normal case: In N ormal case, the images are divided into 224 × 224 size patches with no up-or down-scaling before patch-wise count regression. It is also worth mentioning that there is no need to explicitly look for and eliminate any nocrowd patches in case of N ormal and Z in case images as such background patches are automatically removed during the CDC module classification process, and thus we can also reuse the remaining CDC module crowd patches in both cases for crowd estimate.
The input image that is classified as N ormal case, may contain mixed crowd numbers in different regions. Empirically, it has been observed that the deep CNN based crowd counter (i.e. the CRM regressor) can directly handle such images effectively without any rescaling or classification process. Thus, this work only focuses on the images of extreme case (low-density or high-density) that contain most of the regions with same crowd level (low-or high-density), since these extremes have huge influence on crowd count, and a special attention to these cases will significantly mitigate the over-or under-estimation issue as discussed in the introduction.

D. ZoomCount and Switch-CNN Comparison
Switch-CNN [35] is one of the state-of-the-art crowd counting models, and it classifies the input image each patch into one of three crowd-density labels (low, medium, and high). However, our approach employs this classification process fundamentally different from Switch-CNN. The key differences and comparison are as follows: • Crowd-Density Classification Purpose and Usage.
Switch-CNN primarily uses the crowd-density classification to route the input patch to one of three specialized count regressors. Subsequently, crowd estimation for each patch is being done using one of these deep count regressors. On the other hand, we employ the crowddensity classification process to facilitate the Decision Module (DM) in detecting whether the input image is either one of the two extremes (low-or high-density) or a normal case image. Then, the patches are rescaled accordingly before proceeding to the only count regressor for the crowd estimation. Thus, the underlying usage and purpose of crowd classification completely differ in both methods. • Patch-based vs Image-based Decision. Based on the crowd-density classification as discussed in the above point, Switch-CNN selects the appropriate crowd count regressor individually for each input patch (patch-wise decision). While ZoomCount selects the most appropriate rescaling operation suitable for the whole image (imagebased decision) using this information. Thus, our method also takes the global context of the image into consideration during the decision-making process instead of relying solely on the local image patches.
In addition, Switch-CNN trains using multiple complex training steps including pretraining, differential training, switch training, and coupled training, while ZoomCount employs standard classifier and regressor training process. Switch-CNN uses three specialized count regressors, whereas the proposed approach simplifies this process by employing only a single count regressor. Moreover, our method outperforms the Switch-CNN under all evaluation metrics as demonstrated in the experiments section.

A. Training Details
The CDC classifier and the count regressor (COU N T ER) expect fixed size patch of 224 × 224 as the input. For both modules, we randomly extract 112 × 112, 224 × 224 and 448 × 448 patches from the training images. Around 90,000 such patches with mixed crowd numbers are generated for each of these modules. The count regressor is trained for 80 epochs with Adam optimizer and a batch size of 16 and starting learning rate of 0.001, decreased by half after every 20 epochs. The classifier employs the stochastic gradient descent (SGD) based optimization with multi-step learning rate starting at 0.1 and decreased by half after 25% and 50% epochs with 80 epochs in total. For each dataset, around 10% training data has been used for validation as recommended in the corresponding literature. For the random forest algorithm in RFDB, we utilize machine learning library scikit-learn for python programming. The Random Forest model was trained using 100 RF decision trees, where each RF tree is trained using the bootstrapped sample with Gini Imprity as node split quality criterion. 10% of the training data has been used for validation in case of each RFDB dataset.

B. Evaluation Details
In order to make a fair and consistent comparison with other methods, we employ three evaluation metrics namely Mean Absolute Error (M AE), Mean Normalized Absolute Error (M N AE) and Root Mean Squared Error (RM SE) defined as below: where N denotes the total number of test images, and Y i andŶ i are the ground truth and the estimated counts respectively for the test image i.

V. EXPERIMENTS
In this section, we demonstrate both quantitative and qualitative results from extensive four benchmark datasets: UCF-QNRF [22], ShanghaiTech [49], WorldExpo'10 [48], and AHU-Crowd [19]. These datasets contain images with huge crowd variance, different camera perspective, and complex cluttered background regions. Details about each benchmark are given in Table II. At the end of this section, we also discuss computational time analysis and compare that with state-ofthe-art methods.
Two different versions of the proposed model, the Rule-Set Engine (ZoomCount-RSE) based and the automated RFDB module (ZoomCount-RFDB) based version, are being compared separately with the state-of-the-art techniques throughout this section. Both ZoomCount versions give almost identical and much better performance under most of the evaluation criteria on the four benchmark datasets.

A. Experiments on UCF-QNRF Dataset
The dataset was recently published by Idrees et al. [22], which is a challenging and the first dataset of its kind. On one hand, it contains images with resolution as high as (6666 × 9999) and as low as (300 × 377); on the other hand, crowd count per image ranges from a maximum value of 12, 865 to a minimum count of 65. The total number of annotations in this dataset is 1, 251, 642, indicating the level of crowd complexity. It contains 1535 images in total, out of which 1201 and 334 images are used for training and testing respectively. We compare ZoomCount with the state-of-the-art methods and tabulate the results in Table III. It is evident that both versions of our method outperform all other approaches in terms of MAE and MNAE; while performing competitively closer to the best in terms of RMSE.
In order to evaluate the influence of different rules, we perform the ablation experiments, as shown in Table IV. We analyze the effect of all rules (R1 to R8) by removing them one at a time in ZoomCount-RSE version of the proposed method. As shown in the results, doing so greatly decreases the performance of our method, thus demonstrating the importance of those rules. We also analyze the effect of removing both or either of the zoom-in and zoom-out patch-makers in the experiments. From the results in Table IV, it is evident that both modules play an effective role in improving the overall performance of our method. The last three columns of Table  IV show the number of the test images passed through the zoom-in, normal and zoom-out patch-makers respectively. In the original setting, 75 (∼ 22%) images passed through the zoom-in patch-maker, whereas the zoom-out block handled 162 (∼ 48.5%) images and normal patch-maker was used only for 97 (∼ 29.5%) images, showcasing quantitative importance of these extreme case handlers, as shown in Fig. 8. We also compare the crowd estimate of ten test images each, for both extreme cases with DenseNet [20] direct regression and the state-of-the-art CL [22] density map method. Our method performs much better in both cases, as shown in Fig. 2.

B. Experiments on ShanghaiTech Dataset
The ShanghaiTech part A dataset contains a total of 482 images with 241,677 annotations, randomly collected from the internet, with a split of 300 and 182 images for training and testing respectively. We compare our method with the stateof-the-art methods as shown in Table V. The results show that our method outperforms all other methods on every evaluation  [40] 73.6 -106.4 CSRNet [26] 68.2 -115.0 IG-CNN [4] 72.5 -118.2 L2R [28] 72.0 -106.6 ICC [31] 68.5 -116.2 SA-Net [6] 67.0 -104.5 Deep-NCL [37] 73  Table VI, where we remove each rule one at a time. It is clear that the error increases by removing these rules. In the same table, We also analyze the effect of removing the zoom-in and zoom-out blocks separately and together. As expected, the performance plunges dramatically as error increases without using them. The last three columns show the number of test images passing through the zoom-in, normal and zoom-out patch-makers respectively. In the original setting, 21 (∼ 11.5%), 121 (∼ 66.5%) and 40 (∼ 22%) images are handled by the zoom-in, normal and Fig. 9. 182 test images are divided into ten groups with total crowd count in each group increasing from left to right. Each group contains 18 images except group number 10. Vertical axis indicates average count for each group. It is evident that ZoomCount remains closer to the ground truth (GT) bar in most cases as compared to the state-of-the-art methods. zoom-out blocks respectively, thus proving the quantitative importance of all of them and associated rules in ZoomCount-RSE, as shown in Fig. 8. In Fig. 9, we analyze the performance of our method on the average count across image groups with different total crowd counts. As compared with the state-ofthe-art methods, ZoomCount performs the best in most cases.

C. Experiments on WorldExpo'10 Dataset
The WorldExpo'10 [48] is a large dataset, composed of 1132 video sequences taken by 108 different cameras. The training set consists of 3380 images from 103 different scenes, whereas the testing set has 5 scenes with a total of 600 frames. This benchmark also consists of Region of Interest (RoI) and perspective maps. We only utilize the RoIs in the images during training and testing stages. The MAE based evaluation results on the five test scenes and the average MAE error are shown in Table VII. As shown, the proposed model achieves the lowest average MAE and the best performance on two scenes (S1 and S3). For the other three scenes, the proposed scheme yields reasonable and competitive results to the stateof-the-art. During the testing experiments on ZoomCount-RSE model, 168 (∼ 28%), 240 (∼ 40%) and 192 (∼ 32%) images are handled by the zoom-in, normal, and zoom-out blocks, respectively, as shown in Fig. 8. This demonstrates the quantitative importance of the associated rules in ZoomCount-RSE.

D. Experiments on AHU-Crowd Dataset
AHU-Crowd [19] dataset contains 107 images with 45, 807 human annotations. The crowd count ranges from 58 to 2201 per image. As per the standard being followed for this dataset [19], we performed 5-fold cross-validation and evaluated our method using the same three evaluation metrics. ZoomCount outperforms all the other methods as shown in Table VIII. It is worth-mentioning that ZoomCount decreases MAE and MNAE significantly by ∼ 7.5% (81 to 74.9) and ∼ 4.5% (0.199 to 0.190) respectively, whereas RMSE decreases drastically by ∼ 11.2% (124.9 to 111).

E. CDC Classifier and CRM Regressor Architecture Selection
Choosing an appropriate network architecture for the CDC classifier and the CRM regressor is essential for the effectiveness of the proposed scheme. In this section, we analyze different state-of-the-art architectures for this objective, including VGG-16 [38], ResNet-101 [15], and DenseNet-201 [20]. For the classifier choice evaluation, the final 1000-way classification layer in VGG-16 and ResNet-101 is replaced with a 4-way classification layer. Similarly, for the CRM regressor evaluation, the final FC layer in VGG-16 and ResNet-101 networks is followed by a single neuron to directly regress the crowd count with mean square error (MSE) loss function as given in Eq. 6. As per the standard practice, VGG-16 is configured with the batch-normalization layer being added inbetween each convolution and activation layer. DenseNet-201 classifier and regressor are configured as stated in sec. III-A, and all architectures are trained as discussed in section IV-A.
The results on the test images of ShanghaiTech Part-A dataset are shown in Table IX. We can see from the results that DenseNet-201 based classifier outperforms other state-ofthe-art architectures with the highest classification accuracy and ∼ 7% improvement, thus, we choose this model as the CDC classifier in the proposed method. For the choice of the CRM Regressor, we evaluated different architectures in terms of mean absolute error (MAE) in two separate experimental settings. In the first case, each regressor architecture has been evaluated separately by plugging-in into the proposed scheme as the CRM Regressor with the same architecture, while in the second experiment setting, each regressor architecture has been evaluated independently as a standalone crowd count regression model. The results of both experimental settings are shown in the second-rightmost and the rightmost columns respectively of Table IX. From the results, we can see that using DenseNet based architecture as the CDC classifier and CRM Regressor achieves the highest classification accuracy and the lowest MAE errors, respectively.

F. RSE and RFDB based Decision Modules Performance
The classification accuracy of the decision module directly affects the overall performance of the proposed method. In this section, we show the individual classification accuracy performance of the RSE and RFDB based DM modules on the four benchmarks. The accuracy results are shown in Table  X, from which we can see that both modules perform quite effectively. Also, the RFDB version of the DM module gives slightly better accuracy in most cases compared to the RSE based DM module. The overall system outperforms the stateof-the-art owing to the improved and reasonable classification accuracy of the DM module.

G. RFDB Algorithm Selection
In this paper, we adopt the Random Forest algorithm for the RFDB module. In practice, other machine learning-based classification algorithms can also be employed. In order to choose the best one for our system, we experimented with different classifiers to select the appropriate decision-making algorithm. The results based on the ShanghaiTech and UCF-QNRF datasets are shown in Table XI. We observe that ensemble based methods perform better on our relatively smaller and imbalanced RFDB datasets as they prevent over-fitting and high variance by combining several machine learning techniques. After evaluation, the Random Forest appears to be the best choice as the RFDB algorithm as shown in Table  XI, where the top five best results justify the selection of the Random Forest algorithm. For these experiments, we used machine learning library scikit-learn for python programming [2].

H. Qualitative Results
In Fig. 10, we show some good and bad case qualitative results from UCF-QNRF and ShanghaiTech datasets. We also compare our results with the ground truth (GT), DenseNet [20] Regression (DR) and the state-of-the-art density map methods. In each row, the first three cases demonstrate the good results followed by two bad estimates. The bad case results happen mostly due to the test image being detected as wrong extreme case type by the decision module (DM). We also show some crowd-density classification visual results to demonstrate the qualitative performance of the CDC classifier in Fig. 11.

I. Computational Time Analysis
In this experiment, we compare the proposed network with two state-of-the-art models (CSRNET [26],CP-CNN [40]) in terms of real computational time on the ShanghaiTech [14] dataset. The results are shown in Table XII, where T total and T avg represent the total and average time taken for the whole dataset test images respectively, whereas, T smallest and T largest represent the computation time taken for the smallest and the highest resolution test image respectively in the dataset. For the fairness of comparison, all networks have been evaluated on the same NVIDIA Titan Xp GPU.
As shown in the Table XII, our method takes a reasonable computation time (in between the two state-of-the-art networks with much closer to the one with the lesser time). However, the proposed method outperforms other models on all standard evaluation metrics. Since our method is modular, we have also shown the pipeline approach (ZoomCount − RSE pl , ZoomCount−RF DB pl ) on the same test images, where next image patches start using the CDC classifier once previous image patches are done using the classifier and now passing through the next modules ( 10. Some examples of the qualitative results. First and second rows show qualitative results of our method on the UCF-QNRF [22] and ShanghaiTech [49] datasets respectively. First three columns show good results, followed by two bad case images. Each result also shows the estimates of DenseNet [20] Direct Regression (DR) and the Density map method as a comparison.

Method
T total (secs) Tavg (secs) T smallest (secs) T largest (secs) MAE RMSE CSRNET [26] 60. regressor). We can see that this pipeline approach dramatically decreases the computational time further. This is useful in case of multiple test images. To the best of our knowledge, most of the crowd counting research papers do not provide time analysis, because the main objective for crowd counting in static images is to design a more effective model for counting with lesser error, and less consideration has been given towards the computational time. We believe that efficiency is also a major factor to compare and evaluate different crowd counting models.

VI. CONCLUSION
In this work, we have proposed a novel zoom-in and zoom-out based mechanism for effective and accurate crowd counting in highly diverse images. We propose to employ a decision module to detect the extreme high and low dense cases, where most state-of-the-art regression and density map based methods perform worse. The cluttered background regions are also discarded using the rigorous deep CNN 4-way classifier. Even without using any density maps, ZoomCount outperforms the state-of-the-art approaches on four benchmark datasets, thus proving the effectiveness of the proposed model.


Metaphor Detection using Deep Contextualized Word Embeddings

Metaphors are ubiquitous in natural language, and their detection plays an essential role in many natural language processing tasks, such as language understanding, sentiment analysis, etc. Most existing approaches for metaphor detection rely on complex, hand-crafted and fine-tuned feature pipelines, which greatly limit their applicability. In this work, we present an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs and multi-head attention mechanism to address the task of automatic metaphor detection. Our method, unlike many other existing approaches, requires only the raw text sequences as input features to detect the metaphoricity of a phrase. We compare the performance of our method against the existing baselines on two benchmark datasets, TroFi, and MOH-X respectively. Experimental evaluations confirm the effectiveness of our approach.

Introduction
A metaphor is a figurative form of expression that compares a word or a phrase to an object or an action to which it is not literally applicable but helps explain an idea or suggest a likeness or analogy between them. Metaphors have been used extensively in all types of literature and writings, especially in poetry and songs to communicate complex feelings, emotions, and visuals present in the text to readers effectively. Metaphors are ubiquitous in natural language and help in structuring our understanding of the world even without our conscious realization of its presence [1]. Given the prevalence and significance of metaphorical language, effective detection of metaphors plays an essential role in many natural language processing applications, for example, language understanding, information extraction, sentiment analysis, etc.
Email addresses: shashwata.co@nsit.net.in (Shashwat Aggarwal), rsingh@nic.in (Ramesh Singh) However, automated detection of metaphorical phrases is a difficult problem primarily due to three reasons. First, there is a subjective component involved: the metaphoricity of expression may vary across humans. Second, metaphors can be domain and context dependent. And third, there is a lack of annotated data, which is required to train supervised machine learning algorithms to facilitate automated detection accurately.
Most of the previous approaches for detection of metaphorical phrases, have either relied on manual and lexical detection [2,3,4] which requires heavily handcrafted features built from linguistic resources, that are costly to obtain and greatly limits their applicability or have used supervised machine learning based algorithms [5,6,7] with limited forms of linguistic context, for example using only the subject verb objects triplets (e.g. cat eat fish). Although these techniques automate the detection of metaphorical phrases, however, the prediction accuracies are not as good as the prediction accuracies of these techniques in other text classification tasks.
Inspired by recent works in the field of NLP and transfer learning, in this paper, we present an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs and multi-head attention mechanism to address some of the limitations aforementioned. Our method is notable in the sense that unlike many existing approaches, it requires only the raw text sequences as input and does not depend on any complex or fine-tuned feature pipelines.

Literature Survey
There has been significant work done in automatic detection and discovery of metaphorical phrases in natural language ranging from the traditional rule-based methods which rely on task-specific hand-coded lexical resources to the recent statistical machine learning models to identify metaphors.
One of the first attempts to detect metaphorical phrases automatically in the text was by [8]. Their system called met* could discriminate between literalness, metonymy, and metaphor in the underlying text. [9] proposed a sentence clustering approach for non-literal language recognition using a similarity-based word sense disambiguation method. However, their approach focused only on metaphors expressed by a verb. [10] trained a maximum entropy classifier to discriminate between literal and metaphorical usage of a phrase. Their method got a high accuracy of 95.12%. However, the majority of the verbs present in their dataset were used metaphorically, thus making the task notably easier. [3] used hyponymy (IS-A) relation in WordNet [11] and word bigram counts to predict metaphors in sentences. However, the word bigram counts lose a great deal of information over the verb-noun pairs. Also, they do not deal with literal sentences.
[12] used statistical learning methods to identify metaphors automatically. They start with a small seed of manually annotated metaphorical expressions. The system generates a large number of metaphors of similar syntactic structure from a corpus.
Other approaches using statistical learning methods to identify metaphors automatically include [4], that use concreteness and abstractness to detect metaphors, [13], that apply a Latent Dirichlet Allocation (LDA) based topic modeling method for automatic extraction of linguistic metaphors, [14] that construct a metaphor detection system using a random forest classifier with conceptual semantic features such as abstractness, imageability, and semantic supersenses, etc. Most of these existing approaches use a variety of features which rely on external lexical, syntactic, or semantic linguistic resources to train classification models for metaphor detection which severely limits their applicability.
Recently, there has been an introduction of several deep learning based approaches to tackle the problem of metaphor detection. Several methods such as [15,16,7] train word embeddings to identify and detect metaphors and have shown gains on various benchmarks. [17] used bidirectional LSTMs with raw text, SVO and dependency subsequences as input features. The 2018 VUA Metaphor Detection Shared Task has also introduced several LSTM and CRF based models [18,19,20,21] which are augmented by linguistic features such as POS tags, lemmas, verb clusters, unigrams, WordNet, etc.
Furthermore, there have been works such as [22,23] which have employed the recently proposed contextualized language representation models to detect metaphors. However, the work in this direction is limited. Also, a recently proposed popular multihead attention mechanism [24], which has been used extensively in domains such as speech processing or machine translation in the recent past has also not been tried for the task of metaphor detection yet.
Based on the lines of some of these recent works, in this paper, we combine the contextualized language representation models, ELMo [25] and BERT [26] with multihead attention mechanism and bidirectional LSTMs to detect metaphorical phrases automatically. We compare our proposed approach with existing baselines for the task of metaphor detection and classification.
The rest of the paper is organized as follows: In section 3, we discuss our proposed approach and in section 4 & 5, we describe the datasets we use and the experiments we perform on them.

Proposed Approach
We present an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs and multi-head attention mechanism to address the task of automatic metaphor detection and classification. Our method takes the raw text sequence as input and does not depend on any complex, hand-crafted or fine-tuned feature pipelines.

Contextualized Word Representations
Natural language processing (NLP) is a diversified field consisting of a wide variety of tasks such as text classification, named entity recognition, question answering, etc. However, most of these tasks contain limited datasets with a few thousand or a few hundred thousand human-labeled training examples. This shortage of training data severely affects the performance of modern deep learning-based NLP systems, which proffer benefits when trained from much larger amounts of data.
Recently, there have been techniques such as [25,26], that address this limitation by pretraining language representation models on enormous amount of unannotated text and then transferring the learned representation to a downstream model of interest as contextualized word embeddings. These contextualized word representations have resulted in substantial gains in accuracy improvements in numerous NLP tasks as compared to training on these tasks from scratch. Following recent work, we also use these contextualized word embeddings for our task of metaphor detection. Figure 1 shows the architecture of our proposed method for the task of metaphor classification. Each token t i of the input text is encoded by the contextualized word embeddings, ELMo or BERT, to obtain the corresponding contextualized word representations. The contextualized representations are feed as input to the bidirectional LSTM encoder network. The encoder network outputs an encoded representation across every timestep of the input sequence of text.

Proposed Model
We add a multi-head attention mechanism on top of the encoder network to obtain a context vector c representing the weighted sum of all the BiLSTM output states as depicted in equation 1, where h is the number of attention heads used, x i corresponds to the hidden representations from the encoder network, a j i is the attention weight computed for token t i from attention head j and c j is the context vector obtained from attention head j respectively.
Finally, the context vector c is feed to a dense layer decoder to predict the metaphoricity of the input phrase.

Datasets
We evaluate our method on two benchmark datasets, TroFi [9] and MOH-X [6] respectively. In Table 1, we report a summary of each of the dataset used in our study. The TroFi dataset consists of literal and nonliteral usage of 50 English verbs drawn from the 1987-89 Wall Street Journal (WSJ) Corpus. The MOH-X dataset, on the other hand, is an adaptation of the MOH dataset [27] which consists of simple and concise sentences derived from various news articles.

Implementation Details
We use 1024d ELMo and BERT vectors. The LSTM module has a 256d hidden state. We use 4 attention heads for the multi-head attention mechanism. We train the network using Adam optimizer with learning rate lr = 0.00003, β1 = 0.9, β2 = 0.999 and with a batch size of 16. The network is trained for 50 epochs.

Baselines
We compare the performance of our method with four existing baselines. The first baseline is a simple lexical baseline that classifies a phrase or token as metaphorical if that phrase or token is annotated metaphorically more frequently than as literally in the training set. The other baselines include a logistic regression classifier similar to one employed by [28], a neural similarity network with skip-gram word embeddings [7], and a BiLSTM based model combined with ELMo embeddings proposed in [22].

Evaluation Metrics
To evaluate the performance of our method with the baselines, we compute the precision, recall, F1 measure for the metaphor class and the overall accuracy. We perform 10 fold cross-validation following prior work.

Results
In Table 2, we show the performance of all the baselines and our proposed method on the task of metaphor detection and classification for two benchmark datasets (MOH-X and TroFi). Both of our proposed methods outperform the existing baselines on both the datasets using only the raw text as input features. The model with BERT performs better as compared to the model with ELMo embeddings on both the datasets. Contextualized word embeddings and multi-head attention mechanism improve the performance of both models by a large margin.
In addition to the performance comparison of our methods with the existing baselines, we also show the receiver operating characteristic curve in Figure 2 showing the AUC values for the metaphor detection task and illustrating the diagnostic ability of the best method (ours w/ BERT) as its discrimination threshold is varied.
Further, In Table 3 & 4, we report a set of examples from the TroFi and MOH-X developement sets along with the gold annotation label, the predicted label from both of our methods (with ELMo and with BERT) and the corresponding prediction scores assigned. The system confidently detects most of the metaporical and literal phrases such as "make the people's hearts glow.", "he pasted his opponent." and respectively.  However, the predicted output disagrees with the ground truth annotations for some phrases such as "the white house sits on pennsylvania avenue." and "bicycle suffered major damage." respectively. Majority of errors committed by our method are in detection of a particular type of metaphor called personification. The metaphoricity of phrases such as "the white house sits on pennsylvania avenue." are largely dependant on a few words like "white house" which if replaced by the name of some person changes the metaphoricity of the phrase. The network occasionally gets confused in discriminating between different entities like people, things, etc. Given additional information along with raw text as input to the network, for example the part-of-speech or named-entity tags, the performance of the network could be improved.
Finally, we report some example phrases such as "vesuvius erupts once in a while." or "the old man was sweeping the floor." for which the model with BERT embeddings correctly detects the metaphoricity of the phrase, while the model with ELMo fails to do so. Table 3: Examples from the TroFi development set, along with the gold label, predicted label, and the predicted score from our method with ELMo and BERT.

Input Phrase
Gold Predicted Score (for Metaphor Class) w/ ELMo w/ BERT w/ ElMo w/ BERT "make the people's hearts glow ."  Table 4: Examples from the MOH-X development set, along with the gold label, predicted label, and the predicted score from our method with ELMo and BERT.
Input Phrase Gold Predicted Score (for Metaphor Class) w/ ELMo w/ BERT w/ ElMo w/ BERT "this one speech could sink his candidacy."

Conclusion
In this work, we presented an end-to-end method composed of deep contextualized word embeddings, bidirectional LSTMs, and multi-head attention mechanism to address the task of automatic metaphor detection and classification. Our method requires only the raw text sequences as input and does not depend on any complex or fine-tuned feature pipelines. Our method established new state-of-the-art on both the datasets for metaphor detection.


s2fieldsofstudy:
Computer Science
Linguistics

text:
Alignment Data base for a Sign Language Concordancer

This article deals with elaborating a data base of alignments of parallel Franch-LSF segments. This data base is meant to be searched using a concordancer which we are also designing. We wish to equip Sign Language translators with tools similar to those used in text-to-text translation. To do so, we need language resources to feed them. Already existing Sign Language corpora can be found, but do not match our needs: working around a Sign Language concordancer, the corpus must be a parallel one and provide various examples of vocabulary and grammatical construction. We started with a parallel corpus of 40 short news and 120 SL videos , which we aligned manually by segments of various length. We described the methodology we used, how we define our segments and alignments. The last part concerns how we hope to allow the data base to keep growing in a near future.

Introduction
Sign Languages (SL) convey meaning with hands and body gestures, facial expressions, and eye gaze. Each of these is a part of the language grammar. SLs are languages in their own right, even though they still are considered minority languages with very low visibility, if not none. In 2008, the CRPD (Convention on the Rights of Persons with Disabilities) adopted by the United Nations emphasised the right of people with disabilities to fully access any type of information or communication. This has led to an increasing need for translated content in SL, whether for display in public spaces, public transportation, healthcare or broadcasting.
Sign Language interpretation and Sign Language translation are not to be confused. The classical way to distinguish the two is to say interpretation concerns oral languages while translation concerns written texts. Concerning SL, as they do not have an editable written form, the major difference is the possibility to post-edit the translation. Translators can rework their translations many times, until they reach what they believe to be the best account of the meaning in the target language. Interpreters however can only provide one version of their translation and once it is delivered, it cannot be edited (Filhol & Tannier, 2014).
The number of professional SL translators in France is very low despite the recent creation of a diploma. And the few working SL translators are not equipped with Computer Assisted Translation (CAT) software tools like those used by text-to-text translators, because none of them support SL. CAT software eases the translator's tasks (Koehn, 2009), by providing him with an integrated working environment along with dedicated tools (translation memories, concordancers, terminology management tools for example).
We are currently working on designing CAT software dedicated to SL. This paper presents the steps for the creation of a SL concordancer, which is an essential module of all text-to-text CAT software.

Sign Language concordancer
To learn more about the SL translation process, we conducted studies with professional SL translators (Kaczmarek & Filhol, 2019). We organised a brainstorming session and invited them to express their insights about their everyday practices and the obstacles they thought to often encounter. This session was followed by a free discussion between the participants, where they could discuss previously mentioned ideas and potential solutions to their needs. The scarcity of SL resources was often mentioned, as well as the need for a tool to enable reuse of prior work.
To complement those results with more objective observations, we also filmed translators at work. We set them up to work in pairs, which led them to discuss their current task. It allowed us to clearly identify their needs and the problems they encounter while working. The analysis showed that the translators could use the help of software tools such as NLP features, or automated encyclopedic searches. However, they have access to very few resources in SL, most of them not being easily searchable (content not referenced properly, dictionaries with limited queriable vocabulary), and also timeconsuming to consult (browsing through hour-long videos of SL content, searching for pictures to find ways to describe concepts with no standard signs).
Both subjectively and objectively, it appears that a bilingual concordancer with French and LSF would be a useful tool to the profession. But Sign Languages not having an editable written form, video is the most common way for signers to store messages. This simple statement already raises problems concerning the adaptation of CAT tools for SL, as videos are neither editable nor queriable, and also imply memory-consuming storage.
A turning point in text-to-text translation appearing with CAT software, is the translation memory (TM) feature (O'Hagan, 2009). It stores pairs of source and target segments, saved from the translators' work. Each of these pairs creates an alignment. If either segment is encountered later, or any text close enough to justify a match, the TM will suggest the counterpart segment in the stored pair, and the translator is free to accept, modify or reject the suggestion. The corrected segment will in turn create a new alignment added to the TM, which increases its accuracy. To this date, it is unavailable for SL. We wish to investigate what equivalent could be tested for SL.
Our suggestion here is to design a SL concordancer to keep the benefits from the TM. A concordancer is, regardless of the languages, a software tool allowing to search through corpora and to list the occurrences of a queried word. When it comes to translation, bilingual concordancers are used. The query is done in one of the languages of the pair, and results are provided with the aligned equivalent tagged in the other. Such tool allows translators to look up words or expressions in context. For example, given a list of matches for q query, similarity of the results shows the consistency of a translation in different contexts. On the contrary, a variety of results highlights the impact of the context on the translation.
Using a French-LSF parallel corpus, and since LSF cannot be queried easily in our case, we would query the written French part with text input, and provide examples in context of prior text-LSF translations. The alignments are pairs made of an identified segment of the text, and its translation in the corresponding video identified with time tags. Such alignments are stored as a data base which is built by the users themselves. Multiple segments may therefore overlap. When a query pulls up a match contained in more than one segment, the concordancer answers with the smallest in length (tightest annotation around the match), and the aligned video segment as always. If it has not been aligned, but still previously translated, the concordancer can answer by displaying the entire videos corresponding to the texts in which the query can be found, since the parallel content can be viewed as an alignment in itself, although often too big to be directly useful.
To allow the development and testing of a first prototype, we manually built a first data base of alignments. The next part presents the methodology used.

Data
Bilingual concordancers are based on parallel translated content, so to elaborate our data base, we need a corpus of translations. Written French and French Sign Language are our working languages for this study.
Concerning French Sign Language (LSF), a few corpora can be found. The Dicta-sign corpus (European project, 2010) contains on various types of discourse, but mostly dialogues and spontaneous discourses. Even if it is annotated, the annotations provided are of linguistic nature, and made of glosses which can hardly be considered translation in any case. It contains four different SLs. For the French part, larger chunks were also translated and aligned. An LSF interpreter watched the corpus, recorded himself interpreting the signs to French, then subtitled the video by typing his own interpretation.
This is not quality parallel data because it has been made in one-shot. As explained in the introduction, this is more an interpretation than a translation. We used the only usable French-LSF parallel corpus of forty short news texts, of three to five lines each in a journalistic style filmed using two cameras for a front and a side view for a total of 120 videos ("40-brèves" corpus, which you can find on the online OrtoLang platform 1 ).
Here is an example of a short text contained in French: Quelque 200 personnes pourraient avoir été tuées mercredi matin par un glissement de terrain provoqué par des pluies abondantes dans l'île indonésienne de Java, a annoncé à une source policière.
In English: Around 200 people may have been killed Wednesday morning by a landslide caused by heavy rains on the Indonesian island of Java, a police source said.
In this corpus, each text is translated by three different professional SL translators, and the resulting translations of an average 30-second duration. Each video already provides us with an alignment (2-3 lines texts can be aligned with 30 seconds videos).

Alignments
The goal here was to build smaller alignments than the total ones. In other words, we had to segment the texts into smaller segments and paired them with their translation, in variety of sizes. We had to sort our text segments and only keep those which were suitable to align in such way that the content of both the text and the video segment is the same in order to produce parallel data. We cannot for example, align an adverb in the text with only the facial expression of the signer because the video segment would include other, e.g. manual, features which would not be covered by the text segments. By definition, an alignment implies that both segments are equivalent in meaning (translations of one another). This can be a simple lexical match (sign with no added feature aligned with literal word equivalent) or longer clauses on either side.
The alignments had to be made manually because there is currently no way to do it automatically, unlike text-only alignments. To process our data, we chose to use Brat for 1 https://www.ortolang.fr/market/corpora/40-breves

Annotation of text segments
The aim was to create an important variety of segments to align them and populate the data base. As we had three different signers, we were free to segment the texts in three different ways, and vary the spans for a given expression. The segments were chosen based on their lack of standard signs, or on variety of translations proposed by the signers: figures of speech, idiomatic expressions or grammatical phenomena. But we also included random segments provided we could align them with a counterpart segment.
We used Brat for this task, because it creates for each selected segment a unique key, which upon export, is characterised by the indexes of its first and its last characters in the source text (considering plain text, with no formatting character). Those will be used in the processing of the videos below.

3.2.2
Annotation of video segments Elan is an annotation software which allows to create tracks and tags to annotate a video. We only needed one track to report the keys. For each text segment, the first step was to watch the corresponding videos to find their translations. Once found, we create an annotation on our track, covering the duration of the translation and label it with the segment's unique key taken from our Brat data. The exported data from Elan therefore consisted in lines displaying for each unique identification key the corresponding time tags on the video, as shown on the figure below.

3.2.3
Creating the alignments Now that we have identified both the text segment and its LSF translation in the corresponding video, we bring them together to create the alignment itself. Each alignment is stored in the database in the following format: <TxtID, start pos., length, VidID, start time, duration> Where: • TxtID is the identification code of the text, ranging from 1A to 1T and from 2A to 2T. This code allows to retrieve them in their own storage space. • Start pos. is the position of the first character from the text segment, in the text. • Length is the number of characters of the segment. • VidID is the unique identification of the video in their own storage space, and allow to retrieve them. • Start time is the time tag corresponding to the beginning of the segment in the video. • Duration is the duration in seconds of the segment's translation.

Metadata files
Apart from the alignment file itself, we keep more information about the texts and videos used in metadata files. The aim is to keep trace of complementary information which could be useful to display in the concordancer, or apply filters to its results. The metadata contains information such as titles, authors or sources and topics.

Implementation and open tests
Our current database is made of the 120 alignments from the 40-brèves corpus, plus 343 manual alignments of finer grain in the same texts and videos, i.e. a total of 463 alignments. As we later explain, we will keep making this number grow.
We implemented the concordancer to test this data base, following the design details, to be published elsewhere (Kaczmarek & Filhol, 2020). A screenshot of a result page, which appears after a text query, is given in figure 6. The query is highlighted in yellow in the text and the user is presented with its context. Similarly, the video player shows the matched segment in yellow on the time bar, the user being free to adjust right and left context duration (buttons A and B in the figure), or even watch the entire video. We provided public access to this interface to allow professionals and academics to test it. It is available at https://platform.postlab.fr, using login "demo_limsi" and password "pass_demo_du_limsi".
We invited translators to engage in testing for this project.
We are collecting their feedback, in an iterative process to converge on the most adequate kind of tool for them to use in their everyday practices. Any reader interested in contributing to this feedback is welcome to e-mail us.

Future work and conclusion
We based ourselves on a written French/LSF parallel corpus to build an alignment data base. This data base can be searched using the concordancer we designed. With fewer than 500 entries, it is small for a concordancer resource. We will keep expanding it, but wish to open contribution to professionals and the academic audience as well, by making our concordancer a collaborative platform.
To allow this, our next planned step is the addition of a feature allowing users to add new content. We are currently developing this annotation function, whereby users select a text and a video in our library (one being the translation of at least a part of the other), select text and a video segment, and saving the alignment.
This being the first works concerning CAT & SL, and also concerning the elaboration of such data base, we hope to draw interest on the topic. SL are still seen as minority languages. Assisting translation would result in more visibility and by extension, maybe more consideration and recognition.
As an almost undocumented language, LSF resources are rare and those which can be found are oriented towards the research community and not the speakers -or signers -themselves. The corpus which we used to build our database can benefit professionals, the research community, and other domains, such a SL linguistics, SL teaching and interpretation training.