## Dolphin: A Spoken Language Proficiency Assessment System for Elementary Education

The Dolphin Team ∗† TAL Education Group, Beijing, China

## ABSTRACT

## 1 INTRODUCTION

Spoken language proficiency is critically important for children's growth and personal development. Due to the limited and imbalanced educational resources in China, elementary students barely have chances to improve their oral language skills in classes. Verbal fluency tasks (VFTs) were invented to let the students practice their spoken language proficiency after school. VFTs are simple but concrete math related questions that ask students to not only report answers but speak out the entire thinking process. In spite of the great success of VFTs, they bring a heavy grading burden to elementary teachers. To alleviate this problem, we develop Dolphin, a spoken language proficiency assessment system for Chinese elementary education. Dolphin is able to automatically evaluate both phonological fluency and semantic relevance of students' VFT answers. We conduct a wide range of offline and online experiments to demonstrate the effectiveness of Dolphin. In our offline experiments, we show that Dolphin improves both phonological fluency and semantic relevance evaluation performance when compared to state-of-the-art baselines on real-world educational data sets. In our online A/B experiments, we test Dolphin with 183 teachers from 2 major cities (Hangzhou and Xi'an) in China for 10 weeks and the results show that VFT assignments grading coverage is improved by 22%.

## CCS CONCEPTS

· Applied computing → Interactive learning environments ; Computer-assisted instruction ; E-learning ; · Computing methodologies → Discourse, dialogue and pragmatics ;

## KEYWORDS

Spoken language proficiency, Assessment, Verbal fluency, Disfluency detection, K-12 education

## ACMReference Format:

The Dolphin Team. 2020. Dolphin: A Spoken Language Proficiency Assessment System for Elementary Education. In Proceedings of The Web Conference 2020 (WWW '20), April 20-24, 2020, Taipei, Taiwan. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3366423.3380018

∗ Authors: Wenbiao Ding, Guowei Xu, Tianqiao Liu, Weiping Fu, Yujia Song, Chaoyou Guo, Cong Kong, Songfan Yang, Gale Yan Huang, Zitao Liu. All authors contributed equally to this research.

† Corresponding author: Zitao Liu. Email: liuzitao@100tal.com.

This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

WWW'20, April 20-24, 2020, Taipei, Taiwan

© 2020 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC-BY 4.0 License.

ACM ISBN 978-1-4503-7023-3/20/04.

https://doi.org/10.1145/3366423.3380018

Spoken language proficiency and verbal fluency usually reflect cognitive flexibility and functions of the brain which are very crucial for personal development [4, 6]. However, developing such skills is very difficult. It requires a large number of oral practices. Studies have shown that it's better to improve and practice verbal fluency during school ages (in elementary school) compared to adult period [2, 10, 25]. In China, similar to many developing countries, due to the incredibly imbalanced ratio of teachers and elementary students, students barely have chances to speak during a 45-minute class, let alone to practice their verbal fluency abilities. This leads to the fact that elementary school children usually fail to reach satisfactory levels of spoken language proficiency [5, 8, 18, 21, 23]. In order to improve the Chinese elementary students' verbal abilities, we create verbal fluency tasks (VFTs) to help students practice their spoken language proficiency. Figure 1 illustrates one example of our VFTs.

Sure, Wei. Every time, I climb 4 steps first and then go downstairs 2 steps. Do you know how many times I can finish the entire stairs?

Figure 1: One concrete example of our VFTs. It is originally in Chinese and is translated in English for illustration purpose.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00358v3-figure-9cc94294-0280-4d88-bc36-b45a5cf3a583.jpg)

We create VFTs with concrete real-life scenarios. In VFTs we encourage students to 'think aloud": speak out their entire thinking process and not just focus on the answer itself. Teachers first use a mobile App to select VFTs from a VFT bank and assign them to students. Parents will take video clips when students are 'thinking aloud" and the video clips will be automatically collected and graded. The disfluent score not only depends on the occurrence of interregnum words, like uh I mean , , but also considers long-time silence, out-of-logic sentences, etc. The entire educational design of VFTs is highly recognized by students, parents and educational scholars. However, it brings heavy weekly workloads to teachers when grading every student's VFT clips. In China, an elementary class has 50 students and a teacher holds 4 classes on average. The average length of VFT video clips is about 2 minutes. From our

survey, grading VFTs itself brings 10 hours extra workloads to elementary teachers.

In order to leverage the effectiveness of VFTs and reduce the grading workloads of our elementary teachers, we develop Dolphin, a spoken language proficiency assessment system that is able to automatically evaluate VFTs from both phonological and semantic perspectives. In terms of evaluating phonological fluency, we design a novel deep neural network to learn an effective representation of each video clip and then use logistic regression to predict the phonological fluency score. For detecting semantic relevance, we design a Transformer based multi-attention network to capture semantic relevance between clips and VFTs. We evaluate Dolphin in both offline and online settings. In our offline experiments, we demonstrate the effectiveness of our phonological and semantic relevance scorers by using our real-world educational data sets. In the online A/B experiments, we enable Dolphin for 183 elementary teachers from Hangzhou and Xi'an to help them automatically grade their VFT clips. Various business metrics such as VFT clips grading coverage go up significantly as well as ratings from the teacher satisfaction survey.

## 2 RELATED WORK

Disfluency detection is a well studied NLP task that is informally defined as identifying the interruptions in the normal flow of speech [29]. In general, disfluency detection methods can be divided into three categories: (1) parsing based approaches [12, 24]; (2) sequence tagging based approaches [9, 13, 19, 22, 28, 35, 36]; and (3) noisy channel model [20].

Parsing based disfluency detection methods leverage language parsers to identify the syntactic structure of each sentence and use the syntactic structure to infer the disfluency [12, 24]. Sequence tagging based approaches find fluent or disfluent words within each sentence by modeling word sequences directly and a wide range of models are proposed, such as conditional random fields [9, 22, 35], hidden Markov models [19, 28], recurrent neural networks [13, 36], etc. Noisy channel models assume that each disfluent sentence is generated from a well-formed source sentence with some additive noisy words [20]. They aim to find the n -best candidate disfluency analyses for each sentence and eliminate noisy words by using the probability from language models.

However, several drawbacks prevent existing approaches from evaluating our real-world VFTs. First, all above methods require clean and massive labeled corpus for training, which is not realistic in our case. In our scenario, the text is generated from students' VFT video clips by using automatic speech recognition (ASR) service. It may contain mistakes from ASR service and can easily fail existing approaches. Furthermore, parsing and sequence tagging based approaches usually need a large amount of annotated data to cover different cases. This would be very expensive to obtain, especially in the specific domain of elementary education.

## 3 THE DOLPHIN ARCHITECTURE

## 3.1 The Workflow Overview

The overall workflow of Dolphin is illustrated in Figure 2. The entire workflow is made up of two sub-workflows. In the top subworkflow, when a VFT clip arrives, we extract its audio track. The audio track is then sent to an open-sourced prosodic feature extractor 1 and our self-developed automatic speech recognizer (ASR) 2 . We extract both speech-related features in the prosodic feature extraction component and linguistic features from ASR output. Then, we combine both prosodic and linguistic features and utilize a deep neural network (DNN) to learn an effective representation for each clip and predict the phonological fluency by a regularized logistic regression. In the bottom sub-workflow, we first obtain pre-trained language embeddings of both VFT question and ASR transcriptions. After that, we learn a multi-attention network based on transformers to compute semantic relevance scores. Finally we use a combination heuristic to derive verbal fluency scores from phonological fluency and semantic relevance results. We will discuss more details of our phonological fluency scorer and semantic relevance scorer in Section 3.2 and Section 3.3. The combination heuristic is designed by our teaching professionals based on the company's business logic and we omit its details in this paper.

## 3.2 Phonological Fluency Scorer

3.2.1 Raw Features. In order to capture the phonological fluency of each video clip, we extract raw features from both text and audio, which can be summarized into four categories:

- · Word level features, which contain features such as statistics of part-of-speech tags, number of consecutive duplicated words, number of interregnum words 3 , etc.
- · Sentence level features, which contain different statistics from sentence perspective, such as distribution of clip voice length of each sentence, number of characters in each sentence, voice speed of each sentence, etc.
- · Instance (clip) level features, which contain features like total number of characters and sentences, number of long silence that is longer than 5, 3, or 1 seconds, the proportion of effective talking time to clip duration, etc.
- · Prosodic features, which contain speech-related features such as signal energy, loudness, Mel-frequency cepstral coefficients (MFCC), etc.

Please note that the ASR not only generates the text transcriptions but the start and end timestamps for each sentence. Such information is very useful for computing features such as voice speed, silence duration percentage, etc.

3.2.2 Grouping Based Neural Architecture. Even though we collect various raw features from many dimensions (discussed in Section 3.2.1), they tend to be very sparse and directly feeding them into existing predictive models yields poor performance. Similar to [3, 11, 16, 34], we utilize a deep neural network to conduct non-linear feature transformation. However, labeled examples are very limited in the real-world scenarios. On the one hand, labeling VFT video clips requires professional teaching knowledge. Thus, the available labeling resource is very limited. On the other hand, labeling one student answer requires the labeler to watch the entire 2-minute

1 https://www.audeering.com/opensmile/

2 Due to the fact that speech from children are much more different from adults' speech, we trained our own ASR model on a 400-hour children speech recognition data set. The ASR model's word error rate is 86.2%.

3 Interregnum word is an optional part of a disfluent structure that consists of a filled pause uh or a discourse marker I mean .

Figure 2: The Dolphin workflow. The top/bottom two gray boxes represent the two key components: phonological fluency scorer (Section 3.2) and semantic relevance scorer (Section 3.3).

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00358v3-figure-3d8d7780-e627-44eb-bf44-60530157b4f8.jpg)

clip, which is much more time-consuming compared to standard NLP labeling tasks. With very limited annotated data, many deep representation models may easily run into overfitting problems and become inapplicable.

(

+

+

)

To address this issue, instead of directly training discriminative representation models from a small amount of annotated labels, we develop a grouping based deep architecture to re-assemble and transform limited labeled examples into many training groups. We include both positive and negative examples into each group. Within each group, we maximize the conditional likelihood of one positive example given another positive example and at the same time, we minimize the conditional likelihood of one positive example given several negative examples. Different from traditional metric learning approaches that focus on learning distance between pairs, our approach aims to generate a more difficult scenario that considers distances between not only positive examples but negative examples.

More specifically, let (·) + and (·) -be the indicator of positive and negative examples. Let D + = { x + i } n i = 1 and D -= { x - } l m l = 1 be the collection of positive and negative example sets, where n and m represent the number of total positive and negative examples in D + and D -. In our grouping based DNN, for each positive example x + i , we select another positive example x + j from D + , where x + i , x + j . Then, we randomly select k negative examples from D -, i.e., x -1 , · · · , x -k . After that, we create a group g s by combining the positive pair and the k negative examples, i.e., g s = &lt; x + i , x + j , x -1 , · · · , x -k &gt; . By using the grouping strategy, we can create O n m ( 2 k ) groups for training theoretically. Let G be the entire collection of groups, i.e., G = { g 1 , g 2 , · · · , g K } where K is the total number of groups.

After the grouping procedure, we treat each group g s as a training sample and feed g s s into a standard DNN for learning robust representations. The inputs to the DNN are raw features extracted from Section 3.2.1 and the output of the DNN is a low-dimensional representation feature vector. Inside the DNN, we use the multilayer fully-connected non-linear projections to learn compact representations as shown in Figure 3.

ModelLearning Inspired by the discriminative training approaches in information retrieval, we propose a supervised training approach to learn model parameters by maximizing the conditional likelihood of retrieving positive example x + j given positive example x + i from group g s . More formally, let f ∗ be the learned representation of x ∗ from DNN, where x ∗ ∈ g s . Similarly, f + i and f -l represent the embeddings of the positive example x + i and negative example x -l .

Figure 3: The overview of our grouping based deep architecture.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00358v3-figure-897dd86c-6881-40df-bbc2-1341a5b90cb4.jpg)

Then, the relevance score in the representation space within a group is then measured as cosine ( f + i , f ∗ ) , i.e., r ( x + i , x ∗ ) def = cosine ( f + i , f ∗ ) . In our representation learning framework, we compute the posterior probability of x + j in group g s given x + i from the cosine relevance score between them through a softmax function

$$p ( x _ { j } ^ { + } | x _ { i } ^ { + } ) = \frac { \exp \left ( \eta \cdot r ( x _ { i } ^ { + }, x _ { j } ^ { + } ) \right ) } { \sum _ { x _ { * } \in g _ { s }, x _ { * } \ne x _ { i } ^ { + } } \exp \left ( \eta \cdot r ( x _ { i } ^ { + }, x _ { * } ) \right ) }$$

where η is a smoothing hyper parameter in the softmax function, which is set empirically on a held-out data set in our experiment.

Hence, given a collection of groups G , we optimized the DNN model parameters by maximizing the sum of log conditional likelihood of finding a positive example x + j given the paired positive example x + i from group g s , i.e., L( Ω ) = -˝ log p ( x + j | x + i ) , where Ω is the parameter set of the DNN. Since L( Ω ) is differentiable with respect to Ω , we use gradient based optimization approaches to train the DNN.

## 3.3 Semantic Relevance Scorer

Semantic analysis of video clips is also very important for our spoken language proficiency assessment. After a detailed study on teachers' manually grading cases, we observe that some naughty students fool our phonological fluency scorer by talking about completely irrelevant topics, such as answering different questions, casual life recording, or even playing video games. Therefore, we

Figure 4: The overview of our model. We use colors to better represent intermediate manipulations of hidden states or embeddings.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00358v3-figure-b30ae13f-ba8a-4217-8276-09fe24364c46.jpg)

develop an end-to-end predictive framework to evaluate the semantic relationship between the students' video clips and the VFT questions.

Our framework is based on the recently developed Transformer architecture [32]. It dispenses with any recurrence and convolutions entirely and is based solely on attention mechanisms. Let P = { e p i } n i = 1 be the word embeddings of ASR transcriptions from student's video clip and Q = { e q j } n j = 1 be the word embeddings of the assigned VFT question 4 . e p i and e q j are the k -dimensional pre-trained word embeddings. Without the loss of generality, we convert sentences in both P and Q to n words. The proposed framework is shown in Figure 4 and it is built upon four crucial layers, which are described in the following sections.

3.3.1 Self Semantics Extraction Layer. As shown in Figure 4, we adopt a Transformer encoder to encode the student answer P and the VFT question Q as their contextual representations, i.e., { t p i } n i = 1 = Transformer (P) and { t q j } n j = 1 = Transformer (Q) , respectively. The Transformer encoder consists of multiple identical blocks which contain multi-head attention sub-layers and position-wise fully connected feed-forward sub-layers [32]. Through the multihead self-attention mechanism, the model captures the long-range dependencies among the words in the original texts and re-encodes each word as a combination of all the words related to it. After that, we get the contextual embeddings of answers, i.e., { t p i } n i = 1 , and VFT questions, i.e., { t q j } n j = 1 .

extract the semantic relation between the student answers and the VFT questions. Specifically, we attend contextual embeddings of student answers to VFT questions and attend VFT questions to student answers as well. We use additive ( Add ), subtractive ( Sub ), multiplicative ( Mul ), dot-product ( Dot ) functions as our cross-attention functions to compute four types of attentions scores that are used to obtain the corresponding weighted-sum representations. Let ⊙ be the operation of element-wise dot product. Let s (∗) i j , a (∗) i j , p , a (∗) i j , q be the raw and normalized attention scores from different attention functions and let h (∗) i , p h (∗) j , q be the representations weighted by different types of attention scores where ∗ indicates different attention functions and it can be Add, Sub, or Dot. The detailed descriptions of different attention functions and representations are shown in Table 1. In Table 1, v ( Add ) ∈ R k , v ( Sub ) ∈ R k , v ( Dot ) ∈ R k , W ( Mul ) ∈ R k × k , W ( Dot ) ∈ R k × k , W ( Add ) p ∈ R k × k , W ( Sub ) p ∈ R k × k and W ( Add ) q ∈ R k × k , W ( Sub ) q ∈ R k × k are learned parameters in our framework. softmax p and softmax q are softmax functions of normalizing attention scores for both student answers and VFT questions. The final representations of student answers, i.e., { c p i } n i = 1 and VFT questions, i.e., { c q j } n j = 1 are computed by a linear transformation on the concatenation of four different attention-weighted embeddings, i.e.,

3.3.2 Interactive Semantics Extraction Layer. Following previous work [26, 31, 33], we use a cross multiway attention mechanism to

4 Chinese Word Embedding from Tencent AI Lab is used in the following experiments, see https://ai.tencent.com/ailab/nlp/embedding.html.

$$c _ { i } ^ { p } & = W _ { p } \tilde { h } _ { i } ^ { p } = W _ { p } [ h _ { i, p } ^ { ( \text{Add} ) } \colon h _ { i, p } ^ { ( \text{Sub} ) } \colon h _ { i, p } ^ { ( \text{Mul} ) } \colon h _ { i, p } ^ { ( \text{Dot} ) } ] \\ c _ { j } ^ { q } & = W _ { q } \tilde { h } _ { j } ^ { q } = W _ { q } [ h _ { j, q } ^ { ( \text{Add} ) } \colon h _ { j, q } ^ { ( \text{Sub} ) } \colon h _ { j, q } ^ { ( \text{Mul} ) } \colon h _ { j, q } ^ { ( \text{Dot} ) } ]$$

where [ a : b ] denotes the concatenation of vector a and b and W p ∈ R k × 4 k and W q ∈ R k × 4 k are the linear projection matrices.

Dolphin: A Spoken Language Proficiency Assessment System for Elementary Education WWW'20, April 20-24, 2020, Taipei, Taiwan Table 1: The descriptions of different attention functions. Add, Sub, Mul and Dot represent the additive, subtractive, multiplicative and dotproduct attention functions.

|     | Attentive Function                                                         | Attention Score (Answer)                        | Answer Embedding                                    | Attention Score (Question)                      | Question Embedding                                  |
|-----|----------------------------------------------------------------------------|-------------------------------------------------|-----------------------------------------------------|-------------------------------------------------|-----------------------------------------------------|
| Add | s ( Add ) i j = v ( Add ) ⊤ tanh ( W ( Add ) p t p i + W ( Add ) q t q j ) | a ( Add ) i j , p = softmax p ( s ( Add ) i j ) | h ( Add ) i , p = ˝ n j = 1 a ( Add ) i j , p t q j | a ( Add ) i j , q = softmax q ( s ( Add ) i j ) | h ( Add ) j , q = ˝ n i = 1 a ( Add ) i j , q t p i |
| Sub | s ( Sub ) i j = v ( Sub ) ⊤ tanh ( W ( Sub ) p t p i - W ( Sub ) q t q j ) | a ( Sub ) i j , p = softmax p ( s ( Sub ) i j ) | h ( Sub ) i , p = ˝ n j = 1 a ( Sub ) i j , p t q j | a ( Sub ) i j , q = softmax q ( s ( Sub ) i j ) | h ( Sub ) j , q = ˝ n i = 1 a ( Sub ) i j , q t p i |
| Mul | s ( Mul ) i j = t p i ⊤ W ( Mul ) t q j                                    | a ( Mul ) i j , p = softmax p ( s ( Mul ) i j ) | h ( Mul ) i , p = ˝ n j = 1 a ( Mul ) i j , p t q j | a ( Mul ) i j , q = softmax q ( s ( Mul ) i j ) | h ( Mul ) j , q = ˝ n i = 1 a ( Mul ) i j , q t p i |
| Dot | s ( Dot ) i j = v ( Dot ) ⊤ tanh ( W ( Dot ) ( t p i ⊙ t q j ))            | a ( Dot ) i j , p = softmax p ( s ( Dot ) i j ) | h ( Dot ) i , p = ˝ n j = 1 a ( Dot ) i j , p t q j | a ( Dot ) i j , q = softmax q ( s ( Dot ) i j ) | h ( Dot ) j , q = ˝ n i = 1 a ( Dot ) i j , q t p i |

Table 2: Experimental results on phonological relevance detection.

3.3.3 Integration Layer. We first stack { c p i } n i = 1 and { c q j } n j = 1 into matrix form C p and C q , i.e., C p ∈ R k × n , C p = [ c p 1 ; c p 2 ; · · · ; c p n ] and C q ∈ R k × n , C q = [ c q 1 ; c q 2 ; · · · ; c q n ] and then convert the aggregated representation C p and C q of all positions into a fixed-length vector with structured self-attention pooling [17], which is defined as

$$\tilde { p } & = C _ { p } \text{softmax} ( \mathbf w _ { p 1 } ^ { \top } \tanh ( \mathbf W _ { p 2 } C _ { p } ) ) \\ \tilde { q } & = C _ { q } \text{softmax} ( \mathbf w _ { q 1 } ^ { \top } \tanh ( \mathbf W _ { q 2 } C _ { q } ) )$$

$$\varphi _ { q ^ { 1 } }$$

where w p 1 ∈ R k , w q 1 ∈ R k , W p 2 ∈ R k × k and W q 2 ∈ R k × k are the learned parameters.

3.3.4 Prediction Layer. Similar to [7], we use three matching methods to extract relations between the student answer and the reference answer: (1) concatenation of the two representations (˜, ˜); (2) p q element-wise product ˜ p ∗ ˜ ; q and (3) absolute element-wise difference |˜ p -˜ |, q i.e., x = [ ˜ p : ˜ q : | ˜ p -˜ q | : ˜ p · ˜ q ] . The resulting vector x , which captures information from both the student answer and the reference answer, is fed into a multilayer perceptron classifier to output the probability (score) of the semantic analysis tasks. The objective function is to minimize the cross entropy of the relevance labels.

## 4 EXPERIMENTS

## 4.1 Phonological Fluency Experiment

In this experiment, we collect 880 VFT video clips from students in 2nd grade. In each clip, a student talks about his or her entire thinking process of solving a math question. Our task is to score the fluency of the student's entire speech on a scale between 0 and 1. To obtain a robust annotated label, each video clip is annotated as either fluent or disfluent by 11 teaching professionals and we use the majority voted labels as our ground truth. The proportion of positive (fluent) examples is 0.60, the Kappa coefficient is 0.65 in this data set.

We select several recent work on embedding learning from limited data as our baselines. More specifically, we have

- · Logistic regression, i.e., LR . We train an LR classifier on raw features (discussed in Section 3.2.1);
- · Siamese networks, i.e., SiameseNet [15]. We train a Siamese network that takes a pair of examples and trains the embeddings so that the distance between them is minimized if they're from the same class and is greater than some margin value if they represent different classes;
- · Triplet networks, i.e., TripleNet [27]. We train a triplet network that takes an anchor, a positive (of same class as an

|          |    LR |   SiameseNet |   TripleNet |   RelationNet |   Our |
|----------|-------|--------------|-------------|---------------|-------|
| Accuracy | 0.82  |        0.802 |       0.847 |         0.843 | 0.871 |
| F1 score | 0.873 |        0.859 |       0.889 |         0.89  | 0.901 |

anchor) and negative (of different class than an anchor) examples. The objective is to learn embeddings such that the anchor is closer to the positive example than it is to the negative example by some margin value;

- · Relation network for few-shot learning, i.e., RelationNet [30]. The RelationNet learns to learn a deep distance metric to compare a small number of images within episodes.

Following the tradition to assess representation learning algorithms [3], we evaluate the classification performance via accuracy and F1 score. We choose logistic regression as the basic classifier. For each task, we conduct a 5-fold cross validation on the data sets and report the average performance in Table 2. We conduct pair wise t-tests between our approach and every baseline mentioned above and the results show that the prediction outcomes are statistically significant at 0.05 level. As we can see from Table 2, our grouping based approach demonstrates the best performance. This is due to the fact that our discriminative embedding learning directly optimizes the distance between positive and negative examples. By generating all the groups, we want to separate each positive pair with a few negative examples as far as possible. In terms of TripleNet and RelationNet, both of them optimize the distance function between positive and negative indirectly. For example, TripletNet generates more training samples by creating triplet, which contains anchor, positive and negative examples. It fully utilizes the anchor to connect the positive and negative examples during the training. However, selecting triplet is not straightforward. SiameseNet has the worst performance because it is designed for one-shot learning, which aims to generalize the predictive power of the network not just to new data, but to entirely new classes from unknown distributions. This may lead to the inferior performance in our situation that both positive and negative examples are very limited.

## 4.2 Semantic Relevance Experiment

Wecollect 120,000 pairs of student's video clip and VFT question for training and evaluating the semantic relevance scorer. It contains 90,000 positive and 30,000 negative pairs, which are annotated by professional annotators. The positive pairs indicate that the video clip and VFT question are semantically relevant. We randomly select 30,000 samples as our test data and use the rest for validation and training.

We compare our model with several state-of-the-art baselines of sentence similarity and classification to demonstrate the performance of our multi-attention network. More specifically, we choose

- · Logistic regression, i.e., LR . Similar to [1], we compute embeddings of Q and P by averaging each word's embedding from a pre-trained look-up table. Then we concatenate the two embedding of P and Q as the input for logistic regression;
- · Gradient boosted decision tree, i.e., GBDT . We use the same input as LR but train the GBDT model instead;
- · Multichannel convolutional neural networks, i.e., TextCNN . Similar to [14], we train embedding by using CNN on top of pre-trained word vectors;
- · Sentence encoding by transformer , i.e., Transformer . As discussed in [32], we use transformer as the encoding module and apply a feed-forward network for classification.

Similar to experiments in Section 4.1, we report the classification accuracy and F1 score in Table 3. We conduct pair wise t-tests between our approach and every baseline mentioned above and the results show that the prediction outcomes are statistically significant at 0.05 level. As we can see, our Transformer based multiway attention network shows superior performance among all baselines. It not only aggregates sentence information within Transformer encoder layer, but matches words in both video clips and VFT questions from multiway attention layer. We aggregate all available semantic information by an inside aggregation layer. Due to the lack of any contextual information, LR have the worst classification performance. TextCNN and Transformer improve the accuracy by considering the sentence level sequential information. However, it ignores the semantic relationship between video clips and VFT questions.

Table 3: Experimental results on semantic relevance detection.

|          |    LR |   GBDT |   TextCNN |   Transformer |   Our |
|----------|-------|--------|-----------|---------------|-------|
| Accuracy | 0.83  |  0.863 |     0.877 |         0.881 | 0.888 |
| F1 score | 0.838 |  0.872 |     0.888 |         0.89  | 0.896 |

## 4.3 Online A/B Experiments

In order to fully demonstrate the value of the Dolphin system, we also conduct the online A/B experiments. In our Dolphin system, automatic grades are given to teachers and teachers have the option to switch to manual grading model. We enable Dolphin for 183 elementary teachers from Hangzhou and Xi'an from 2018.09.05 to 2018.11.19. The total number of weekly graded VFT assignments is around 80,000. In summary, 92% of teachers enable the Dolphin automatic grading functionality. Among these enabled subjects, 87% of the automatic gradings of VFT assignments are accepted. Here, we would like to mention that there are 13% teachers choose to grade manually doesn't mean our grading error rate is 13%. Furthermore, we compare with teachers that have no access to Dolphin in terms of the overall grading coverage. The results show that Dolphin improves VFT assignments grading coverage 5 by 22%.

5 Coverage is calculated as number of graded assignments divided by total number of assignments.

We also conduct a questionnaire-based survey about teachers' Dolphin usage experience specifically. The teachers participated in our A/B experiments are asked to give a satisfaction rating from 1 to 5 (5 = Very Satisfied and 1 = Very Dissatisfied). There are 183 teachers participating this survey (85 from Hangzhou and 98 from Xi'an). The satisfaction rating distributions are listed in Figure 5. As we can see from Figure 5, teachers are pretty positive on this automatic VFT grading tool and the overall dissatisfied ratio (sum of rating 2 and rating 1) is very low (Hangzhou: 0% and Xi'an: 1.4%). Thevery satisfied percentage in Xi'an is much lower than Hangzhou. This is because the VFT assignments were recently introduced to schools in Xi'an and teachers are very serious and conservative to both the VFT assignments and the automatic grading tool.

Figure 5: Satisfaction rating distributions from teachers participated in our A/B experiments.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00358v3-figure-490df980-7355-45c4-94d7-38bef72f532a.jpg)

## 4.4 Discussion of Ethical Issues

Automatic grading tools are double-edged swords. On the one hand, They greatly reduce the burdens of tedious and repetitive grading workloads for teachers and let them have enough time to prepare class materials and teaching content. On the other hand, it could mislead students and sweep students' problems under the carpet. As a rule of thumb, only use them for skill practice instead of entrance examinations. Moreover, we design policies and heuristics to mitigate the misleading risks. First, we ask both the teachers and parents to randomly check one result in every 5 VFT assignments. They are able to report the wrong grading immediately from the mobile application. Second, when the grading tool gives 3 consecutive low ratings to a child, an alarm will be fired to the corresponding teacher and the teacher is required to check all recent ratings.

## 5 CONCLUSION

In this paper we present Dolphin, a spoken language proficiency assessment system for elementary students. Dolphin provides more opportunities for Chinese elementary students to practice and improve their oral language skills and at the same time reduces teachers' grading burden. Experiment results in both offline and online environments demonstrate the effectiveness of Dolphin in terms of model accuracy, system usage, users (teacher) satisfaction rating and many other metrics.

## REFERENCES

- [1] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-tobeat baseline for sentence embeddings. In Proceedings of the 5th International Conference on Learning Representations .
- [2] Ida Sue Baron, Kristine Erickson, Margot D Ahronovich, Kelly Coulehan, Robin Baker, and Fern R Litman. 2009. Visuospatial and verbal fluency relative deficits in 'complicated' late-preterm preschool children. Early Human Development 85, 12 (2009), 751-754.
- [3] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (2013), 1798-1828.
- [4] Virginia W Berninger and Frances Fuller. 1992. Gender differences in orthographic, verbal, and compositional fluency: Implications for assessing writing disabilities in primary grade children. Journal of School Psychology 30, 4 (1992), 363-382.
- [5] Jiahao Chen, Hang Li, Wenxin Wang, Wenbiao Ding, Gale Yan Huang, and Zitao Liu. 2019. A Multimodal Alerting System for Online Class Quality Assurance. In International Conference on Artificial Intelligence in Education . Springer, 381-385.
- [6] Morris J Cohen, Allison M Morgan, Melanie Vaughn, Cynthia A Riccio, and Josh Hall. 1999. Verbal fluency in children: Developmental issues and differential validity in distinguishing children with attention-deficit hyperactivity disorder and two subtypes of dyslexia. Archives of Clinical Neuropsychology 14, 5 (1999), 433-443.
- [7] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 670-680.
- [8] Sidney K D'Mello, Andrew M Olney, Nathan Blanchard, Borhan Samei, Xiaoyi Sun, Brooke Ward, and Sean Kelly. 2015. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. In Proceedings of the 2015 ACM on international conference on multimodal interaction . ACM, 557-566.
- [9] James Ferguson, Greg Durrett, and Dan Klein. 2015. Disfluency detection with a semi-markov model and prosodic features. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies . 257-262.
- [10] William D Gaillard, Lucie Hertz-Pannier, Stephen H Mott, Alan S Barnett, Denis LeBihan, and William H Theodore. 2000. Functional anatomy of cognitive development fMRI of verbal fluency in children and adults. Neurology 54, 1 (2000), 180-180.
- [11] Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, et al. 2012. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine 29 (2012).
- [12] Matthew Honnibal and Mark Johnson. 2014. Joint incremental disfluency detection and dependency parsing. Transactions of the Association for Computational Linguistics 2 (2014), 131-142.
- [13] Julian Hough and David Schlangen. 2015. Recurrent neural networks for incremental disfluency detection. In Proceedings of the 16th Annual Conference of the International Speech Communication Association .
- [14] Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing .
- [15] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015. Siamese neural networks for one-shot image recognition. In Proceedings of the ICML 2015 Deep Learning Workshop , Vol. 2.
- [16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 26th Annual Conference on Neural Information Processing . 1097-1105.
- [17] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In Proceedings of the 5th International Conference on Learning Representations .
- [18] Tiaoqiao Liu, Wenbiao Ding, Zhiwei Wang, Jiliang Tang, Gale Yan Huang, and Zitao Liu. 2019. Automatic Short Answer Grading via Multiway Attention Networks. In International Conference on Artificial Intelligence in Education . Springer, 169-173.
- [19] Yang Liu, Elizabeth Shriberg, Andreas Stolcke, Dustin Hillard, Mari Ostendorf, and Mary Harper. 2006. Enriching speech recognition with automatic detection of sentence boundaries and disfluencies. IEEE Transactions on Audio, Speech, and Language Processing 14, 5 (2006), 1526-1540.
- [20] Paria Jamshid Lou and Mark Johnson. 2017. Disfluency detection using a noisy channel model and a deep neural language model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics .
- [21] Andrew M Olney, Patrick J Donnelly, Borhan Samei, and Sidney K D'Mello. 2017. Assessing the Dialogic Properties of Classroom Discourse: Proportion Models for Imbalanced Classes. International Educational Data Mining Society (2017).
- [22] Mari Ostendorf and Sangyun Hahn. 2013. A sequential repetition model for improved disfluency detection.. In Proceedings of the 14th Annual Conference of the International Speech Communication Association . 2624-2628.
- [23] Melinda T Owens, Shannon B Seidel, Mike Wong, Travis E Bejines, Susanne Lietz, Joseph R Perez, Shangheng Sit, Zahur-Saleh Subedar, Gigi N Acker, Susan F Akana, et al. 2017. Classroom sound can be used to classify teaching practices in college science courses. Proceedings of the National Academy of Sciences 114, 12 (2017), 3085-3090.
- [24] Mohammad Sadegh Rasooli and Joel Tetreault. 2013. Joint parsing and disfluency detection in linear time. In Proceedings of the Conference on Empirical Methods in Natural Language Processing . 124-129.
- [25] Daria Riva, Francesca Nichelli, and Monica Devoti. 2000. Developmental aspects of verbal fluency and confrontation naming in children. Brain and Language 71, 2 (2000), 267-284.
- [26] Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočisk`, y and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proceedings of the 4th International Conference on Learning Representations .
- [27] Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 815-823.
- [28] William Schuler, Samir AbdelRahman, Tim Miller, and Lane Schwartz. 2010. Broad-coverage parsing using human-like memory constraints. Computational Linguistics 36, 1 (2010), 1-30.
- [29] Elizabeth Ellen Shriberg. 1994. Preliminaries to a theory of speech disfluencies . Ph.D. Dissertation. Citeseer.
- [30] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 1199-1208.
- [31] Chuanqi Tan, Furu Wei, Wenhui Wang, Weifeng Lv, and Ming Zhou. 2018. Multiway attention networks for modeling sentence pairs. In Proceedings of the 27th International Joint Conferences on Artificial Intelligence . 4411-4417.
- [32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems . 5998-6008.
- [33] Zhiguo Wang, Haitao Mi, and Abraham Ittycheriah. 2016. Sentence similarity learning by lexical decomposition and composition. In Proceedings of the 26th International Conference on Computational Linguistics . 1340-1349.
- [34] Guowei Xu, Wenbiao Ding, Jiliang Tang, Songfan Yang, Gale Yan Huang, and Zitao Liu. 2019. Learning Effective Embeddings From Crowdsourced Labels: An Educational Case Study. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) . IEEE, 1922-1927.
- [35] Victoria Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2014. Multi-domain disfluency and repair detection. In Proceedings of the 15th Annual Conference of the International Speech Communication Association .
- [36] Vicky Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using a bidirectional LSTM. In Proceedings of the 17th Annual Conference of the International Speech Communication Association .
Wenbiao Ding Guowei Xu Tianqiao Liu Weiping Fu Yujia Song Chaoyou Guo Cong Kong Songfan Yang Gale Yan Huang Zitao Liu
Dolphin: A Spoken Language Proficiency Assessment System for Elementary Education
Spoken language proficiency is critically important for children's growth and
personal development. Due to the limited and imbalanced educational resources
in China, elementary students barely have chances to improve their oral
language skills in classes. Verbal fluency tasks (VFTs) were invented to let
the students practice their spoken language proficiency after school. VFTs are
simple but concrete math related questions that ask students to not only report
answers but speak out the entire thinking process. In spite of the great
success of VFTs, they bring a heavy grading burden to elementary teachers. To
alleviate this problem, we develop Dolphin, a spoken language proficiency
assessment system for Chinese elementary education. Dolphin is able to
automatically evaluate both phonological fluency and semantic relevance of
students' VFT answers. We conduct a wide range of offline and online
experiments to demonstrate the effectiveness of Dolphin. In our offline
experiments, we show that Dolphin improves both phonological fluency and
semantic relevance evaluation performance when compared to state-of-the-art
baselines on real-world educational data sets. In our online A/B experiments,
we test Dolphin with 183 teachers from 2 major cities (Hangzhou and Xi'an) in
China for 10 weeks and the results show that VFT assignments grading coverage
is improved by 22\%.