Rainbow Keywords: Efficient Incremental Learning for Online Spoken Keyword Spotting

Catastrophic forgetting is a thorny challenge when updating keyword spotting (KWS) models after deployment. This problem will be more challenging if KWS models are further required for edge devices due to their limited memory. To alleviate such an issue, we propose a novel diversity-aware incremental learning method named Rainbow Keywords (RK). Specifically, the proposed RK approach introduces a diversity-aware sampler to select a diverse set from historical and incoming keywords by calculating classification uncertainty. As a result, the RK approach can incrementally learn new tasks without forgetting prior knowledge. Besides, the RK approach also proposes data augmentation and knowledge distillation loss function for efficient memory management on the edge device. Experimental results show that the proposed RK approach achieves 4.2% absolute improvement in terms of average accuracy over the best baseline on Google Speech Command dataset with less required memory. The scripts are available on GitHub.


Introduction
Spoken keyword spotting (KWS) [1] aims to identify the specific keywords in the audio input. It serves as a primary module in many real-world applications, such as Apple Siri and Google Home, which are widely utilized on the edge device [2,3]. Current deep-learning-based keyword spotting systems are usually trained with limited keywords in the compact model for lower computation and smaller footprint [4,5,6,7]. Therefore, the performance of the KWS model trained by the source-domain data may degrade significantly when confronted with unseen keywords of the target-domain at run-time [8,9].
To alleviate such a problem, prior work [10,11] utilize fewshot fine-tuning [12] to adapt KWS models with training data from the target-domain for new scenarios. However, performances on data from the source domain after adaptation could be poor, which is also known as the catastrophic forgetting problem [13]. Recent work [14] proposes a progressive continual learning [15,16,17] strategy for small-footprint keyword spotting to alleviate the catastrophic forgetting problem when adapting the model trained by source-domain data with the target-domain data. The limitations of such an approach are two-fold. First, the approach requires the task-ID as auxiliary information to learn the knowledge of different tasks, which is not always available in practice. Second, the storage volume occupied by the model will increase with the higher task numbers [18]. The storage volume will be unaffordable for light edge devices.
This paper proposes a novel diversity-aware incremental learning approach named Rainbow Keywords (RK) to address the issues mentioned above, requiring no task-ID information with fewer parameters. Specifically, the proposed RK approach introduces a diversity-aware sampler to select few but diverse examples from historical and incoming keywords by calculating classification uncertainty. As a result, the model will not forget the prior knowledge when learning new keywords even utilizing limited historical examples. Furthermore, we utilize a mixedlabeled data augmentation to additionally improve the diversity of selected examples for higher performances. Besides, we propose a knowledge distillation loss function to guarantee that the prior knowledge could remain from the limited selected examples. We conduct our experiments on Google Speech Command dataset following the setup of prior work [19,20]. Experimental results show that the proposed RK approach achieves 4.2% absolute improvement in terms of Average Accuracy over the best baseline with less required memory.

RK Architecture
With online keyword spotting systems, we assume that the model should identify all keywords in a series of tasks without catastrophic forgetting. For each task τt, we have input pairs (xt, yt), where xt denote audio utterances and yt are keyword labels. We aim to minimize a cross-entropy loss [21,22] of all keywords N t up to the current task τt formulated as : where o denotes the output logits of the model in the task τt.

Rainbow Keywords Network
We now introduce the proposed Rainbow Keywords (RK) approach, which consists of three parts: a diversity-aware sampler, a data augmentation strategy, and a knowledge distillation loss function. The detailed workflow of each module is as followed.
As illustrated in Figure 1, we first receive several incoming audio utterances of the task τt as inputs D  Figure 1: Block diagram of the proposed Rainbow Keywords approach. Specifically, D t S denotes incoming audio stream data of the task τt. D t E and D t−1 E denote the examples of the task τt and τt−1, respectively. We x,x and K present each sample in Dc, the five perturbations of x and the five perturbation strategies. "Compute u(x)" is to compute u(x) by Eq.3.

Diversity-Aware Sampler
The diversity-aware sampler aims to select diverse examples to manage memory efficiently. Such diverse examples are defined by the relative location of each example in feature space, which are estimated by the uncertainty of the sample through the inference by the classification model [23]. The required three steps are shown in the right red box of Figure 1: Split by keywords: The first step gathers the historical examples D t−1 E and incoming data D t S , and groups them into subsets as Dc, c = 1...N t by unique keywords, where N t denotes the total numbers of unique keywords in D t−1 E ∪ D t S set. Estimate uncertainty: The second step estimates the uncertainty of each sample x in Dc by Monte-Carlo (MC) method [24], which is defined in Equation 2.
where x,x, y denote each audio utterance of keyword D N t , the five perturbations of x, and the label of x. Therefore, the uncertainty of the audio utterance x is formulated as u(x): where K presents five perturbation strategies, including Clipping Distortion [25], TimeMask [25], Shift [26], PitchShift [26] and FrequencyMask [25]. The larger u(x) indicates that the less confidence of model to predict the perturbations.
Select by uncertainty: The third step selects L examples from Dc descending by uncertainty u(x) with the step size of Only these examples are available for training.

Data Augmentation
As the examples in D t E are few due to memory limitation, we apply the data augmentation to further increase the diversity [27] of D t E . Specifically, we randomly mix two audio utterances to increase the amounts of training data without extra storage.

Knowledge Distillation Loss
Recent studies [28,29,30] show that Knowledge Distillation (KD) is effective for transferring knowledge between teacherstudent models. Inspired by such theory, we consider the model of task τt−1 as the teacher model and the model of task τt as the student model. We propose a knowledge distillation loss to preserve the prior knowledge from the teacher for the student model to avoid catastrophic forgetting, which is formulated as: where o t−1 (x) and o t (x) denote the output logits of the teacher model and student model, respectively. N t is all keywords up to the task τt. σ is the knowledge distillation softmax function parameterized by the temperature T . The temperature T is the experiential hyper-parameters of knowledge distillation set as 2.0. As a result, we aim to minimize the total loss of all keywords N t up to the current task τt formulated as: Where LCE is the cross-entropy loss defined in Eq.1, and LKD is the knowledge distillation loss defined above. λ is the experiential hyper-parameters defined as 1 − N t−1 N t .

Dataset
We conduct experiments on the Google Speech Command dataset v1 (GSC) [31], which includes 64,727 one-second audio clips with 30 English keywords categories. We utilize 80% of data for training and 20% of data for testing. All of the audio clips in GSC are sampled at 16kHz in our experiment.

Network configuration
We employ the TC-ResNet-8 [32] as our testbed to evaluate the proposed rainbow keywords approach. It includes a 1-D convolution layer followed by three residual blocks, which consist of 1-D convolution, batch normalization and ReLU active function. Each layer has {16,24,32,48} channels. We first pre-train the TC-ResNet-8 model on the GSC dataset with 32,000 audio clips, including 15 unique keywords. To evaluate the learning ability of the proposed RK approach, we split the rest data as 5 tasks. Each task includes 3 new unique keywords, which is unseen in previous tasks. To simulate the condition of edge devices, we set the max amount of examples L due to the limited memory in edge devices [19,20].
During the training stage, we utilize the Mel-frequency cepstrum coefficients (MFCC = 40) as inputs. The network is optimized by the Adam [33] algorithm with the learning rate 0.1. The batch size is set to 128 and the number of epochs is 50.

Reference baselines
We built eight baselines for comparisons.
• Fine-tune training: adapts the TC-ResNet-8 model for each new task without class incremental learning (CIL) strategies. We consider it as the lower-bound baseline.
• NR [34]: is a CIL approach which randomly selects training samples from previous tasks for future training.
• iCaRL [30]: is a CIL approach which selects the samples close to the mean of its own class. Then iCaRL utilizes examples for future training.
• EWC [35]: is a CIL approach which incorporates a quadratic penalty to regularize parameters of model that were important to past tasks. The importance of parameters is approximated by the Fisher Information Matrix.
• RWalk [36]: is a CIL approach which improves the EWC. Both Fisher Information Matrix approximation [35] and online path integral [37] are fused to calculate the importance for each parameter. RWalk also selects historical examples and utilizes them for future training. • BiC [29]: is a recent CIL approach with more attentions. BiC introduces an additional layer to correct task bias of the network. BiC also uses the same sampling method as iCaRL to select historical examples.
• PCL-KWS [14]: is a CL approach for Spoken Keyword Spotting. Specifically, the PCL-KWS includes several task-specific sub-networks to memorize the knowledge of the previous keywords. Then, a keyword-aware network scaling mechanism is introduced to reduce the network parameters. The PCL-KWS requires the task-ID to select conspronding sub-networks.
• Joint training: trains the TC-ResNet-8 model with the whole dataset, regardless of any constrains. We consider it as the upper-bound baseline.

Metrics
We report performances in terms of the accuracy and efficiency metrics. The accuracy metrics include Average Accuracy (ACC), and Backward Transfer (BWT) [38]. Specifically, the 'Average Accuracy' reports an accuracy averaged on all learned tasks after the entire training ends. The "BWT" evaluates accuracy changes on all previous tasks after learning a new task, indicating the forgetting degree. The efficiency metrics include Parameters and Memory [39]. The 'Parameter' measures the total parameters of the model in the strategy. The 'Memory' indicates the memory requirement of total training data in each task.

Effect of the knowledge distillation loss
We first analyse and summarize the performances with knowledge distillation loss. As shown in Table 1, we observe that the proposed knowledge distillation loss function achieves 54.5% relative improvements in terms of BWT.

Effect of various task numbers on RK approach
We further analyse and summarize the performances of the proposed RK approach with increasing task numbers as shown in Figure 2. The x-axis represents the task numbers (= 20, 10, 5, 4) and the y-axis is the evaluation metric of ACC. For example, when the task number is set to 20, we pre-train the TC-ResNet-8 with 21,000 audio clips including 10 unique keywords and the rest data is split into 20 tasks including 1 unique unseen keyword. The corresponding ACC is the accuracy of the testing set after each task is finished training. The memory size L is 1500 for the proposed RK approach here. We observe that the proposed RK approach achieves the best ACC performances with increasing task numbers. Even though the difficulty of preserving prior knowledge is increased with the increasing task numbers, the proposed RK approach still can obtain over 85.0% ACC, which is much better than other baselines.  Table 2: Average Accuracy (ACC) and Backward Transfer (BWT) in a comparative study of the proposed data augmentation. "NoAugment" means no data augmentation is applied in the experiment. L denotes the memory size for RK.

Effect of various memory size on RK approach
We then report the effect of various memory size (L=300 or 500) on RK approach, as shown in Figure 3. We constrain all methods under the same memory size as that of the RK approach. We observe that even all approaches perform much better with increasing memory size (i.e., more available training data), the proposed RK approach still outperforms other methods. Furthermore, even with limited memory size (L=300), the proposed RK approach can obtain over 65.0% ACC performances, much better than other methods.

Effect of the data augmentation
We also summarize the performances of the effects with two data augmentation methods on the proposed RK approach, as This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-100E-2018-006).
The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg). shown in Table 2. We observe that all data augmentation methods improve the performances in terms of ACC. The best performances are achieved by the 'Mixup' data augmentation. We adopt the 'Mixup' data augmentation hereafter. Table 3 summarizes the comparison between the proposed RK and other competitive methods in terms of Average Accuracy (ACC), Backward Transfer (BWT), Parameters and Memory. The task number is set to 5. We observe that the proposed RK achieves the best performance with memory size L of 3000.

Benchmark against other competitive methods
Comparing with the best baselines: RWalk, the proposed RK-3000 achieves 4.2% absolute improvements in terms of Average Accuracy with fewer memory, which is closer to the upperbound performances. Furthermore, even with only 16.2M training data, our approach RK-500 has comparable performance to other baseline methods, which is effective on edge devices.

Conclusions
In this paper, we propose a novel diversity-aware class incremental learning method named Rainbow Keywords (RK) approach to avoid catastrophic forgetting with less memory. Experimental results show that the proposed RK approach achieves 4.2% absolute improvement in terms of average accuracy over the best baseline. Ablation study also indicates that the proposed data augmentation and knowledge distillation loss are quite effective on edge devices.