summaries: In this paper, the authors study the online knowledge distillation problem and propose a method called AFD (Online Adversarial Feature map Distillation), which aims to transfers the knowledge of intermediate feature map (first propose) using adversarial training. Then, a cyclic learning scheme is proposed to train more than two networks simultaneously and efficiently. Ablation study on CIFAR100 shows that the adversarial training in AFD can improve the accuracy significantly, while the direct method such as using L1 distance is worse. The comparison experiments with several online distillation methods also show the effectiveness of proposed method.

summaries: A new online knowledge distillation is investigated by utilizing feature map information next to the logits via GAN. Instead of direct feature map alignment, the algorithm tries to transfer the distribution of the feature maps. There is no teacher per se, but the big and small nets are trained via an adversarial game where 2 discriminators try to minimize the distributions of the two nets. The idea is understandable but some issues remain:

summaries: This paper presents a new deep mutual learning (i.e., online peer-teaching) method based on Knowledge Distillation (KD) in a feature map level. The target task is similar with the original KD in the sense that the a network is taught by another network as well as groundtruth labels, but different with the KD in the sense that the networks are not a (frozen) teacher and a student but teaching each other in an online manner. Most approaches in this relatively new line of research rely on logit-based KD for transferring knowledges between networks, and the paper demonstrates that by an additional feature map level KD the performance can be further improved.

abstractText: Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network’s feature map distribution. Discriminators and networks are trained concurrently in a minimax twoplayer game. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.

authors:
Bo Dai
Sanja Fidler
Raquel Urtasun
Dahua Lin

title: Towards diverse and natural image descriptions via a conditional gan

venue: In Proceedings of the IEEE International Conference on Computer Vision,

authors:
Ian Goodfellow
Jean Pouget-Abadie
Mehdi Mirza
Bing Xu
David Warde-Farley
Sherjil Ozair
Aaron Courville
Yoshua Bengio

title: Generative adversarial nets

venue: In Advances in neural information processing systems,

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Geoffrey Hinton
Oriol Vinyals
Jeff Dean

title: Distilling the knowledge in a neural network

venue: arXiv preprint arXiv:1503.02531,

authors:
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A Efros

title: Image-to-image translation with conditional adversarial networks

authors:
Jangho Kim
SeongUk Park
Nojun Kwak

title: Paraphrasing complex network: Network compression via factor transfer

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Alex Krizhevsky
Vinod Nair
Geoffrey Hinton

title: Cifar-100 (canadian institute for advanced research). URL http://www.cs.toronto.edu/ ̃kriz/cifar.html

authors:
Xu Lan
Xiatian Zhu
Shaogang Gong

title: Knowledge distillation by on-the-fly native ensemble

venue: In Proceedings of the 32nd International Conference on Neural Information Processing Systems,

authors:
Hao Li
Asim Kadav
Igor Durdanovic
Hanan Samet
Hans Peter Graf

title: Pruning filters for efficient convnets

venue: arXiv preprint arXiv:1608.08710,

authors:
Peiye Liu
Wu Liu
Huadong Ma
Tao Mei
Mingoo Seok

title: Ktan: knowledge transfer adversarial network

venue: arXiv preprint arXiv:1810.08126,

authors:
Xudong Mao
Qing Li
Haoran Xie
Raymond YK Lau
Zhen Wang
Stephen Paul Smolley

title: Least squares generative adversarial networks

authors:
Takeru Miyato
Andrew M Dai
Ian Goodfellow

title: Adversarial training methods for semisupervised text classification

venue: arXiv preprint arXiv:1605.07725,

authors:
David Pfau
Oriol Vinyals

title: Connecting generative adversarial networks and actor-critic methods

venue: arXiv preprint arXiv:1610.01945,

authors:
Mohammad Rastegari
Vicente Ordonez
Joseph Redmon
Ali Farhadi

title: Xnor-net: Imagenet classification using binary convolutional neural networks

venue: In European Conference on Computer Vision,

authors:
Adriana Romero
Nicolas Ballas
Samira Ebrahimi Kahou
Antoine Chassang
Carlo Gatta
Yoshua Bengio

title: Fitnets: Hints for thin deep nets

venue: arXiv preprint arXiv:1412.6550,

title: Unsupervised and semi-supervised learning with categorical generative adversarial networks

venue: arXiv preprint arXiv:1511.06390,

authors:
Sergey Zagoruyko
Nikos Komodakis

title: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer

venue: arXiv preprint arXiv:1612.03928,

title: Wide residual networks

venue: arXiv preprint arXiv:1605.07146,

authors:
Ying Zhang
Tao Xiang
Timothy M Hospedales
Huchuan Lu

title: Deep mutual learning

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Jun-Yan Zhu
Taesung Park
Phillip Isola
Alexei A Efros

title: Unpaired image-to-image translation using cycle-consistent adversarial networks

venue: In Proceedings of the IEEE international conference on computer vision,

heading: 1 INTRODUCTION

text:
With the advent of Alexnet (Krizhevsky et al., 2012), deep convolution neural networks have achieved remarkable success in a variety of computer vision tasks. However, high-performance of deep neural network is often gained by increasing the depth or the width of a network. Deep and wide networks cost a large number of computation as well as memory storage which is not suitable for a resource-limited environment such as mobile or embedded systems. To overcome this issue, many researches have been conducted to develop smaller but accurate neural networks. Some of the well-known methods in this line of research are parameter quantization or binarization (Rastegari et al., 2016), pruning (Li et al., 2016) and knowledge distillation (KD) (Hinton et al., 2015).
KD has been an active area of research as a solution to improve the performance of a light-weight network by transferring the knowledge of a large pre-trained network (or an ensemble of small networks) as a teacher network. KD sets the teacher network’s class probabilities as a target which a small student network tries to mimic. By aligning the student’s predictions to those of the teacher, the student can improve its performance. Recently, some studies have shown that rather than using a pretrained teacher, simultaneously training networks to learn from each other in a peer-teaching manner is also possible. This approach is called online distillation. Deep mutual learning (DML) (Zhang et al., 2018) and on-the-fly native ensemble (ONE) (Lan et al., 2018) are the representative online distillation methods that show appealing results in the image classification tasks. Conventional distillation method requires pre-training a powerful teacher network and performs an one-way transfer to a relatively small and untrained student network. On the other hand, in online mutual distillation, there is no specific teacher-student role. All the student networks learn simultaneously by teaching each other from the start of training. It trains with the conventional cross-entropy loss from the ground truth label along with the mimicry loss to learn from its peers. Networks trained in such an online distillation way achieve results superior not only to the networks trained with the cross-entropy loss alone but also to those trained in conventional offline distillation manner from a pre-trained teacher network.
However, aforementioned online distillation methods make use of only the logit information. While the logit contains the probabilistic information over classes, the feature map, the output of convolution layer, has more meaningful and abundant feature information on image intensity and spatial correlation. In offline distillation which utilizes a pre-trained model as a teacher network, many methods such as FitNet (Romero et al., 2014), attention transfer (AT) (Zagoruyko & Komodakis, 2016a) and factor transfer (FT) (Kim et al., 2018) make use of this intermediate feature representation as a target to learn for the student network, but in online distillation, to the best of our knowledge, no feature map-based knowledge distillation method has been proposed.
This is due to some challenges. Unlike the offline methods that have a clear target to mimic, there is no static target to follow in an online method. At every training iteration, the feature maps of the co-trained network change, thus in online feature map-level distillation, the problem turns into mimicking the moving target properly. While each node of the logit is confined to represent its assigned class probability which does not change drastically over iterations, at the feature map-level, much more flexibility comes into play, which makes the problem more challenging. Therefore, the direct aligning method such as using L1 or L2 distance is not suitable for online mutual feature map distillation because it updates the network parameters to generate a feature map that tries to mimic the current output feature map of the other network. In other words, the direct alignment method only tries to minimize the distance between the two feature map points (one for each network), hence it ignores the distributional difference between the two feature maps (Fig. 1(a)).
To alleviate this problem, in this paper, we propose a novel online distillation method that transfers the knowledge of feature maps adversarially as well as a cyclic learning framework for training more than two networks simultaneously. Unlike the direct aligning method, our adversarial distillation method enables a network to learn the overall feature map distribution of the co-trained network (Fig. 1(b)). Since the discriminator is trained to distinguish the difference between the networks’ feature map distributions (containing the history of feature maps for different input images) at every training iteration, by fooling the discriminator, the network learns the co-trained network’s changing feature map distribution. Exchanging the knowledge of feature map distribution facilitates the networks to converge to a better feature map manifold that generalizes better and yields more accurate results.
Our method consists of two major losses: 1) logit-based loss and 2) feature map-based loss. Logitbased loss is defined by two different loss terms which are conventional cross-entropy (CE) loss and the mutual distillation loss using the Kullback-Leibler divergence (KLD). Our newly proposed feature map-based loss is to distill the feature map indirectly via discriminators. We use the feature map from the last convolution layer since deeper convolution layer generates more meaningful features with a high-level abstraction (Kim et al., 2018). The adversarial training scheme of generative adversarial networks (GAN) (Goodfellow et al., 2014) is utilized to transfer the knowledge at feature map-level.
The contributions of this paper can be summarized as follows: 1) we propose an online knowledge distillation method that utilizes not only the logit but also the feature map from the convolution layer. 2) Our method transfers the knowledge of feature maps not by directly aligning them using
the distance loss but by learning their distributions using the adversarial training via discriminators. 3) We propose a novel cyclic learning scheme for training more than two networks simultaneously.

heading: 2 RELATED WORK

text:
The idea of model compression by transferring the knowledge of a high performing model to a smaller model was originally proposed by Buciluǎ et al. (2006). Then in recent years, this research area got invigorated due to the work of knowledge distillation (KD) by Hinton et al. (2015). The main contribution of KD is to use the softened logit of pre-trained teacher network that has higher entropy as an extra supervision to train a student network. KD trains a compact student network to learn not only by the conventional CE loss subjected to the labeled data but also by the final outputs of the teacher network. While KD only utilizes the logit, method such as FitNet (Romero et al., 2014), AT (Zagoruyko & Komodakis, 2016a), FT (Kim et al., 2018) and KTAN (Liu et al., 2018) use the intermediate feature representation to transfer the knowledge of a teacher network.
Online Knowledge Distillation: Conventional offline methods require training a teacher model in advance while online methods do not require any pre-trained model. Instead, the networks teach each other mutually by sharing their knowledge throughout the training process. Some examples of recent online methods are DML (Zhang et al., 2018) and ONE (Lan et al., 2018) which demonstrate promising results. DML simply applies KD losses mutually, treating each other as teachers, and it achieves results that is even better than the offline KD method. The drawback of DML is that it lacks an appropriate teacher role, hence provides only limited information to each network. ONE pointed out this defect of DML. Rather than mutually distilling between the networks, ONE generates a gated ensemble logit of the training networks and uses it as a target to align for each network. ONE tries to create a powerful teacher logit that can provide more generalized information. The flaw of ONE is that it can not train different network architectures at the same time due to its architecture of sharing the low-level layers for the gating module. The common limitation of existing online methods is that they are dependent only on the logit and do not make any use of the feature map information. Considering that KD loss term is only applicable to the classification task, transferring knowledge at feature map-level can enlarge the applicability to other tasks. Therefore, our method proposes a distillation method that utilizes not only the logit but also the feature map via adversarial training, moreover, our method can be applied in case where the co-trained networks have different architectures.
Generative Adversarial Network (GAN): GAN (Goodfellow et al., 2014) is a generative model framework that is proposed with an adversarial training scheme, using a generator network G and a discriminator network D. G learns to generate the real data distribution while D is trained to distinguish the real samples of the dataset from the fake results generated by G. The goal of G is to trickD to make a mistake of determining the fake results as the real samples. Though it was initially proposed for generative models, its adversarial training scheme is not limited to data generation. Adversarial training has been adapted to various tasks such as image translation (Isola et al., 2017; Zhu et al., 2017), captioning (Dai et al., 2017), semi-supervised learning (Miyato et al., 2016; Springenberg, 2015), reinforcement learning (Pfau & Vinyals, 2016), and many others. In this paper, we utilize GAN’s adversarial training strategy to transfer the knowledge at feature map-level in an online manner. The networks learn the other networks’ feature map distributions by trying to deceive the discriminators while the discriminators are trained to distinguish the different distributions of each network.

heading: 3 PROPOSED METHOD

text:
In this section, we describe the overall process of our proposed Online Adversarial Feature map Distillation (AFD). As can be seen in Figure 2, when training two different networks, Θ1 and Θ2, in an online manner, we employ two discriminators, D1 and D2. We train D1 such that the feature map of Θ2 is regarded as a real and that of Θ1 is classified as a fake and do vice versa for discriminator D2. Then, each network Θ1 and Θ2 are trained to fool its corresponding discriminator so that it can generate a feature map that mimics the other network’s feature map. Throughout this adversarial training, each network learns the feature map distribution of the other network. By exploiting both logit-based distillation loss and feature map-based adversarial loss together, we could observe
a significant improvement of performance in various pairs of network architectures especially when training small and large networks together. Also we introduce a cyclic learning scheme for training more than two networks simultaneously. It reduces the number of required discriminators from 2×2 CK(when employing discriminators bidirectionally between every network pairs.) to K where K is the number of networks participating. This cyclic learning framework not only requires less computation than the bidirectional way but also achieves better results compared to other online training schemes for multiple networks. First, we explain the conventional mutual knowledge distillation method conducted among the networks at the logit-level. Then we introduce our novel online feature map distillation method using the adversarial training scheme in addition to the cyclic learning framework for training more than two networks at the same time.

heading: 3.1 LOGIT-BASED MUTUAL KNOWLEDGE DISTILLATION

text:
We use two loss terms for logit-based learning, one is the conventional cross-entropy(CE) loss and the other is mutual distillation loss between networks based on Kullback Leibler(KL) divergence. We formulate our proposed method assuming training two networks. Training scheme for more than two networks will be explained in Sec 3.3. Below is the overall logit-based loss for two networks:
L1logit = Lce(y, σ(z1)) + T 2 × Lkl(σ(z2/T ), σ(z1/T )) (1) L2logit = Lce(y, σ(z2)) + T 2 × Lkl(σ(z1/T ), σ(z2/T )). (2)
Here, σ(·) refers to softmax function and z ∈ RC is the logit produced from a network for Cclass classification problem. The temperature term T is used to control the level of smoothness in probabilities. As the temperature term T goes up, it creates a more softened probability distribution. We use T = 3 for every experiment. Lce is the CE loss between the ground truth label y and the softmax output σ(z) that is commonly used in image classification. Lkl is the KL loss between the softened logit of each network. We multiply the KL loss term with T 2 because the gradients produced by the soft targets are scaled by 1/T 2. While the CE loss is between the correct labels and the outputs of the model, the KL loss is the KL distance between the outputs of two training networks. The KL loss provides an extra information from the peer network so that the network can improve its generalization performance. The difference with DML is that while DML updates asynchronously which means that it updates one network first and then the other network, our AFD updates the networks synchronously, not alternatingly. The CE loss trains the networks to predict the correct truth label while the mutual distillation loss tries to match the outputs of the peer-networks, enabling the networks to share the knowledge at logit-level.

heading: 3.2 FEATURE MAP-BASED LEARNING VIA ADVERSARIAL TRAINING

text:
Our AFD uses adversarial training to transfer knowledge at feature map-level. We formulate our adversarial feature map distillation for two networks which will be extended for more networks later. We divide a network into two parts, one is the feature extractor part that generates a feature map and the other is the classifier part that transforms the feature map into a logit. Each network also has a corresponding discriminator which distinguishes different feature map distributions. The architecture of the discriminator is simply a series of Conv-Batch Normalization-Leaky ReLU-Conv-Sigmoid. It takes a feature map of the last layer and it reduces the spatial size and the number of channel of the feature map as it goes through the convolution operation so that it can produce a single scalar value. Then we apply the sigmoid function of the value to normalize it between 0 and 1. We utilize the feature extractor part to enable feature map-level distillation. For the convenience of mathematical notation, we name the feature extractor part as Gk and its discriminator as Dk, k indicates the network number. As depicted in Figure 2, each network has to fool its discriminator to mimic the peer network’s feature map and the discriminator has to discriminate from which network the feature map is originated. Following LSGAN (Mao et al., 2017), our overall adversarial loss for discriminator and the feature extractor can be written as below:
LD1 = [1−D1(G2(x))]2 + [D1(G1(x))]2 (3) LG1 = [1−D1(G1(x))]2. (4)
The feature extractors G1 and G2 take input x and generate feature maps. The discriminator D1 takes a feature map and yields a scalar between 0 (fake) and 1 (real). It is trained to output 1 if the feature map came from the co-trained network (in this case, G2) or 0 if the feature map is produced from the network it belongs to (G1 in this case). The goal ofD1 is to minimize the discriminator loss term LD1 by correctly distinguishing the two different feature map distributions while G1’s goal is to minimize the loss term LG1 by fooling D1 to make mistake of determining G1’s feature map as real and yield 1. Each training network’s object is to minimize LGk to mimic the peer network’s feature map distribution. This adversarial scheme works exactly the same by changing the role of two networks.
In case where the two networks’ feature map outputs have different channel sizes, for example a pair like (WRN-16-2, WRN-16-4) (Zagoruyko & Komodakis, 2016b), we use a transfer layer that is composed of a convolution layer, a batch normalization and a ReLU which converts the number of channels to that of peer network. The above loss terms change as LD1 = [1−D1(T2(G2(x)))]2 + [D1(T1(G1(x)))]
2 and LG1 = [1−D1(T1(G1(x)))]2 when using the transfer layer Tk. Optimization: Combining both logit-based loss and the adversarial feature map-based loss, the overall loss for each network Θ1 and Θ2 are as follows:
LΘ1 = L1logit + LG1 , LΘ2 = L2logit + LG2 (5)
However, the logit-based loss termLklogit and the feature map-based loss termLGk are not optimized by the same optimizer. In fact, they are optimized alternatingly in a same mini-batch. At every minibatch iteration, we infer an image into a model and it computes a logit and a feature map. Then we calculate the two loss terms and optimize the networks based on the two losses separately, meaning that we update the parameters by the logit-based loss once and then update again by the feature map-based loss. The reason we optimize separately for each loss term is because they use different learning rates. The adversarial loss requires much slower learning rate thus if we use the same optimizer with the same learning rate, the networks would not be optimized. Note that we do not infer for each loss term, inference is conducted only once, only the optimization is conducted twice, one for each loss term.

heading: 3.3 CYCLIC LEARNING FRAMEWORK

text:
Our method proposes a novel cyclic peer-learning scheme for training more than two networks simultaneously. As can be seen in Figure 3, each network transfers its knowledge to its next peer network in an one-way cyclic manner. If we train K number of networks together, each network distills its knowledge to its next network except the last network transfers its knowledge to the first
network, creating a cyclic knowledge transfer flow as 1 → 2, 2 → 3, · · · , (K − 1) → K,K → 1. The main contribution of using this cyclic learning framework is to avoid employing too many number of discriminators. If we apply our adversarial loss for every pair of networks, it would demand two times the amount of every possible pair of K networks which would cost a lot of computation. Also in Sec 4.5, we empirically show that our cyclic training scheme is better than other online methods’ training scheme for multiple networks.

heading: 4 EXPERIMENT

text:
In this section, to show the adequacy of our method, we first present comparison experiment with distance method and ablation study to analyze our method. Then we compare our approach with existing online knowledge distillation methods under different settings. First of all, we demonstrate results on using the same sub-network architectures in Sec 4.3. Then, we apply our method on subnetworks with different architectures in Sec 4.4. In Sec 4.5, we also show the results of training more than two networks to demonstrate that our method generalizes well even when the number of networks increases.
In most of the experiments, we use the CIFAR-100 (Krizhevsky et al.) dataset. It consists of 50K training images and 10K test images over 100 classes, accordingly it has 600 images per each class. All the reported results on CIFAR-100 are average of 5 experiments. Since our method uses two loss terms, logit-based loss and feature map-based loss, we use different learning details for each loss term. For overall learning schedule, we follow the learning schedule of ONE(Lan et al., 2018) to conduct fair comparison which is 300 epochs of training. In terms of logit-based loss, the learning rate starts at 0.1 and is multiplied by 0.1 at 150, 225 epoch. We optimize the logit-based loss using SGD with mini-batch size of 128, momentum 0.9 and weight decay of 1e-4. This learning details for logit-based loss is equally applied to other compared online distillation methods. For feature map-based loss, the learning rate starts at 2e-5 for both discriminators and feature extractors and is decayed by 0.1 at 75, 150 epoch. The feature map-based loss is optimized by ADAM(Kingma & Ba, 2014) with the same mini-batch size and weight decay of 1e-1.
In tables, ‘2 Net Avg’ and ‘Ens’ represents the average accuracy of the two sub-networks and the ensemble accuracy respectively. The average ensemble is used for AFD, DML and KD while ONE uses gated ensemble of sub-networks according to its methodology.

heading: 4.1 COMPARISON WITH DIRECT FEATURE MAP ALIGNMENT METHODS

text:
Since our goal is to distill feature map information that suits for mutual online distillation, we briefly compare our method with conventional direct alignment method in Table 1. We train two networks together, in one setting, we use the same architecture (ResNet-32 (He et al., 2016)) and in the other, we use different types (WRN-16-2, WRN-28-2 (Zagoruyko & Komodakis, 2016b)). For L1, each
network is trained not only to follow the ground-truth label by CE loss, but also to mimic the other network’s feature map using the L1 distance loss. For L1+ KD, KD (Hinton et al., 2015) loss is applied mutually along with the L1 loss between the feature maps. We also compare our results with offline method, L1+ KD (offline) employs a pre-trained network as a teacher network and distills its feature map knowledge to an untrained student network by L1 loss as well as the KD loss at logit level. ResNet-32 and WRN-28-2 that shows 69.79% and 73.62% accuracy are used as the teacher networks in the two settings respectively. The results clearly show that learning the distributions of feature maps with adversarial loss performs better than direct alignment method in both mutual online distillation and offline distillation. We could observe that using L1 distance loss actually disturbs the networks to learn good features in online environment. The accuracy of ResNet-32 has dropped more than 2% compared to its vanilla version accuracy (69.38%) and the accuracy of WRN-16-2 is also lower than its vanilla network (71.07%). Even when combined with KD loss(L1 + KD), direct alignment method shows poor performance compared to ours in both online and offline manner. Though distance loss is used in many conventional offline methods, they suffer when it comes to online environment. In case of different architecture types, our method also outperforms the direct alignment method. It indicates that when it comes to online feature map distillation, transferring feature map information with direct alignment method such as L1 distance is worse than indirect distillation that uses feature map distribution via adversarial loss.

heading: 4.2 ABLATION STUDY

text: Table 2 shows the ablation study of our proposed method. We conduct experiments using the same and different sub-network architectures. We run three experiments with different training settings for each model case. The three settings are full model, without mutual knowledge distillation at logitlevel and without adversarial feature map distillation. When trained without the adversarial feature map distillation, the accuracy decreases in all three model cases. The accuracy of both ResNet-32 and WRN-16-2 dropped by 0.65% and 0.52% respectively, and those of (WRN-16-2, WRN-28-2) pair declined by 0.89% and 0.44% compared to the full model. Ensemble results are also lower than those of the full models. When only the adversarial feature map distillation is applied, the accuracy has increased by 0.71% and 0.87% compared to the vanilla versions of ResNet-32 and WRN-16-2 respectively. Especially in case of different sub-network architecture, the accuracy of WRN-16-2 has increased by almost 1%. Based on these experiments, we could confirm that adversarial feature map distillation has some efficacy of improving the performance in online environment.

heading: 4.3 SAME ARCHITECTURE

text:
We compare our method with DML and ONE for training two sub-networks with the same architecture. The vanilla network refers to the original network trained without any distillation method. As shown in Table 3, in both ResNet and WRN serises, DML, ONE and AFD all improves the networks’ accuracy compared to the vanilla networks. However, AFD shows the highest improvement of performance in both sub-network and ensemble accuracy among the compared distillation methods. Especially in case of ResNet-20, ResNet-32 and WRN-16-2, our method significantly improves the
accuracy by more than 4% compared to the vanilla version while other distillation methods improve around 3% on average except the ResNet-32 of DML.

heading: 4.4 DIFFERENT ARCHITECTURE

text: In this section, we compare our method with DML and KD using different network architectures. We set Net2 as the higher capacity network. For KD, we use the ensemble of the two sub-networks as a teacher to mimic at every iteration. The difference with original KD (Hinton et al., 2015) is that it is an online learning method, not offline. We did not include ONE because ONE can not be applied in case where the sub-networks have different model types due to its architecture of sharing the lowlevel layers. In table 4, we could observe that our method shows better performance improvement than other methods in both Net1 and Net2 except for a couple of cases. The interesting result is that when AFD is applied, the performance of Net1 (smaller network) is improved significantly compared to other online distillation methods. This is because AFD can transfer the higher capacity network’s meaningful knowledge (feature map distribution) to the lower capacity one better than other online methods. When compared with KD and DML, AFD’s Net1 accuracy is higher by 1.66% and 1.15% and the ensemble accuracy is better by 0.61% and 0.67% on average respectively. In case of (WRN-16-2, WRN-28-4) pair, the Net1’s parameter size (0.70M) is more than 8 times smaller than Net2 (5.87M). Despite the large size difference, our method improves both networks’ accuracy, particularly our Net1 performance is better than KD and DML by 1.72% and 1.28% respectively. The performance of KD and DML seems to decline as the difference between the two model sizes gets larger. Throughout this experiment, we have shown that our method also works properly for different architectures of sub-networks even when two networks have large difference in their model sizes. Using our method, smaller network considerably benefits from the large network.

heading: 4.5 EXPANSION TO 3 NETWORKS

text:
To show our method’s expandability for training more than two networks, we conduct experiment of training 3 networks in this section. As proposed in Sec 3.3, our method uses a cyclic learning framework rather than employing adversarial loss between every network pairs in order to reduce
the amount of computation and memory. DML calculates the mutual knowledge distillation loss between every network pairs and uses the average of the losses. ONE generates a gated ensemble of the sub-networks and transfers the knowledge of the ensemble logit to each network. As it can be seen in Table 5, AFD outperforms the compared online distillation methods on both 3 Net average and ensemble accuracy in every model types. Comparing the results of Table 5 to that of Table 3, the overall tendency of performance gains compared to DML and ONE is maintained.

heading: 4.6 IMAGENET EXPERIMENT

text: We evaluate our method on ImageNet dataset to show that our method can also be applicable to a large scale image dataset. We use ImageNet LSVRC 2015 (Russakovsky et al., 2015) which has 1.2M training images and 50K validation images over 1,000 classes. We compare our method with DML using two pre-trained networks ResNet-18 and ResNet-34 as a pair. The results are after 30 epochs of training. As shown in Table 6, our method improves the networks better than DML.

heading: 5 CONCLUSION

text: We proposed an online knowledge distillation method that transfers the knowledge not only at logitlevel but also at feature map-level using the adversarial training scheme. Unlike existing online distillation methods, our method utilizes the feature map information and showed that knowledge transfer at feature map-level is possible even in an online environment. Through extensive experiments, we demonstrated the adequacy of adopting the distribution learning via adversarial training for online feature map distillation and could achieve better performance than existing online methods. We also introduced a novel cyclic learning framework for training multiple networks concurrently and presented its efficacy by comparing with existing approaches. We also confirmed that our method is broadly suitable to various architecture types from a very small network (ResNet-20) to a large (WRN-28-4) network. We hope that due to the work of our research, the area of knowledge distillation can be further advanced and studied by many researchers.

title: FEATURE-MAP-LEVEL ONLINE ADVERSARIAL KNOWLEDGE DISTILLATION


summaries: The paper aims at extending GloVe word embedding model so that the resulting embeddings should capture sentiments (e.g. "good" is positive while "bad" is negative). The key idea is to employ an extension term to deal with the fact that some words appearing in text with sentiment information. Furthermore, to deal with the fact that many words an infrequent, besides maximum likelihood estimation, the paper proposes to use bayesian estimation. In the experiments, Stanford sentiment tree (SST) corpus is used. The word embeddings from the two models (each trained on different estimation methods) show their capability of expressing sentiments, compared with popular methods like Glove, word2vec.

summaries: This paper proposes a method to learn word embedding by incorporating additional sentiment information. The proposed method extends from D-GloVe by adding the probability of positive sentiment to the loss function. The paper presents three experiments: word similarity, word-level sentiment analysis, and sentence-level sentiment analysis. The experiments show that the method performs comparably with other baseline methods and outperforms in the low-frequency sentence setting (i.e. sentence containing lower frequency words).

summaries: The paper proposed a word embedding model to incorporate the sentiment information. The paper provided both maximum likelihood estimation and maximum posterior estimation for the proposed framework. Improved experiment results on word similarity and low frequency embeddings are presented. Overall, the paper incorporates the sentiment information in a neat way. And my main concern is the around the Bayesian inference and the prior knowledge distilled into the model.  Detail comments are as following,

abstractText: The word embedding models have achieved state-of-the-art results in a variety of natural language processing tasks. Whereas, current word embedding models mainly focus on the rich semantic meanings while are challenged by capturing the sentiment information. For this reason, we propose a novel sentiment word embedding model. In line with the working principle, the parameter estimating method is highlighted. On the task of semantic and sentiment embeddings, the parameters in the proposed model are determined by using both the maximum likelihood estimation and the Bayesian estimation. Experimental results show the proposed model significantly outperforms the baseline methods in sentiment analysis for low-frequency words and sentences. Besides, it is also effective in conventional semantic and sentiment analysis tasks.

authors:
Ronan Collobert
Jason Weston

title: A unified architecture for natural language processing: Deep neural networks with multitask learning

venue: In Proceedings of the 25th international conference on Machine learning,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: Bert: Pretraining of deep bidirectional transformers for language understanding

venue: arXiv preprint arXiv:1810.04805,

authors:
Petar M Djuric
Yufei Huang

title: Estimation of a bernoulli parameter p from imperfect trials

venue: IEEE Signal Processing Letters,

title: A bayesian analysis of some nonparametric problems

venue: The annals of statistics,

authors:
Shoaib Jameel
Steven Schockaert

title: D-glove: A feasible least squares model for estimating word embedding densities

authors:
Shoaib Jameel
Zihao Fu
Bei Shi
Wai Lam
Steven Schockaert

title: Word embedding as maximum a posteriori estimation

venue: In Proceedings of the AAAI Conference on Artificial Intelligence,

title: Convolutional neural networks for sentence classification

venue: arXiv preprint arXiv:1408.5882,

authors:
Omer Levy
Yoav Goldberg

title: Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume

authors:
Omer Levy
Yoav Goldberg
Ido Dagan

title: Improving distributional similarity with lessons learned from word embeddings

venue: Transactions of the Association for Computational Linguistics,

authors:
Yang Li
Quan Pan
Tao Yang
Suhang Wang
Jiliang Tang
Erik Cambria

title: Learning word representations for sentiment analysis

venue: Cognitive Computation,

authors:
Zhanyu Ma
Yuping Lai
W Bastiaan Kleijn
Yi-Zhe Song
Liang Wang
Jun Guo

title: Variational bayesian learning for dirichlet process mixture of inverted dirichlet distributions in non-gaussian image feature modeling

venue: IEEE transactions on neural networks and learning systems,

authors:
Andrew L Maas
Raymond E Daly
Peter T Pham
Dan Huang
Andrew Y Ng
Christopher Potts

title: Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologiesvolume 1, pages 142–150

authors:
Tomas Mikolov
Kai Chen
Greg Corrado
Jeffrey Dean

title: Efficient estimation of word representations in vector space

venue: arXiv preprint arXiv:1301.3781,

authors:
Bo Pang
Lillian Lee

title: Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 115–124

authors:
Jeffrey Pennington
Richard Socher
Christopher Manning

title: Glove: Global vectors for word representation

venue: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP),

authors:
Matthew E Peters
Mark Neumann
Mohit Iyyer
Matt Gardner
Christopher Clark
Kenton Lee
Luke Zettlemoyer

title: Deep contextualized word representations

venue: arXiv preprint arXiv:1802.05365,

authors:
Seuk-Yen Phoong
Mohd Tahir Ismail

title: A comparison between bayesian and maximum likelihood estimations in estimating finite mixture model for financial data

venue: Sains Malaysiana,

title: Word and Relation Embedding for Sentence Representation

venue: PhD thesis, Arizona State University,

authors:
Bei Shi
Zihao Fu
Lidong Bing
Wai Lam

title: Learning domain-sensitive and sentimentaware word embeddings

venue: arXiv preprint arXiv:1805.03801,

authors:
Duyu Tang
Furu Wei
Bing Qin
Ming Zhou
Ting Liu

title: Building large-scale twitterspecific sentiment lexicon: A representation learning approach

venue: In Proceedings of coling 2014,

authors:
Duyu Tang
Furu Wei
Bing Qin
Nan Yang
Ting Liu
Ming Zhou

title: Sentiment embeddings with applications to sentiment analysis

venue: IEEE Transactions on Knowledge and Data Engineering,

authors:
Anqi Wu
Sebastian Nowozin
Edward Meeds
Richard E Turner
José Miguel HernándezLobato
Alexander L Gaunt

title: Deterministic variational inference for robust bayesian neural networks. 2018

authors:
Zhilin Yang
Zihang Dai
Yiming Yang
Jaime Carbonell
Ruslan Salakhutdinov
Quoc V Le

title: Xlnet: Generalized autoregressive pretraining for language understanding

authors:
Liang-Chih Yu
Jin Wang
K Robert Lai
Xuejie Zhang

title: Refining word embeddings using intensity scores for sentiment analysis

venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,

heading: 1 Introduction

text:
Word embeddings provide continuous low-dimensional vector representations of words from documents (Li et al., 2017). Aiming to capture semantic and syntactic contextual information from large datasets, the word embedding models are extensively employed to represent words in natural language processing tasks (Levy and Goldberg, 2014). For this reason, many modelling methods are proposed to generate dense representations of words (Rath, 2017). Seeing the flourish of word embeddings, Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) are considered as the edge-cutting approaches to deal with the word contexts. C&W is another most widespread method due to the progress in neural networks (Collobert and Weston, 2008). Besides, other algorithms are integrated into the existing models. For instance, Jameel and Schockaert proposes D-GloVe by combing GloVe and Dirichlet-Multinomial language modeling (Jameel and Schockaert, 2016). More recently, the contextualized word embedding models, which improve the accuracy to a large extent, are put forward. As such, the traditional approaches are concluded as pre-trained word embeddings. Whereas, the newly proposed methods, such as ELMo by Peters M. E. et al.(Peters et al., 2018) , BERT by Devlin J. et al.(Devlin et al., 2018) and XLNet by Yang Z et al.(Yang et al., 2019) , cost large amount of computing resource for training whilst obtain a better working performance in downstream tasks. In this way, the pre-trained word embeddings still hold a great promise in handling complicated natural language processing tasks. The aforementioned models are effective in dealing with semantic-oriented tasks. Likewise, in sentiment analysis, research is still ongoing to capture sufficient sentiment information while the sentiment embeddings typically depend on the sentiment polarity labels provided by labeled corpora to guide learning processes via objective functions (Yu et al., 2017). Tang et al. propose a method for learning sentiment embeddings by regulating the C&W model, which encodes sentiment information in the continuous word representations (Tang et al., 2015). By exploiting the prior knowledge, Li et al. incorporate the sentiment information to analyze the sentiment label of each word in target and contexts (Li et al., 2017). Maas et al. apply a semi-supervised method to get sentiment information and carry out the maximum likelihood estimation for parameter determination (Maas et al., 2011).
Notwithstanding, the pre-trained word embeddings still have challenges in tackling sentiment analysis tasks, which are concluded as the following two aspects. On the one hand, the semantically similar word may have opposite sentiment polarities. Thus the sentiment polarity identification process has to be dedicatedly designed (Tang et al., 2015) (Li et al., 2017) (Shi et al., 2018). On the other hand, the capturing of sentiment information from low-frequency words is most pronounced. Typically, the low-frequency words can be regarded as the derivation of entity nouns, new terms and some deformation high-frequency words, which also contain significant semantic information. Nevertheless, due to the low frequency, current models are absent of processing their sentiment. The objective of this work is to devise a sentiment word embedding model. Specifically, the issue of parameter setting is deeply studied. Methods for effectively estimating the involving parameters based on Bayesian estimation and maximum likelihood estimation are proposed. For low-frequency word analysis, the Bayesian estimation is applied to determine the co-occurrence probabilities and the sentiment probabilities. This work describes current parameter estimation approaches and the model of GloVe in Section 2, illustrates our sentiment word embedding model in Section 3, shows the experiments in Section 4, and presents the research findings in Section 5.

heading: 2 Preliminary

text: This section introduces the basic theory related to parameter estimating algorithms and the GloVe model, so as to facilitate the description of subsequent model architecture.

heading: 2.1 Parameter Estimating Algorithms

text:
Typically, word vectors are taken as learning variables in the word embeddings, which results in the use of parameter estimating algorithms. The way of establishing objective function is therefore be employed. According to (Tang et al., 2014), the objective function of the Skip-Gram model is to maximize the average log probability, which is expressed as:
J = 1
T T∑ i=1 ∑ −c⩽j⩽c,j ̸=0 ln p (wi+j | ei) (1)
where T is the number of words in the corpus and c indicates the size of window. We take ei as the embedding of target word wi and wi+j as the context of wi. The outcome p (wi+j |ei) is obtained via the hierarchical softmax. Similarly, the objective of GloVe refers to the maximum of likelihood probability and is defined as (Jameel et al., 2019):
J = ∏ i,j N ( lnxij ;wi · w̃j + bi + b̃j , σ2 )
(2)
where N ( .;µ, σ2 ) represents the normal distribution with the mean µ and the variance σ2.
In GloVe, the variance is determined by each word couple (i, j). In addition to objective function constructing, the estimation algorithms are applied to compute other parameters within word embeddings. In (Maas et al., 2011), the maximum posterior probability estimation identifies the parameter to weigh the semantic information (Maas et al., 2011). In D-GloVe, Jameel and Schockaert also use the semantic information weighing parameter, whose value corresponds to the Bayesian estimating outcome (Jameel and Schockaert, 2016).

heading: 2.2 The GloVe model

text:
Basically, the GloVe model is a word-embedding method that combines evidence from the local context and the global counts. Typically, three distinguished words are used in this
model, which are Wi,Wj and Wk. Both Wi and Wj are target words while Wk is the context. Let x be the matrix representing the word-word co-occurrence counts. We define the element xik as the times for word Wk appearing in the context of Wi. Correspondingly, xi = ∑ k xik indicates the frequency of each word occurs in the context of Wi. The cooccurrence probability of Wk being the context word of Wi is given as
Pik = P (Wk|Wi) = xik/xi (3)
The parameter Pik/Pjk is taken to determine the relation of Wi to Wk and Wj to Wk. For Wk has a similar relation to Wi and Wj , i.e. both relevant or irrelevant, the ratio approaches 1. The information in the ratio of co-occurrence probabilities is:
F ( wTi w̃k − wTj w̃k ) = Pik/Pjk (4)
where wϵRn refers to the target word vector and w̃ϵRn to the context vector. Commonly, GloVe extracts the semantic relation between different words by using the ratio of cooccurrence probabilities while the semantic information are identified via the maximum likelihood estimation (Maas et al., 2011).

heading: 3 Methodology

text: This section depicts the architecture of the sentiment word embedding, working principle of the parameter estimating process using two different estimation algorithms.

heading: 3.1 Sentiment Word Embedding Model Architecture

text:
In sentiment analysis tasks, the sentiment information is captured during processing. Aiming to identify the sentiment polarities of different words, a word embedding model, incorporating the sentiment information, is established. Typically, we tend to characterize the proposed model by the loss function. To compute the sentiment embeddings, we define the probability of Wi being positive as Bi and negative as 1−Bi. Assuming that Wi = good and Wj = bad, the value of Bi/Bj is larger than 1, which indicates good is more positive than bad. In turn, the value (1−Bi) / (1−Bj) is less than 1 since bad shows a negative polarity. In this way, the relations of the word sentiment are expressed by the ratio of sentiment probabilities. For Bi+(1−Bi) = 1, Bi/Bj and (1−Bi) / (1−Bj) make the same sense in conveying the sentiment, we take Bi/Bj to construct the sentiment relation of Wi and Wj . More details of the words’ relation and the ratios are presented in Appendix 1. Considering the bias vector corresponds to positive sentiment polarity, we take sϵRn to indicate the bias vector to match the size of word vector. By transforming Wi and Wj into word vectors wi and wj , the difference established upon si and sj is written as:
F (wTi si − wTj sj) = Bi/Bj (5)
Assuming that F is confirming to the homomorphisms between groups (R,+) and (R>0,×), the semantic and sentiment information is combined to get:
F (( wTi w̃k − wTj w̃k ) + ( wTi si − wTj sj )) =F ( wTi w̃k − wTj w̃k ) · F ( wTi si − wTj sj
) = Pik Pjk · Bi Bj
(6)
Due to properties of group homomorphism, eqn.6 is transformed into
F ( wTi w̃k − wTj w̃k + wTi si − wTj sj ) =F (( wTi w̃k + w T i si ) − ( wTj w̃k + w T j sj ))
= F ( wTi w̃k + w T i si )
F ( wTj w̃k + w T j sj ) = Pik Pjk · Bi Bj
(7)
in line with
F ( wTi w̃k + w T i si ) = Pik ·Bi (8)
According to eqn.7,the basic objective function F is in the form of exponential, that is F (x) = exp (x). Thus, we apply the logarithm operation to each side and have:
wTi w̃k + w T i si = ln (Pik ·Bi) = lnPik + lnBi (9)
By incorporating the sentiment information, the loss function of the word embedding model is defined as
loss (wi, w̃k, si) = V∑ i,k=1 [ wTi w̃k + w T i si − lnPik − lnBi ]2 (10) where V indicates the size of the vocabulary. The parameters wTi , w̃k and si are computed via gradient descent algorithms.

heading: 3.2 Incorporating Sentiment Information

text:
As pointed out in the Introduction, current models use the maximum likelihood estimating algorithm for parameter determination. In this part, we preliminarily carry out the parameter estimation based on the maximum likelihood principle. For each target word Wi, xi times Bernoulli experiments are conducted to extract the context independently with V different outcomes in each experiment (Djuric and Huang, 2000). The occurrence number of the kth outcome and its probability are represented by xik and Pik. If the random variable Xi = (Xi1, Xi2, · · ·, XiV ) stands for the occurrence times of all the possibilities, i.e. Xik is the number for the kth one, the parameter Xi obeys the Multinomial distribution, i.e. Xi ∼ Multinomial
(−→xi ,−→Pi) with −→Pi = (Pi1, Pi2, · · · , PiV ) and −→xi = (xi1, xi2, · · · , xiV ). Hence, a log-likelihood function is constructed:
 max Pi1,Pi2,··· ,Pik,··· ,PiV lnL(Pi1, Pi2, · · · , Pik, · · · , PiV ) = max Pi1,Pi2,··· ,Pik,··· ,PiV ln [(Pi1)xi1 · (Pi2)xi2 · · · (Pik)xik · · · (PiV )xiV ] = max Pi1,Pi2,··· ,Pik,··· ,PiV V∑ k=1 xik · lnPik s.t. V∑ k=1 Pik = 1
(11)
According to eqn.11, the objective function can be resolved as an optimal problem that equality constraints. Thus, the corresponding Lagrangian function is formulated as
J (Pi1, Pi2, · · · , PiV , λ) = V∑
k=1
xik · logPik + λ ( 1−
V∑ k=1 Pik
) (12)
where we have Pik = xikλ determined by ∂J(Pi1,Pi2,··· ,PiV ,λ) ∂Pik = xikPik − λ = 0. Likewise, λ = ∑V k=1 xik = xi is calculated with respect to ∑V k=1 Pik = ∑V k=1 ( xik λ ) = ∑V
k=1 xik/λ = 1. Thus, the estimation of Pik is written as
P̂ik = xik/xi (13)
Notably, the obtained Pik is the same with that from GloVe according to eqn.3, which demonstrates the feasibility for parameter estimation in our model. In this way, the outcome of parameter sentiment probability can also be computed by using the maximum likelihood estimator. As such, a maximum likelihood estimation-based sentiment word embedding, namely MLESWE, is put forward. In this case, the Bernoulli experiments are applied to pick up the sentiment polarity of the target word Wi and the outcome can be either positive or negative. Since Bi is the probability of Wi being positive, we designate the distribution of Wi obeys−→ ti = (ti0, ti1) where ti0 is the number of negative texts and ti1 indicates that of the positive ones. Thus, the total number of texts including Wi is expressed as ti = ti0 + ti1. Support a random variable Ti = (Ti0, Ti1) denotes the times of all the possibilities of outcomes and Ti conforms to the binomial distribution, i.e. Ti ∼ Binomial (−→ ti , −→ Bi ) where −→Bi = (Bi, 1−Bi).
The log-likelihood function of sentiment probabilities is delivered as:
max Bi lnL (Bi) = max Bi
ln [ (Bi) ti1 · (1−Bi)ti0 ] = max
Bi [ti1 · lnBi + ti0 · ln (1−Bi)] (14)
Similarly, B̂i = ti1ti is obtained based on ∂(ln L) ∂Bi = ti1Bi − ti0 1−Bi = 0. Combining the semantic and sentiment information, the final loss function based on maximum likelihood principle is
loss (wi, w̃k, si) = V∑ i,k=1 [ wTi w̃k + w T i si − ln xik xi − ln ti1 ti ]2 (15)

heading: 3.3 Parameter Estimating using Bayesian Estimation

text:
The Bayesian estimating method is highlighted due to its not sensitive to initialization via proper prior distributions to parameters (Ma et al., 2018). By using the prior knowledge, the deficiency of lacking information of small datasets can be resolved, which leads to the converge to the actual value (Phoong and Ismail, 2015). Accordingly, the generalization ability of the model can be improved (Wu et al., 2018). The Bayesian approach, in this way, is able to present an elegant solution for automatically determining the parameters (Ferguson, 1973). We thus employ the Bayesian estimation for the parameter estimating of the proposed model. The Bayesian estimation-based sentiment word embedding, namely BESWE, is performed. In accordance to the assumption of maximum likelihood principle mentioned before, the prior distribution P (−→ Pi ) is assumed to obey the Dirichlet distribution of −→α = (αi, αi, · · · , αV ).
The prior distribution is converted to P (−→ Pi ) = Dir (−→α ) = Γ( ∑ k αk)∏
k Γ(αk)
∏ k P αk−1 ik , with the
identical likelihood function:
P (−→x i|−→P i) = Mult(−→x i,−→P i) = xi!∏V
k (xik!) · V∏ k P xikik (16)
Considering the Dirichlet-Multinomial conjugate structure, the posterior distribution is
P (−→ P i|−→x i ) = Dir (−→α +−→x i) = Γ ( ∑
k αk + xik)∏ k Γ (αk + xik) · ∏ k Pαk+xik−1ik (17)
where αk = λ1 · nk∑ k nk
, nk is the total number of occurrences of word Wk in the corpus and λ1 > 0 is determined by tuning data. By satisfying
cik = EP (−→ P i|−→x i ) [lnPik] (18) we compute the Bayesian estimating outcome of lnPik in the loss function provided by eqn.10, which is also the mean value of posterior probability in line with Pik. As stated in (Jameel and Schockaert, 2016), the computation of E
P (−→ P i|−→x i ) [lnPik] is facilitated via Taylor expansion:
E P (−→ P i|−→x i ) [lnPik] ≈ lnEP(−→P i|−→x i) [Pik]− V ar
P (−→ P i|−→x i ) [Pik] 2 · E2
P (−→ P i|−→x i ) [Pik] (19)
where we have V ar P (−→ P i|−→x i ) [Pik] = αk+xik∑ k(αk+xik)
· ( 1− αk+xik∑
k(αk+xik)
) · 1∑
k(αk+xik)+1 and
E P (−→ P i|−→x i ) [Pik] = αk+xik∑ k(αk+xik)
. Note that lnPik is estimated via Bayesian principle in eqn.18 whose form is unlike that of eqn.13. Comparing to the maximum likelihood estimation, a direct outcome is obtained without using Laplace smoothing in experiment. Comparatively, P (−→ Bi ) is designed to obey Beta distribution with the parameter −→β = (β0, β1), along with
the prior distribution given as
P (−→ B i ) = Beta (−→ β ) = Γ (β0 + β1)
Γ (β0) Γ (β1) (1−Bi)β0−1 ·Bβ1−1i (20)
from which the log-likelihood function is
P (−→ t i| −→ B i ) = b (−→ t i, −→ B i ) = Cti1ti · (1−Bi) ti0 ·Bti1i (21)
and the posterior distribution subject to the conjugate structure of Beta-Binomial is
P (−→ B i| −→ t i ) = Beta (−→ β + −→ t i ) = Γ (β0 + ti0 + β1 + ti1)
Γ (β0 + ti0) Γ (β1 + ti1) · (1−Bi)β0+ti0−1 ·Bβ1+ti1−1i
(22)
where mk stands for the texts of the sentiment label k, λ2 > 0 is a parameter depending on tuning data and βk = λ2 · mk∑
k mk . Thereupon, to determine the lnBi in eqn.10, we take
the Bayesian estimation approach. The solution to the posterior probability expectation of lnBi, which is involved with Bi is characterized as
ei = EP (−→ B i| −→ t i ) [lnBi] (23) Furthermore, the Taylor expansion is employed to update the equation:
E P (−→ B i| −→ t i ) [lnBi] ≈ lnEP(−→B i|−→t i) [Bi]− V ar
P (−→ B i| −→ t i ) [Bi] 2 · E2
P (−→ B i| −→ t i ) [Bi] (24)
where we have V ar P (−→ B i| −→ t i ) [Bi] = β1+ti1∑ k(βk+tik)
· ( 1− β1+ti1∑
k(βk+tik)
) · 1∑
k(βk+tik)+1 and
E P (−→ B i| −→ t i ) [Bi] = β1+ti1∑ k(βk+tik) . Hence, the final loss function of BESWE can be obtained:
loss (wi, w̃k, si) = V∑ i,k=1 [ wTi w̃k + w T i si − cik − ei ]2 (25)

heading: 4 Experiments

text: In this section, the working performance of BESWE and MLESWE are evaluated. The task of word similarity analysis is carried out. To deliver the sentiment embeddings, both wordand sentence-level sentiment analysis using different models is conducted.

heading: 4.1 Experiment Settings

text:
Datasets. The dataset SST (Stanford Sentiment Tree) is employed for the mode training. There are five classes annotations within SST, which are very negative, negative, neutral, positive and very positive. Typically, we assign the value 3 and 4 to represent the positive polarity, 0 and 1 to negative and 2 to else. For our models, the word representation dimension is 50, the learning rate is 0.05 and the iteration number is 50. Besides, the loss function is optimized with the deployment of AdaGrad. Baseline Methods. We evaluate the proposed model in comparison to other state-of-theart models. The models of word embeddings, such as C&W, word2vec and GloVe, together with models of sentiment embeddings, including SE-HyRank and DLJT2, are implemented. For the baseline methods, we use default settings in the provided implementations or described as their papers and the word representation dimension is 50. Word Similarity. Computing word similarity (Levy et al., 2015) aims to capture the general meanings of words. In this research, the word similarity tasks are conducted on the dataset EN-WS-353-ALL, EN-WS-353-SIM and SCWS, which are detailed illustrated in (Jameel et al., 2019). Word-level Sentiment Analysis. We conduct word-level sentiment analysis on two sentiment lexicons, namely MPQA and NRC. The number of positive and negative items for MPQA is 2301 and 4151 while for NRC is 2231 and 3324. The N-fold cross validation with N=5 and N=10 is performed. An SVM classifier is trained whose average accuracy is the evaluation metric. Specifically, the words from SST corpus are extracted and converted into word embeddings, which are taken as the features of SVM.
As Bayesian estimating principle is capable of tackling low-frequency words, we distinctively pick up the words with a frequency less than 5 for analysis. Statistically, the SST corpus contains 9984 low-frequency words. Sentence-level Sentiment Analysis. Considering the sentiment analysis for sentence, the movie review polarity datasets MovieReview is employed (Pang and Lee, 2005), which contains 10622 samples with the proportion of each polarity 1:1. We use a convolutional neural network (CNN) model, namely Text-CNN, with its online implementation (Kim, 2014). Likewise, the inputs of Text-CNN are word embeddings as well. The training episode is set as 200 epochs using the default settings. Similarly, we also pick the low-frequency words with the occupation over 10% as the lowfrequency sentences for testing. There are totally 1258 sentences cater to the demands and are all sent to Text-CNN classifier for processing.

heading: 4.2 Experimental Results

text:
Word Similarity. On the task of working performance evaluation, we first present the results for of word similarity analysis (Fig. 1). It can be observed BESWE outperforms other algorithms on all datasets, indicating that our model is capable to capture sufficient semantic information. Distinctively, the implementation of MLESWE, although not as good as BESWE, still achieves a better result on the average accuracy (i.e. Ave_Acc in Fig. 1) than the original GloVe. Yet the maximum likelihood estimating algorithm can also be applied to parameter determination of the word embeddings.
Word-level Sentiment Analysis Results. The word-level sentiment analysis task is conducted on the dataset of single-word entries. The DLJT2 model outperforms other word embedding models by incorporating sentiment information into the learning processes, as shown in Fig.2. Compared to the state-of-the-arts, our model fails to exceed the outcome of the best method on average accuracy. Encouragingly, the BESWE model shows an even better performance in tackling the low-frequency words.
Sentence-level Sentiment Analysis Results. The working performance of the proposed model is further evaluated on the sentence-level sentiment analysis task. From Fig.3, we see that SE-HyRank has a better performance than any other algorithms in average accuracy. Clearly, the outcome of BESWE is anyhow decent which is comparable with that of SEHyRank. Regarding low-frequency sentences, the minimum performance gap of over 9% against SE-HyRank is reported. Consequently, for the sentiment analysis of low-frequency words or low-frequency sentences, BESWE always obtain the best and most consistent results in the identification of sentiment polarity.
Effects of λ1 and λ2. The hyperparameters in BESWE, i.e. regulatory factors λ1 and λ2, are used to represent the semantic and the sentiment information. The settings of the involving parameter can be therefore determined. The values of λ1 and λ2 are varied within the collection of {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01}. Firstly, the value of λ1 is set as {1, 0.75, 0.5, 0.25, 0.1, 0.05, 0.02, 0.01}. When λ2 = 1, we name BESWE as BESWE#1 and λ2 = 0.75 as BESWE#2, and so on so forth. Correspondingly, the value of λ2 is also picked from the same set and named from BESWE#9 to BESWE#16 in the same order. Totally, we get 16 different models. The results on the sentence-level sentiment analysis against different hyperparameter settings are shown in Fig.4(a) and Fig.4(c). Likewise, the results for low-frequency sentences are in Fig.4(b) and Fig.4(d). We take LowFreSentence#n to nominate the outcomes from low-frequency sentences.
The sentence-level sentiment analysis reaches the highest accuracy 71.64% at the point λ1 = 0.05 and λ2 = 0.02. For the analysis of low-frequency sentences, the optimal values of λ1 and λ2 are 0.1 and 0.01, which results in an accuracy of 79.92%.
The experimental results verify the effectiveness of the proposed sentiment word embedding. The BESWE model outperforms other state-of-the-art in word similarity identification. In the sentiment analysis of both word level and sentence level, our method still presents comparable outcomes. Specifically, by integrating the prior knowledge into sentiment probabilities estimating, the BESWE model is a better alternative for low-frequency-word sentiment capturing. It is reasonable to expect better performance in sentiment analysis tasks, as it is the case.

heading: 5 Conclusion

text: In this work, the designing and deploying of the sentiment word embeddings is deeply studied. On the foundation of current word embedding models, the estimation principle of the objective function, together with other parameters, are investigated. Motivated by the significance of sentiment information, a novel word embedding model for sentiment analysis is established. Within the proposed model, both semantic and sentiment information is integrated into the word vectors. Aiming to construct the objective function, the group homomorphism theory is applied. As for the parameter determination, the maximum likelihood estimator and the Bayesian estimator are employed. Experiments are conducted on various tasks to evaluate the working performance. In comparison to the baseline models, our model is capable of tackling word similarity tasks. For the purpose of sentiment embeddings representation, the proposed model is effective in word-level and sentence-level sentiment analysis. Specifically, it outperforms other methods on low-frequency words and sentences sentiment polarity identification to demonstrate its efficacy.

heading: A Appendix I

text:
For Wi = good and Wj = bad, we have Bi⁄Bj > 1, (1−Bi) / (1−Bj) < 1,
i.e. Wi = good is more positive than Wj = bad. For Wi = good and Wj = great, we have Bi/Bj ≈ 1,
(1−Bi) / (1−Bj) ≈ 1, i.e. Wi = good and Wj = great are of positive polarities.
For Wi = then and Wj = home, we have Bi/Bj ≈ 1,
(1−Bi) / (1−Bj) ≈ 1, i.e. Wi = then and Wj = home are of neutral polarities.
The specific sentiment probabilities calculated by maximum likelihood estimation are presented in Table 1.
Likewise, the outcomes based on Bayesian estimation are in Table 2.

title: A Novel Bayesian Estimation-Based Word Embedding Model For Sentiment Analysis