Linguistic features have shown promising applications for detecting various cognitive impairments. To improve detection accuracies, increasing the amount of data or the number of linguistic features have been two applicable approaches. However, acquiring additional clinical data can be expensive, and handcrafting features is burdensome. In this paper, we take a third approach, proposing Consensus Networks (CNs), a framework to classify after reaching agreements between modalities. We divide linguistic features into non-overlapping subsets according to their modalities, and let neural networks learn low-dimensional representations that agree with each other. These representations are passed into a classifier network. All neural networks are optimized iteratively. In this paper, we also present two methods that improve the performance of CNs. We then present ablation studies to illustrate the effectiveness of modality division. To understand further what happens in CNs, we visualize the representations during training. Overall, using all of the 413 linguistic features, our models significantly outperform traditional classifiers, which are used by the state-of-the-art papers. Alzheimer's disease (AD) and its usual precursor, mild cognitive impairment (MCI), are prevalent neurodegerative conditions that inhibit cognitive abilities. Cognitive impairments are traditionally diagnosed only with standard clinical tests like MoCA (Nasreddine et al., 2005) and the Rey-Auditory Verbal learning Test (Rey, 1941), but hiring clinicians to administer these tests and analyze their results is costly. Fortunately, many cognitive impairments can be observable in daily life, because they impact one's language abilities. For example, cognitively impaired people tend to use more pronouns instead of nouns, and pause more often between sentences in narrative speech (Roark et al., 2011). This insight makes automatic detection possible. Machine learning classifiers can detect cognitive impairments given descriptive linguistic features. In recent work, linguistic features including pronoun-noun-ratios, pauses, and so on, are used to train classifiers to detect cognitive diseases in various tasks. For example, Fraser et al. (2015) achieved up to 82% accuracy on DementiaBank 1 , the largest publicly available dataset on detecting cognitive impairments from speech, and Weissenbacher et al. (2016) achieved up to 86% accuracy on a corpus of 500 subjects. Yancheva et al. (2015) estimated Mini-Mental State Estimation scores (MMSEs), describing the cognitive status and characterizing the extent of cognitive impairment. To improve the accuracy of automated assessment using engineered linguistic features, there are usually two approaches: incorporating more clinical data or calculating more features. Taking the first approach, Noorian et al. (2017) incorporated normative data from Talk2Me 2 and the Wisconsin Longitudinal Study (Herd et al., 2014) in addition to DementiaBank, which increased AD:control accuracy up to 93%, and mod-erateAD:mildAD:control three-way classification accuracy to 70%. Taking the second approach, Yancheva and Rudzicz (2016) used 12 features derived from vector space models and reached a .80 F-score on DementiaBank. Santos et al. (2017) calculated features depicting characteristics of co-occurrence graphs of narrative transcripts (e.g., the degree of each vertex in the graph). Their classifiers reached 65% accuracy on DementiaBank 1 https://talkbank.org/DementiaBank 2 https://www.cs.toronto.edu/talk2me/ (MCI versus a subset of Control). There are limitations in either of the two approaches. On one hand, acquiring additional clinical data can be expensive (Berndt and Cockburn, 2013). Moreover, the additional data should be similar enough to existing training data to be helpful. On the other hand, crafting new features requires creativity and collaboration with subject matter experts, and the implementation can be time consuming. Neither of these approaches is satisfactory. These limitations motivate us to take a third, novel approach. Instead of using new data or computing new features, we use the existing linguistic features. If the speaker is cognitively impaired, and their language ability is affected, features from each of the acoustic, syntactic, and semantic modalities should reflect such change (Szatloczki et al., 2015; Moro et al., 2015; Fraser et al., 2015). We therefore need to distill the common information revealed by features from multiple, mainly distinct, modalities. To utilize information common across different modalities, Becker and Hinton (1992) and de Sa (1994) let classifiers look at each modality and supervise each other. These examples illustrated the effectiveness of multi-view learning in utilizing common information among different observations, but their algorithms fail to train useful classifiers for cognitive impairments in our datasets. Without explicit supervision, self-supervised models almost always converge to a state producing the same predictions for all people, giving trivial classifiers. Instead of aligning the predictions from modalities, we let the representations of the modalities agree. Generative adversarial networks (GANs) provide an approach. In GANs, a discriminator network is trained to tell whether a vector is drawn from the real world or produced synthetically by a generator neural network, while the generator is trained to synthesize images as close to real data as possible. We borrow this setting, and encourage the neural networks interpreting different modalities to produce representations of modalities as similar to each other as possible. This leads to our classifier framework, consensus networks (CNs). Consensus networks constitute a framework using adversarial training to utilize common information among modalities for classification. In this framework, several neural networks (ePhysi-cians) are juxtaposed, each learning the representation of a partition of linguistic features for each transcript. Being trained towards producing agreed representations, we show they are increasingly able to capture common information contained across disparate subsets of linguistic features. We empirically add two extensions to CN that improve the classification accuracies, called the noise modality and cooperative optimization respectively, as explained below. To illustrate the effectiveness of the consensus mechanisms, we present two ablation studies. First, we compare neural networks built with consensus (CN) and those without (MLP). On partial or complete modalities, CN outperforms MLP significantly. Second, we compare CNs built with linguistic features divided into random subsets. Division according to their natural modalities train better consensus networks. We also visualize the representations during training procedure, and show that when the representations agree, their distributions appear symmetric. Overall, taking all 413 linguistic features, our models significantly outperform traditional classifiers (e.g., support vector machines, quadratic discriminant analysis, random forest, Gaussian pro-cess), which are used by the state-of-the-art. Generative Adversarial Networks The idea of aligning representations by making them indistinguishable is inspired by GAN (Goodfellow et al., 2014), where a generator produces fake images (or other data) that are as similar to real data as possible. However, our model does not have a generator component as GANs do. Instead, we only compress features into representations while trying to align them. Multi-view Learning Learning from multiple modalities is also referred to as multi-view learning. Becker and Hinton (1992) set up multiple neural networks to look at separate parts of random-dot stereograms of curved surfaces, and urge their prediction to equal each other. The trained neural networks were able to discover depth without prior knowledge about the third dimension. de Sa (1994) divided linguistic features into two modalities, and passed them to two separate neural networks. The two neural networks supervised each other (i.e., output labels that are used to train the other) during alternative optimization steps to reach a consensus. Their self-supervised system reached 79  2% accuracy using the Peterson-Barney vowel recognition dataset (Peterson and Barney, 1952). Benediktsson et al. (1997) computed multiple views from the same feature sets and classified by taking their majority votes. Pou-Prom and Rudzicz (2018) used canonical correlation analysis (CCA) to classify using multiple aspects. Contrary to that work, our consensus networks take in distinct subsets of features as modalities. Co-training (Blum and Mitchell, 1998) and tri-training (Zhou and Li, 2005) use distinct subsets of features, but they use them to train distinct classifiers, and let the results directly supervise each other. Their approach bootstrapped' clas-sifications based on a few labeled data, but our method explicitly uses a modality discriminator that enforces alignments between modalities. Domain Adaptation In domain adaptation and multi-task learning, there have been many attempts to learn indistinguishable embeddings between domains. For example, Ganin et al. (2016) and Joty et al. (2017) applied a gradient reversal layer to let encoders minimize the domain classification loss. Baktashmotlagh et al. (2013) minimized the maximum-mean discrepancy (MMD) loss in a reproductive kernel Hilbert space (RKHS) of the latent representations. Motiian et al. (2017) used semantic similarity loss between latent representations of different class data to encourage alignments between domains. Liu et al. (2017) and Chen and Cardie (2018) used shared and private networks to learn information contained either commonly in domains or domain-specific. Our work is unique. First, there is only one domain in our problem setting. Second, we use iterative optimization to encourage discrepancies between domains. Third, we have two empirical improvements (noise modality and cooperative optimization) that make our Consensus Networks outperform traditional classifiers. We use DementiaBank, the largest publicly available dataset for detecting cognitive impairments. It includes verbal descriptions (and associated transcripts) of the Cookie Theft picture description task from the Boston Diagnostic Aphasia Examination (Becker et al., 1994). The version we have access to contains 240 speech samples labeled Control (from 98 people), 234 with AD (from 148 people), and 43 with MCI (from 19 people) 3 . All participants were older than 44 years. The dataset contains narrative speech descriptions and their transcriptions. We preprocess them by extracting 413 linguistic features for each speech sample. These linguistic features are proposed by and identified as the most indicative of cognitive impairments by various previous works, including Roark et al. (2007), Chae and Nenkova (2009), Roark et al. (2011), Fraser et al. (2015), and Hernandez-Domnguez et al. (2018). After calculating these features, we use KNN imputation to replace the undefined values (resulting from divide-by-zero, for example), and then normalize the features by their z -scores. The following are brief descriptions of these features, grouped by their natural categories. More detailed descriptions are included in the Appendix. There are 185 acoustic features (e.g., average pause time), 117 syntactic features (e.g., Yngve statistics (Yngve, 1960) of the parse tree, computed by the LexParser in CoreNLP (Manning et al., 2014)), and 31 semantic features (e.g., cosine similarity between pairs of utterances) Moreover, we use 80 part-of-speech features that relate to both syntax and semantics but are here primarily associated with the latter. Modality Division After representing each sample with a 413-dimensional vector x consisting of all available linguistic features, we divide the vector into M partitions (modalities') of approximately equal sizes [ x 1 , x 2 , ..., x M ], according to the groups mentioned above. Unless mentioned otherwise (e.g., in the ablation study shuffling 3 The version of DementiaBank dataset we acquired contains a slightly different number of samples from what some previous works used. In Control:AD, Fraser et al. (2015) used 233 Control and 240 AD samples; Yancheva and Rudzicz (2016) had 241 Control and 255 AD samples; Hern  andez-Dom  nguez et al. (2018) had 242 Control and 257 AD samples (with 10% control samples excluded from the evalua-tion). In Control:MCI, Santos et al. (2017) used all 43 transcriptions from MCI and 43 sampled from the Control group. With no clear descriptions of the sampling procedures, the constituents of the Control group might differ from our sample. In this paper, we run our models on the same tasks (i.e., Control:AD) and compare to the results of models used in the literature. modalities), this is our default choice for assigning modalities. Figure 1 is an example of our model structure (with M = 3 modalities), and this section elaborates the inference procedure, the training algorithm, and our two improvements. Inference With the extracted linguistic features divided into subsets by their modalities, each speech sample is described by M = 3 nonoverlapping feature vectors x = [ x 1 , ..., x m ] . These feature vectors are then passed into corresponding ePhysician networks, each outputting a vector i m , which is a distilled representation of the subject from a modal-specific perspective (e.g., the semantic). We also refer to it as the interpretation vector and use them interchangeably. Formally, the m th ePhysician can be written as a function, f m ( . ) generating the representation: i m = f m ( x m ) To challenge the similarity of representations from different modalities, we let a discriminator neural network f D ( . ) take in each of the M representations and predict the likelihood of the originating modality m . where . To attempt a diagnosis, a classifier network f C ( . ) takes in the combination of M representations of each speech sample, and outputs a prediction probability for detection result y: P ( y = l | x ) = e f C ( i 1 .. M ) l (cid:80) l e f C ( i 1 .. M ) l where l  { 0 , 1 } for two-class classification (i.e., 0 for healthy and 1 for dementia). The predicted class corresponds to those with the highest probability:  y = arg max l P ( y = l | x ) Optimization The training procedure optimizes the adversarial objective, and the conventional classifier objective:  The adversarial objective sets up the ePhysicians and the Discriminator to work in an adversarial manners. The ePhysicians try to produce indistinguishable representations, while the discriminator tries to tell them apart. min D max PLD where LD = E x E m =1 ..M { log P (  m = m | i m ) } (1)  Make the classifier network as accurate as possible. This is done by minimizing the cross entropy classification loss: min CLC where LC = E x { log P ( y = y | i 1 .. M ) } (2) Overall, min CLC and min D max PLD set up a complex optimization problem. We use iterative optimization steps, similar to Goodfellow et al. (2014). There are two tricks that we found to improve the performance of the models. Namely, the noise modality and the cooperative optimization . We explain them below. Noise Modality For each participant session, we add a noise modality representation i 0 drawn from a Gaussian distribution with the mean and variance identical to those of other representation vectors. This additional representation vector is passed into the discriminator, but not passed into the classifier. The first optimization goal (1) is therefore: min D max PLD where LD = E x E m =0 ..M { log P (  m = m | i m ) } (3) To some extent, the noise representation vector works like a regularization mechanism to refrain the discriminator from making decisions based on superficial statistics. We show in 4.1 that this addition empirically improves classifier performance. Cooperative Optimization When optimizing the classifier, we find that allowing gradients to propagate back to the ePhysicians improves the model's overall performance. During optimization, the ePhysicians need to cooperate with the classifier (while adversarial to the discriminator). The second optimization goal (2) is therefore: min C,P LC where LC = E x { log P ( y = y | i 1 .. M ) } (4) Implementation As a note of implementation, all ePhysicians, the classifier, and the discriminator networks are fully connected networks with Leaky ReLU activations (Nair and Hinton, 2010) and batch normalization (Ioffe and Szegedy, 2015). The hidden layer sizes are all 10 for all ePhysician networks, and there are no hidden layers for the discriminator or classifier networks. Although modalities might contain slightly different numbers of input dimensions, we do not scale the ePhysician sizes. This choice comes from the intuition that the ePhysicians should extract into the representation as similar information as possible. We use three Adam optimizers (Kingma and Ba, 2014), each corresponding to the minimization of ePhysician, Discriminator, and the Classifier, and optimize iteratively for no more than 100 steps. The optimization is stopped prior to step 100 if the classification loss LC converges (i.e., does not differ from the previous iteration by more than 1  10  4 ) on training set. The train / validation / test set are divided randomly in 60/20/20 proportions. We first show the effectiveness of our two improvements to the model. Next, we do two ablation studies on the arrangements of modalities. Then, we evaluate our model against several traditional supervised learning classifiers used by state-of-the-art works. To understand the model further, we also visualize the principal components of the representation vectors throughout several runs. We compare a CN model with a noise modality to one without (with other hyper parameters including hidden dimensions and learning rates identi-cal). Table 1 shows that in the AD:MCI classification task, the model with an additional noise modality is better than the one without ( p = 0 . 04 on 2-tailed t -test with 18 DoF). Here is a possible explanation. Without adding a noise modality, the discriminators may simply look at the superficial statistics, like the mean and variances of the representations. This strategy tends to neglect the detailed aspects encoded in the representation vectors. Adding in the noise modality penalizes this strategy and trains better discriminators by forcing them to study the details . In the following experiments, all models contain the additional noise modality. mance. We compare Consensus Network classifiers trained with cooperative optimization (i.e., min C,P 1 ..M LC ) to models with the same hyper-parameters but trained non-cooperatively (i.e., min CLC ). As shown in Table 2, the cooperative variant produces higher-score classifiers than the non-cooperative one ( p < 0 . 001 on a 2-tailed t -test with 18 DoF). With the cooperative optimization setting, the ePhysicians are encouraged towards producing representations both indistinguishable (by the discriminator) and beneficial (for the classifier). Although the representations might agree less with each other, they could contain more complementary information, leading to better overall classifier performances. cooperative optimization. In this and the next experiment, we illustrate the effectiveness of our models on different config-urations of modalities in an ablation study. We show that our models work well because of the effectiveness of the consensus between modalities scheme. In this experiment, we compare our Consensus Network models (i.e., with agreements) with fully-connected neural network classifiers (i.e., without agreements) taking the same partial input features. The networks are all simple multiple layer per-ceptrons containing the same total number of neurons as the classifier pipeline' of our models (i.e., ePhysicians plus the classifier) 4 with batch nor-4 For example, for models taking in two modalities, if our malization between hidden layers. A few observations could be made from Table 3: 1. Some features from particular modalities are more expressive than others. For example, acoustic features could be used for building better classifiers than those in the semantic ( p = . 005 for 2-tailed t -test with 18 DoF) or syntactic modality ( p < . 001 for 2-tailed t test with 18 DoF). More specifically, the syntactic features themselves do not tell much. We think this is because the syntactic features are largely based on the contents of the speech, and remain similar across speakers. For example, almost none of the speakers asked questions, giving zero values in occur-rences of corresponding syntactic patterns. 2. Our model is able to utilize multiple modalities better than MLP. For MLP classifiers, combining features from different modalities does not always give better models. The syntactic modality features confuse MLP and drag down the accuracy. However, our models built with the consensus framework are able to utilize the less informative features from additional modalities. In all scenarios using two modalities, our models achieve accuracies higher than neural networks trained on any of the two individual modalities. 3. Given the same combinations of features, letting neural networks produce representation in agreement does improve the accuracy in all four scenarios 5 . This is the second ablation study towards modality arrangement. We show that dividing features into subsets according to their natural modalities (i.e., the categories in which they are defined) is better than doing so randomly. In this experiment, we train CNs on features grouped by either their natural modalities, or randomly divided. For natural groupings, we try to model contain ePhysicians with one layer of 20 hidden neurons, the interpretation vector dimension 10, and classifier 5 neurons, then the benchmarking neural network contains three hidden layers with [20  2, 10  2, 5] neurons. 5 p = 3  10  12 on syntactic+semantic features, p = 0 . 044 on acoustic + semantic, p = 0 . 005 on acoustic + syntactic, and p = 0 . 046 on all modalities. All 18 DoF one-tailed t -tests. Two groups, natural: (a) acoustic + semantic, 216 features; (b) syntactic + PoS, 197 features. Three groups, natural: (a) acoustic, 185 features; (b) semantic and PoS, 111 features; and (c) syntactic, 117 features. This is the default configuration used in other experiments in this paper. For random grouping, we divide the features into almost equal-sized 2/3/4 groups randomly. As shown in Table 4. The two natural modality division methods produce higher accuracies than those produced by any of the random modality division methods. To further understand what happens inside consensus network models during training, we visualize the representation vectors with PCA. Figure 2 consists of the visualizations drawn from an arbitrary trial in training the model. Each representation vector is shown on the figure as a data point, with its color representing its originating modality (including the noise modality). Several common themes could be observed: 1. The clusters are symmetric . Initially the con-figurations of representations largely depend on the initializations of the network parameters. Gradually the representations of the same modality tend to form clusters. Optimizing the ePhysicians towards both targets make they compress modalities into representations which are symmetrical in an aggregate manner. 2. The agreements are simple . The variances explained by the first a few principal components usually increase as the optimizations proceed. When distilling information relevant to detection, the agreement tend to become simple. 3. The agreements are imperfect . As shown in Figure 2, the modal representations do not overlap. Also, the discriminator loss is low (usually at 10  3 when training is done). This confirms that these representations are still easily distinguishable. This may because the modalities inherently have some complementary information, leading to the ePhysicians projecting the modalities differently. 4. The representations are complex . Their shapes do not resemble the noise representations (Gaussian) lying at the center of the three petals. This shows that the representations are not simply Gaussian. 5. The accuracy increases . The accuracy in validation set generally increases as the training proceeds. Note that the distributions of representation vectors are increasingly similar in shape but remains distinct in spatial allocations. This confirms our conjecture that the information about cognitive impairment resides in complicated details instead of superficial statistics, which neural networks could represent. With the previous sets of experiments, we have a best working architecture. We now evaluate it against traditional classifiers, which are used by Note that the results could be different from what they reported, because the feature sets are different. We test several traditional supervised learning benchmark algorithms here: support vector machine (SVM), quadratic discriminant analysis (QDA), random forest (RF), and Gaussian process (GP). For completeness, multiple layer per-ceptrons (MLPs) containing all features as inputs are also mentioned in Table 5. On the binary classification task (healthy control vs. dementia), our model does better than them all. We introduce the consensus network framework, in which neural networks are encouraged to compress various modalities into indistinguishable representations (interpretation vectors'). We show that consensus networks, with the noise modality and cooperative optimization, improve upon traditional neural network baselines given the same features. Specifically, with all 413 linguistic features, our models outperform fully-connected neural networks and other traditional classifiers used by state-of-the-art papers. In the future, the agreement among modalities concept may be applied to design objective functions for training classifiers in various tasks, and from other data sets (for example, education and occupation modalities for the bank marketing prediction task). Furthermore, the mechanisms that represent linguistic features into symmetric spaces should be analyzed within the context of explainable AI.
abstain abstain abstain objective method abstain abstain result method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain objective objective abstain result abstain method abstain abstain abstain abstain method abstain abstain abstain result result method objective abstain method abstain result result other abstain abstain other other other other other other other other method other method other other other other other abstain objective method abstain method abstain method abstain abstain method abstain method abstain abstain method method method abstain abstain method method method method abstain abstain method method abstain other abstain abstain abstain abstain abstain method method abstain method method abstain abstain abstain method method abstain other abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result result abstain abstain