While neural dependency parsers provide state-of-the-art accuracy for several languages, they still rely on large amounts of costly labeled training data. We demonstrate that in the small data regime, where uncertainty around parameter estimation and model prediction matters the most, Bayesian neural modeling is very effective. In order to overcome the computational and statistical costs of the approximate inference step in this framework, we utilize an ecient sampling procedure via stochastic gradient Langevin dynamics to generate samples from the approximated posterior. Moreover, we show that our Bayesian neural parser can be further improved when integrated into a multi-task parsing and POS tagging framework, designed to minimize task interference via an adversarial procedure. When trained and tested on 6 languages with less than 5  training instances, our parser consistently outperforms the strong Bi LSTM baseline (Kiper-wasser and Goldberg, 2016). Compared with the B i AFFINE parser (Dozat et al., 2017) our model achieves an improvement of up to 3% for Vietnamese and Irish, while our multi-task model achieves an improvement of up to 9% across five languages: Farsi, Russian, Turkish, Vietnamese, and Irish. Dependency parsing is essential for many Natural Language Processing (NLP) tasks (Levy and Goldberg, 2014; Angeli et al., 2015; Toutanova et al., 2016; Hadiwinoto and Ng, 2017; Marcheggiani et al., 2017). While earlier work on dependency parsing required careful feature engineering (Mc-Donald et al., 2005b; Koo et al., 2008), this has become less of a concern in recent years with the emergence of deep neural networks (Kiperwasser and Goldberg, 2016; Dozat et al., 2017). Nonetheless, an accurate parser still requires a large amount of labeled data for training, which is costly to obtain, while the lack of data often causes overfitting and poor generalization. Several approaches for parsing in the small data regime have been proposed. These include augmenting input data with pretrained embedding (Dozat et al., 2017; Che et al., 2018), leveraging unannotated data via semi-supervised learning (Corro and Titov, 2018), predicting based on a pool of high probability trees (Niculae et al., 2018; Keith et al., 2018), and transferring annotation or model across languages (Agic et al., 2016; Lacroix et al., 2016; Rasooli and Collins, 2017). Despite the empirical success of these approaches, an inherent problem still holds: The maximum likelihood parameter estimation ( MLE ) in deep neural networks (DNNs) introduces statistical challenges at both estimation (training), due to the risk of overfitting, and at test time as the model ignores the uncertainty around the estimated parameters. When training data is small these challenges are more pronounced. The Bayesian paradigm provides a statistical framework which addresses both challenges by (i) including prior knowledge to guide the learning in the absence of sucient data, and (ii) predicting under the full posterior distribution of model parameters which oers the desired degree of uncertainty by exploring the posterior space during inference. However, this solution comes with a high computational cost, specifically in DNNs, and is often replaced by regularization techniques such as dropout (Srivastava et al., 2014) as well as ensemble learning and prediction averaging (Liu et al., 2018; Che et al., 2018). Bayesian neural networks (BNNs) have attracted some attention (Welling and Teh, 2011; Hernndez-Lobato and Adams, 2015; Li et al., 2016; Gong et al., 2018). Yet, its current application to NLP is limited to language modeling (Gan et al., 2017), and BNNs have not been developed for structured prediction tasks such as dependency parsing. In this paper we aim to close this gap and propose the first BNN for dependency parsing (BNNP). To address the costs of inference step, we apply an ecient sampling procedure via stochastic gradient Langevin dynamics ( SGLD ) (Welling and Teh, 2011). At training, samples from the posterior distribution of the parser parameters are generated via controlled noise injection to the maximum a posteriori ( MAP ) gradient update. The generated samples are then used during the inference step to create multiple viable parses based on which the final dependency parse is generated. Another means of directing a model towards more accurate predictions in the small data regime is via multi-task learning (Caruana, 1997). Such a framework allows models for multiple tasks to reinforce each other towards more accurate joint solutions, which is particularly useful when training data is scarce. We hence present a multi-task framework where our BNNP is integrated with a POS tagger through an adversarial procedure designed to guide the two models towards improved joint solutions. Our experiments with monolingual and delexicalized cross-lingual parsing using the Universal Dependency treebank (Zeman et al., 2017) demonstrate the eectiveness of our approach. Particularly, our BNNP consistently outperforms the single task Bi LSTM baseline (Kiperwasser and Goldberg, 2016), while outperforming the B i AFFINE parser (Dozat et al., 2017) by up to 3% on Vietnamese and Irish. Additionally, our multi-task model achieves an improvement of up to 9% over the B i AFFINE parser for five low-resource languages: Farsi, Russian, Turkish, Vietnamese, and Irish. Our parser extends the graph-based Bi LSTM parser of Kiperwasser and Goldberg (2016). We briefly review their work and its notable extensions, and then discuss our extension of their architecture and the limitations of MLE training. where  ( . ) denotes trainable embedding,  denotes pretrained external embedding, and  denotes concatenation. The context dependent representation of each word is generated through a Bi LSTM ,   =  LSTM  (  1  )   LSTM  (     ) . Here,  and  denote the LSTM s' parameters. The resulting sequence of  -dimensional vectors,  1  , is then used for computing the ARC-SCORE matrix (    ) where cell ARC-SCORE ( ,  ) is computed as,   tanh (  (    )    head specialized   +  (    )    modifier specialized   +  ) , indicating the likelihood of having the arc from   to   . Here,  is (1   ) ,  is (   1) , and  ( . ) is a (    ) matrix used for creating specialized versions of words in  1  . The  -dimensional specialized representation of   as a head or a modifier is generated via multiplication with  (    ) and  (    ) , respectively. Following a first-order dependence assumption between the arcs, the parsing is done via a dynamic programming solution that finds the dependency parse   such that,   =argmax  ( SCORE (  )=  ( , )  ARC-SCORE ( ,  ) ) . Next, given   , for each arc ( ,  )    the LABEL-SCORE ( , ,  ) is computed as: (    tanh (  (    )    head specialized   +  (    )    modifier specialized   +   ) ) [  ] , where  is a dependency relation type,   {   }   =1 ,   is (    ) ,   is (   1) , and  ( . ) is (    ) . The label for each dependency arc ( ,  ) is then chosen by a max operation. Dozat et al. (2017) proposed an extension of the Bi LSTM parser by replacing the non-linear transformation of ARC-SCORE and LABEL-SCORE with a linear transformation ( B i AFFINE ). This was further extended by Che et al. (2018) who utilized contextualized word embeddings (Peters et al., 2018). Both extensions showed success in dependency parsing shared tasks (Zeman et al., 2017, 2018). architecture extends the Bi LSTM model with an additional Bi LSTM layer and input signals. While our architecture is not the core contribution of this paper, we aim to implement our BNNP on a strong architecture. In 5.3 we demonstrate the contribution of these additions to our final results. In our BNNP, each word is represented as,   =  (   )   (   )   (   )   (   ) , where, similar to the B i AFFINE parser,  (   ) is a character-level representation of the word   , generated by a Bi LSTM :  (   ) =  LSTM  (  1 |   | )   LSTM  (  |   | 1 ) . =  LSTM  (  1  )   LSTM  (     ) , where   is the encoding generated by the 1  layer of the Bi LSTM s. Learning For structured prediction, the objective function is to maximize the margin between the gold structure  and the other structures. To achieve this, the cost augmented hinge loss is used, where the fixed margin  is replaced by a cost function, COST (  ,  ) , sensitive to the local mistakes (here, each local mistake has a constant cost of 1 ), and computed as:   =max ( 0 , COST (  ,  )+ SCORE (  ) SCORE (  ) ) For label prediction, hinge loss with  = 1 is used (denoted by   ). We refer to the parser loss as:   =   +   . Beyond MLE Training The point-estimate of DNN parameters is computationally ecient, but ignores the uncertainty around model parameters during learning. This results in an overconfidence over model predictions during the inference phase. A common generic practice to incorporate a degree of uncertainty is to consider an ensemble of models. Indeed, for dependency parsing ensemble learning has shown to improve accuracy (Surdeanu and Manning, 2010; Kuncoro et al., 2016; Che et al., 2018). However, ensembles are computationally demanding due to the large number of participating models. The de-facto approach to overcome this 2 Stacking Bi LSTM s is believed to be helpful. In our case, the addition of a third layer led to overfitting. has been to randomly perturb the structure of the network for each training instance by switching o connections between the nodes, a practice known as dropConnect (Wan et al., 2013), or eliminating the nodes entirely, which is known as dropout (Sri-vastava et al., 2014). Hinton et al. (2012) demonstrated that dropping out a node with probability  from a neural model  at training, and scaling the same node's output at test time (in the fully connected  ), performs very well in practice. While this scaling trick avoids the need for running multiple models at test time, it lacks theoretical guarantees when applied to DNNs. Gal and Ghahramani (2016) showed that dropout can be casted as integrating out the posterior of model parameters, and in this regard is a form of Bayesian approximation. Still, dropout is not as eective when applied to the small data regime (Goodfellow et al., 2016). We hence propose a fully Bayesian approach. The Bayesian paradigm is appealing in its ability to capture uncertainty around parameters, avoid overfitting, and incorporate prior knowledge to compensate for the lack of sucient training data in resource lean settings. Dependency parsing may be a natural candidate for Bayesian modeling since a typical sentence can have multiple viable parses corresponding to dierent grammatical ambiguities, and considering these possibilities have shown to improve predictive accuracy (Niculae et al., 2018; Keith et al., 2018). In the same spirit, Bayesian inference can be interpreted as a statistically grounded means to generate this pool of high quality trees. In detail, Bayesian inference for parsing computes the predictive distribution  (  | ,  ) 3 of the dependency parse tree  for an input sentence  given the training data  by posterior averaging :  (  | ,  ) =   (  | ,  )  (  |  ) , where  (  |  )   (  |  )  (  ) denotes the posterior distribution of the BNNP parameters  given observations  . Here the prior distribution  (  ) is a standard Gaussian  (0 ,  ) , and, by assuming i.i.d training datapoints, the likelihood term 3 When  (  | , ) is replaced by the unnormalized score from the first-order arc-factored model,  (  | ,  ) can be interpreted as the expected score assigned to a dependency tree under the full posterior distribution of model parameters. is  (  |  ) =       (   |  ) where   = (   ,   ) is a training instance. Therefore, ecient generation of posterior samples is the key for applying the Bayesian treatment. Since the posterior is intractable in DNNs, additional machineries such as Markov Chain Monte Carlo ( MCMC ) (Robert and Casella, 2010) are required for sampling. In this paper we explore an ecient class of Stochastic Gradient Markov Chain Monte Carlo ( SG-MCMC ) methods, designed for NNs by adjusting stochastic gradient descent with properly scaled Gaussian noise to generate posterior samples. We first consider the Stochastic Gradient Langevin Dynamics ( SGLD ) (Welling and Teh, 2011) sampler to generate posterior samples. Intuitively, this mechanism guides SGD to explore the posterior distribution rather than finding the maximum a posteriori ( MAP ) solution. More concretely, at each time step  we first compute a stochastic estimate of the gradient for the negative log-posterior distribution on a mini-batch   :       (    log  (   )+ |  | |   |          log  (   |   ) ) , and then update the BNNP parameters by SGD with noise injection,   +1       2     +   ,     (0 ,    ) (2) Theoretically, the distribution of   converges to the true posterior  (  |  ) when   =  ,    and    0 (see Teh et al. (2016) for SGLD convergence rate proofs and analysis). In practice, the learning rate   =  (  +  )   , starting high and decaying, plays a crucial role in eqn. 2 dynamics. Intuitively, while the variance of the injected noise is   , the variance of the gradient is of  (  2  ) . In the earlier stages of the optimization, the   2    term is dominant as gradients are of greater impact and the learning rate is high, simulating stochastic gradient descent. As the optimization proceeds, the impact of the gradient shrinks and the injected Gaussian noise becomes the dominant term, transitioning from SGD to Langevin Monte Carlo (Neal, 2011). Since    0 might cause slow mixing in practice, we also set a minimum threshold for the learning rate to allow mixing at later stages. SGLD and SGD have the same time complexity as sampling occurs along with parameter updates with a negligible overhead of drawing   s. As SGLD relies on a single learning rate along all dimensions of  , it has the same potential ine-ciencies of SGD for optimizing functions with different curvatures along each dimension. Inspired by more advanced optimization techniques that adjust to the geometry of parameter space, Li et al. (2016) proposed preconditioning of SGLD (similar to RMSprop (Tieleman and Hinton, 2012)) which scales the gradients with respect to the weighted average of the gradients along each dimension, such that a unified learning rate is sucient. The proposed preconditioner diagonal matrix is computed as follows,  (   ) = diag (1  (  1 +   (   ))) ,  (   ) =  (   1 ) + (1   )         , where ,  are element-wise matrix product and division, and ,  are hyperparameters controlling the extremes of the curvature, and weighting average of previous and current gradients, respectively. The updated preconditioned eqn. 2 is,   +1       2  (   )    +   ,     (0 ,    (   )) . Here, replacing the preconditioner matrix  (   ) with an identity matrix will recover the SGLD update formulation. In case of a diagonal preconditioner, it has the same time complexity of SGLD , although the constant factors in the complexity figure can be slightly higher due to additional computations involved in  (   ) and  (   ) . While SGLD is designed to explore dierent modes of the posterior distribution, we found (5.4) that in practice we still require dropout to facilitate modes exploration via random perturbation of the model structure. The generated posterior samples can be used to compute the approximate expectation of eqn. 1 by scoring all plausible parses via explicit model averaging. As this solution is not computationally feasible, we use the sampled parameters and follow a procedure that minimizes the Bayes risk ( MBR ) (Goodman, 1996). Given each sampled parameter, first we generate the maximum scoring parse using the arc-factored decomposition (McDonald et al., 2005a) and dynamic programming (Eisner, 1996). This can be done concurrently for all samples, resulting in a running time identical to the non-Bayesian approach. For each labelled edge, we replace its score in the ARC-SCORE matrix with its occurrence count in the collection of sampled trees and infer the final tree using counts as scores. The predicted structure is then passed to the label predictor, which assigns labels to the edges (2). This decoding approach, while selecting the global structure with the highest probability under the approximate posterior, could potentially allow for additional corrections of the highest scoring tree in the pool of samples (Shareghi et al., 2015; Kuncoro et al., 2016). Multi-task learning (Caruana, 1997) lends itself as a natural choice for low-resource settings as it aims at leveraging the commonality between tasks to improve their performance in the absence of sucient amount of training data. This framework hence naturally complements Bayesian modeling in dealing with the challenges of the small data regime. We couple our BNNP with POS tagging due to the strong connection between the two tasks (Rush et al., 2010) and the availability of joint training data in several languages (Zeman et al., 2017). While multi-task frameworks have shown success in some areas (Reichart et al., 2008; Finkel and Manning, 2009; Liu et al., 2016; Malca and Reichart, 2018), in our case we found that our two tasks interfered with each other and degraded the parser performance (see similar findings for other tasks at Sgaard and Goldberg (2016); Plank and Alonso (2017)). To minimize task interference, an approach shown eective (Ganin and Lempitsky, 2015; Kim et al., 2017; Chen et al., 2017; ZareMoodi and Haf-fari, 2018) is to implicitly guide the update signals during training via an adversarial procedure that avoids shared parameters contamination. We adapt this idea to our multi-task learning. Architecture Given an input word,   , the score for a POS tag  is computed as tanh (     +   )[  ] , where   {   }   =1 ,   is (    ) , and   is (  1) . To train the POS tagger, the cross-entropy loss is  Character BiLSTM Input Representation 1 st BiLSTM 2 nd BiLSTM i s s i  Shared BiLSTM D i sc . used (denoted by   ). Taking   s as input, the shared Bi LSTM strictly encodes a task-agnostic  dimensional representation of the input (explained in the next subsection),   =  LSTM  (  1  )   LSTM  (     ) . The input to each task is then defined as the concatenation of task-agnostic and task-specific representations. We considered a basic architecture where both tasks have their separate parameters (identical number of layers, dimensions, etc.) and they only share the shared Bi LSTM (denoted as MULTI TASK in Table 1). 4 Adversarial Training The shared Bi LSTM output,   , is meant to encode a task-agnostic representation of the input word   . In order to enforce this criterion, we apply an adversarial training procedure. The shared representation,   , is forwarded to a task discriminator (denoted as Disc. in Figure 1) through a gradient reversal layer. The task discriminator predicts the task identity for each word in the input via a linear transformation (     +   )[  ] followed by a softmax, where   { parser,tagger } and   and   are (2   ) and (2  1) , respectively. To train the discriminator, a sum of the cross-entropy losses for  1  is used (denoted by   ). As the parameters of the discriminators are being updated, the gradient signals to minimize the discriminator's error are backpropagated with an opposite sign to the shared Bi LSTM layer, which adversarially encourages the shared Bi LSTM to fool the discriminator. Our training schedule alternates between the two modes, in one mode optimizing the shared and task-specific parameters based on   and   (in 4 We also tried layer-wise placements of tasks (Sgaard and Goldberg, 2016; Hashimoto et al., 2017) and the results were slightly worse. Details are omitted for space reason. We experiment with mono-lingual and cross-lingual dependency parsing using the treebanks of the CoNLL 2017 shared task on parsing to Universal Dependencies (UD) (Zeman et al., 2017). 5 5.1 Experimental Setup We use the UDPipe baseline outputs for segmentation and POS tagging of the raw test data (released along with the raw test data). While segmentation and POS errors substantially impact the quality of the final predicted parse, their exploration is beyond our scope. Our evaluation metric is Labeled Attachment Score (LAS), computed by the shared task evaluation script. Statistical significance, when mentioned, is computed over 20 runs, via the Kolmogorov-Smirnov test (Reimers and Gurevych, 2017; Dror et al., 2018) with  = 0 . 01 . Mono-Lingual Experiments We experiment with Persian (fa), Korean (ko), Russian (ru), Turkish (tr), Vietnamese (vi) and Irish (ga), all with less than 5  training sentences (Table 1). For comparison we report the scores published by the top system of the CoNLL 2017 shared task, B i AFFINE (Dozat et al., 2017), noting the following dierences between their input and output and ours. The B i AFFINE parser: (i) uses the UDPipe outputs for segmentation but corrects POS errors before parsing, (ii) includes both language specific and universal POS tags in the input layer while we only include the universal POS tags, and (iii) applies post-process correction for non-projective languages. Cross-Lingual Experiments We use the English (en), French (fr), Russian (ru), and Persian (fa) datasets of the UD treebanks as our training and test data, with the addition of 3 languages for which we did not have any training data: Kurmanji (kmr), Buriat (bxr), and Northern Sami (sme). We also report the results for each language, where the combination of training datasets for the rest of the languages (marked as + ) was used for training. The cross-lingual experiments are done on delexicalized 5 For train and dev sets (1-1983), test set (1-2184), and pretrained embeddings (1-1989) see: https: //lindat.mff.cuni.cz/repository/xmlui/handle/11234/{1-1983,1-2184,1-1989} parses after replacing the words with their Universal POS tags. Models and Baselines Single-Task We consider the following models: BASE is the Bi LSTM model of Kiperwasser and Goldberg (2016); BASE ++ extends BASE by having 2 layers of Bi LSTM s and using 1 layer of character level Bi LSTM (2); + SHARED includes an additional Bi LSTM (dashed box in Figure 1). We included this to provide a fair comparison (in terms of the number of parameters) with the multi-task experiments but we apply a higher dropout rate to resolve overfitting; ENSEMBLE denotes a collection of 9 + SHARED models each randomly initialized (Reimers and Gurevych, 2017) and trained for MLE with MBR (3.3) applied for prediction; MAP denotes the + SHARED model optimized for MAP instead of MLE ; + SGLD denotes Bayesian learning and prediction (3.1), and + PRECOND denotes preconditioned SGLD (3.2). + SGLD and + PRECOND are applied to the + SHARED model. Models and Baselines Multi-Task We consider the same models as in the single-task setup, with the following changes. BASEMT is the variant of MULTI TASK architecture where the shared Bi LSTM is removed and all components except for decoders are shared between the two tasks. + SHARED and + ADV denote the results without and with the adversarial training (4), respectively. Training is done for 330 epochs with early stopping, using successive 6 mini-batches of 5000 words, 7 with drop-out rate 0 . 33 unless stated otherwise. SGLD is only applied for learning the parameters of the parser, with learning rate decaying from 0 . 01 to 0 . 0001 (  = 0 . 5 ), and the preconditioned matrix hyperparameters are  = 10 5 ,  = 0 . 99 . The rest of the parameters are updated using Adam (Kingma and Ba, 2014) with default DyNet parameters (Neu-big et al., 2017). The size and selection of the samples used in Bayesian inference are tuned on Irish (see 6.1 of the main paper). In all experiments, training stops based on loss convergence of a single model on the dev set. We also tried GRU s and our results were substantially worse than LSTM s. units, 100 -D word embeddings, 25 -D POS embeddings, and dropout rate of 0 . 33 ; BASE ++ extends BASE by having 2 layers of Bi LSTM s with 200 -D hidden units, 100 -D external word embeddings, and using 1 layer of character level Bi LSTM with 200 D hidden units to generate 100 -D character embeddings (2); + SHARED includes an additional Bi LSTM with 200 -D hidden units in BASE ++ to provide a fair comparison (in terms of the number of parameters) with the multi-task experiments but we apply a higher dropout rate of 0 . 66 to resolve overfitting. Mono-Lingual Parsing Table 1 summarizes the training data statistics and the LAS results of the various models, with bold-font marking cases where a model outperforms the B i AFFINE parser. 1. SINGLE TASK Comparison between BASE ++ and + SHARED reveals that the additional Bi LSTM in the + SHARED does not improve the results. As expected, model averaging in ENSEMBLE improved the results over the + SHARED model. Additionally, the MAP results show slight improvements compared with + SHARED . Both of these findings suggest that the quality of predictions is likely to improve if some notion of uncertainty (via averaging, or prior insertion) is included. For Bayesian solutions, both + SGLD and + PRECOND consistently (and in a statistically significant manner) outperform the ENSEMBLE models and the strong Bi LSTM baselines on all languages, while PRECOND outperforms B i AFFINE by up to 3% on two languages (vi, and ga). The consistency of improvements as we incorporate richer means of capturing the uncertainty suggests that these gains are independent of our specific choice of neural architecture and that they might also hold for future parsing architectures. 2. MULTI TASK As explained earlier (4), comparing the parser performances under + SHARED models in single and multi task settings indicates that parser quality was degraded with the inclusion of a POS tagging task, possibly due to interference between the tasks. This issue was alleviated with the inclusion of task discriminator and the adversarial training. The + ADV model consistently improves the results (in a statistically significant manner), while outperforming the SINGLE TASK (+ SHARED ) by 2 . 2% for Persian (fa), and 5 . 2% for Irish (ga). This is an indicator that low-resource lan-fa ko ru tr vi ga TRAIN (sen.) 4798 4400 3850 3685 1400 566 Labeled Attachment Score (LAS) B i AFFINE 86.31 82.49 83.65 62.79 42.13 70.06 BASE 80.97 64.76 75.45 52.64 39.36 62.50 SINGLE TASK BASE ++ 83.15 76.70 79.44 58.92 41.03 66.58 + SHARED 83.11 76.32 79.62 58.72 41.10 66.42 ENSEMBLE 84.12 77.28 80.17 59.36 41.89 68.12 MAP 83.59 76.61 79.78 59.13 41.33 66.95 + SGLD 84.98 78.91 80.86 60.53 43.12 69.51 + PRECOND 85.76 79.83 81.9 61.71 44.52 70.91 MULTI TASK BASEMT 81.54 75.78 78.61 57.12 40.08 60.51 + SHARED 81.03 75.08 78.64 57.09 40.04 65.24 + ADV 84.93 78.12 81.23 60.66 43.11 69.89 ENSEMBLE 85.01 78.31 81.56 60.92 43.3 70.00 MAP 85.02 78.26 81.59 60.72 43.25 70.36 + SGLD 85.89 79.50 83.06 62.13 44.57 72.52 + PRECOND 86.75 80.97 84.51 63.24 45.96 74.12 Table 1: Mono-Lingual results. guages could potentially benefit more from multitask learning. Other variants of + ADV are also reported in Table 1 and similar patterns to the single task setting are observed. Next, we test the eectiveness of Bayesian learning and inference using  = 9 thinned samples (5.4). For all languages in the multi-task setting the gain from including + SGLD in + ADV is statistically significant. However, the gain decays as the amount of training data increases: from 3 . 8% on Irish (ga) to 1 . 1% on Persian (fa). This verifies our expectation that Bayesian learning helps parameter estimation and prediction in the small data regime. Compared to + SGLD , + PRECOND provides further improvements (all are statistically significant) of 2 . 2% on Irish and 1% on Persian, showing a similar generic negative correlation with data size. To summarize our mono-lingual results, the Bayesian framework shows its merit in improving predictive quality, and multi-task learning introduces new and informative signal via a related task which allows for better parameter estimation. The integration of a Bayesian parser into a multitask learning framework outperforms the B i AFFINE parser on up to 5 languages, while we observe decreasing improvements as the data size grows. Also, since our Bayesian approach builds on the BASE model, it is subject to the shortcomings of this model: For instance, on Korean (ko) the dier-ence between BASE and B i AFFINE is too large to be closed without further adjustment of model design, sen. Labeled Attachment Score (LAS) en fr ru fa kmr bxr sme en 13  81 . Cross-Lingual Delexicalized Parsing We tested our most successful mono-lingual model, MULTI TASK (+ PRECOND ), in the cross-lingual setup against an ensemble of  = 9 randomly initialized SINGLE TASK ( BASE ++) models. Table 2 summarizes the training data size (rounded up) and the LAS results of both models. As expected, the ensemble models perform best when the training and test data come from the same language or language family (as in Buryat to Russian and Kurmanji to Persian). Showing a similar pattern to mono-lingual parsing, in all cases the performance gain of the Bayesian MULTI TASK (+ PRECOND ) setting, reported as under-text in the table, consistently outperforming the ENSEMBLE (in a statistically significant manner) and the impact decays with the training set size. We aim to answer two questions: (i) how many of the generated samples are used during inference? and (ii) how much the success of SGLD depends on other sources of noise during training? 1. UTILIZINGSAMPLES Once the training stops, dierent strategies could be applied to utilize the collection of  posterior samples to recompute ARC-SCORE matrix (see 3.3). An extreme option is including all samples (denoted by All ), another option is to only use the last sample (denoted by Last ). An intermediate alternative is to choose  samples with interval     , an approach known as Thinning (Gan et al., 2017). terestingly, when we compared Thinning with a variant (denoted by  in Figure 2a) that just uses the last 5 samples , it appears that Thinning which includes earlier samples is still superior. This is likely to be an indicator that a larger (and more diverse) set of trees can potentially improve MBR decoding. In practice, we use Thinned samples. 2. DROPOUT , SGLD , ANDMINI-BATCH We would next like to better understand the interdependence between SGLD and two existing sources of noise: dropout and mini-batch size. Results are reported in Figure 2b. In all four stacked bars, the top segment demonstrates the gain when including + SGLD . The right-most stacked bar illustrates the performance of MULTI TASK (+ADV) without (bottom part of the bar) and with + SGLD . This is our reference and in the following three experiments, we make a single change to this model while keeping everything else fixed. In the left-most stacked bar, we exclude dropout. This significantly hurts the performance, an indication that the random perturbation of model by dropout can provide a positive complimentary effect for better exploration of the posterior modes via SGLD . Comparing the top parts of all stacked bars (representing the gain with + SGLD ) reveals that this gain is at its maximum when dropout is switched o. An interpretation can be that in the absence of dropout, SGLD becomes the main component for countering overfitting. In the second stacked bar from the right, we remove the noise caused by mini-batching and use the entire dataset as one batch. This slightly improves the results, although this improvement is not statistically significant. Based on eqn. 2, full training data gradient updates cause     0 in fewer steps. Consequently, the learning rate (which controls the variance of the noise) is at a higher value when the gradient is close to zero, giving a better chance for the sampler to mix via   . In the second stacked bar from left, parameters are updated for each training sentence. In this setting the learning rate,   , is consumed much faster and can potentially reach zero on the MAP solution, discarding the eect of the injected noise and + SGLD . Hence, the gain for including SGLD is at its minimum. We speculate that adjusting the decaying speed of the learning rate (which we left untouched) could allow for this extreme case to perform better. We proposed a Bayesian framework for neural dependency parsing (BNNP). We employ ecient SG-MCMC approximate inference mechanisms through stochastic gradient Langevin dynamics to generate posterior samples during optimization. The collected samples are then used via a minimum Bayes risk parsing algorithm to generate the final parse tree. In mono-lingual and cross-lingual experiments in the small data regime, where Bayesian learning in expected to be most eective, our BNNP consistently outperformed the strong Bi LSTM baselines. Moreover, when integrating the BNNP into a multi-task learning framework, utilized to prevent task interference, we outperformed the B i AFFINE parser (best system of the CoNLL17 shared task) on 5 low-resource languages by up to 9% LAS. In future work, we intend to investigate other types of priors over the network parameters (e.g., sparse priors (Lobacheva et al., 2017)). We would also like to explicitly quantify the uncertainty captured in our framework under dierent sampling strategies or MCMC-SG methods (e.g., similar to McClure and Kriegeskorte (2016); Teye et al. (2018)). This work is supported by the ERC Consolidator Grant LEXICAL (648909). The first author would like to thank Costanza Conforti, Victor Prokhorov, and Gamal Crichton for their comments on the presentation of this work. The authors would like to thank the anonymous reviewers for their helpful suggestions.
abstain objective method result result result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain result result abstain abstain other other other