In structured prediction problems, cross-lingual transfer learning is an efficient way to train quality models for low-resource languages, and further improvement can be obtained by learning from multiple source languages. However, not all source models are created equal and some may hurt performance on the target language. Previous work has explored the similarity between source and target sentences as an approximate measure of strength for different source models. In this paper, we propose a multi-view framework, by leveraging a small number of labeled target sentences, to effectively combine multiple source models into an aggregated source view at different granularity levels (language, sentence, or sub-structure), and transfer it to a target view based on a task-specific model. By encouraging the two views to interact with each other, our framework can dynamically adjust the confidence level of each source model and improve the performance of both views during training. Experiments for three structured prediction tasks on sixteen data sets show that our framework achieves significant improvement over all existing approaches, including these with access to additional source language data. Structured prediction is the task of mapping input sentences to structured outputs. It is a fundamental task in natural language processing and has many applications, i.e., sequence labeling (DeRose, 1988; Lample et al., 2016; Ma and Hovy, 2016; Hu et al., 2020b), dependency parsing (Chen and Manning, 2014; Dozat and Manning, 2016; Ahmad et al., 2019) and semantic role labeling (van der Plas et al., 2011; Strubell et al., 2018; Cai and Lapata, 2020). To achieve strong performance, structured prediction models mostly require manually labeled data that are costly to obtain in general. Cross-lingual transfer learning (Yarowsky and Ngai, 2001; Wang and Manning, 2014; Guo et al., 2018; Lin et al., 2019; Hu et al., 2021) recently attracted attention for tackling that problem, by transferring the knowledge from high-resource languages to low-resource ones. Existing works can be categorized into two types: single-source transfer and multi-source transfer. The former is limited to transferring knowledge from one source language and generally results in inferior performance than the latter (McDonald et al., 2011; Rahimi et al., 2019), especially when the target language is similar to multiple source language over various characteristics, i.e., domain, word order, capitalization, and script style. However, in practice, we are more likely to encounter the situation where some source languages are not as similar to the target language and may lead to worse performance (Rosenstein et al., 2005; Rahimi et al., 2019) (we provide an example in the Appendix A). To tackle this challenging problem, most of the previous works do majority voting (Plank and Agic, 2018) and truth inference on hard predictions of multiple sources (Rahimi et al., 2019). To better incorporate target language information, some recent works train a new model on the target unlabeled data with hard/soft predictions from multiple source models, such as mixture-of-experts model (Chen et al., 2019) and knowledge distillation (KD) (Wu et al., 2020), and assign weights to multiple sources based on language similarity. However, these similarity-based approaches are heuristic-based, and cannot well learn the confidence level of multiple source models. In this paper, we propose to leverage a small number of labeled target data to selectively transfer the knowledge from multiple source models. In many real applications, we are generally easy to obtain a small number of target labeled data. These small amounts of data can reflect the diverse strength and weakness of different source models. Concretely, the (small-size) labeled data can be utilized to learn the aggregation strategy of multiple source models or train a new task-specific model in the target language. Both the aggregation model and target task-specific model can map the inputs to the structured outputs but there exists a tradeoff. The aggregation model generally has strong cross-lingual ability since source models are firstly well trained 1 , but has lower flexibility since source models are usually frozen. Instead, the target task-specific model tends to be more flexible and has strong capacity but has poor performance since the model is easily over-fitted on the small training sample. Inspired by previous work on multi/cross-view learning (Clark et al., 2018; Jiang et al., 2019; Fei and Li, 2020), we regard the aggregation model (aggregated source view) and the target task-specific model (target view) as two views since they both can map the input sentence to structured outputs. We propose a novel multi-view framework to achieve a good trade-off between the two views. To capture the diverse strength and weakness of multiple source models, we propose three approaches to obtain the aggregated source view from language/sentence/sub-structure level in a coarse-to-fine manner. By encouraging two views to influence each other, the proposed framework can dynamically learn the confidence level of multiple source models in three coarse-to-fine granularity and make the best use of the small number of labeled data, and make both views improved during training. Benefited from the multi-view framework, our proposed approaches can leverage plenty of target unlabeled data to capture the useful target language information (Wu et al., 2020). The contributions of this work are: 1. We propose to leverage a small number of target labeled data to better aggregate multiple source models. 2. Our approach contains three novel coarse-to-fine approaches to aggregate multiple source models (section 2.2). 1 Following Wu et al. (2020), source models are previously trained on their corresponding labeled training set and frozen during training. 3. We propose a novel multi-view learning framework (section 2.3). 4. By utilizing both the label & unlabeled dataset, our approach improves two views simultaneously (section 2.4). We extensively experiment on three structured prediction tasks, which are named entity recognition (NER), part-of-speech tagging (POS), and dependency parsing. Our proposed approaches outperform several state-of-the-art approaches. The left part of Figure 1 depicts the proposed general framework. Our framework contains two views, a target view which is a target structured predictor, and an aggregated source view based on multiple pre-trained source models. Both views can map the input sentences to the structured outputs and have diverse statistical properties, and thus can provide complementary information to each other (learned by the consensus component ). In the general framework, the target view is a task-specific model. We leverage the multilingual bert (mBERT) (Devlin et al., 2019) as the sentence encoder. We feed the input sentence x to the mBERT and obtain the contextual internal states h , which are utilized by a task-specific module to produce a structured output y . Specifically, we use a Softmax layer for sequence labeling tasks and a biaffine attention mechanism (Dozat and Manning, 2016) followed by (Wu and Dredze, 2019a) for graph-based tasks like dependency parsing. The conditional probability of the structured output given the input sequence is computed by, p ( y | x ) = exp( (cid:80) u  y s ( h , u )) (cid:80) y (cid:48) exp( (cid:80) u  y (cid:48) s ( h , u )) where y (cid:48) is the candidate structured outputs, y is the structured outputs and u is the sub-structure of y . Sub-structure is the label of each token for sequence labeling and dependency head for dependency parsing. During training with gold labels, the sequence labeling objective function is the cross entropy between the gold labels and the model's <latexit sha1_base64="rlRVJQ5ZWcsNxvi/wxbomE+ien0=">AAACz3icjVHLTsJAFD3UF+ILdemmkZi4IsVgdEl04xKigAmgacsAE/pKO9UQgnHrD7jVvzL+gf6Fd8YhUYnRadqeOfeeM3PvdSKPJ8KyXjPG3PzC4lJ2Obeyura+kd/caiRhGrus7oZeGF86dsI8HrC64MJjl1HMbN/xWNMZnsp484bFCQ+DCzGKWMe3+wHvcdcWRLXbvi0GTm98PrkqX+cLVtFSy5wFJQ0K0Ksa5l/QRhchXKTwwRBAEPZgI6GnhRIsRMR1MCYuJsRVnGGCHGlTymKUYRM7pG+fdi3NBrSXnolSu3SKR29MShN7pAkpLyYsTzNVPFXOkv3Ne6w85d1G9He0l0+swIDYv3TTzP/qZC0CPRyrGjjVFClGVudql1R1Rd7c/FKVIIeIOIm7FI8Ju0o57bOpNImqXfbWVvE3lSlZuXd1bop3eUsacOnnOGdB46BYOixatXKhcqJHncUOdrFP8zxCBWeook7eER7xhGejZtwad8b9Z6qR0ZptfFvGwwcgjpQi</latexit> S 4 <latexit sha1_base64="PnWJObnPpgCE/rrcs1LzRx9Ckg8=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVRIf6LLoxmV99AFtKUk6bUPzMpmIpRRx6w+41a8S/0D/wjvjFNQiOiHJmXPvOTP3Xjvy3IQbxmtGm5mdm1/ILuaWlldW1/LrG9UkTGOHVZzQC+O6bSXMcwNW4S73WD2KmeXbHqvZg1MRr92wOHHD4IoPI9byrV7gdl3H4kS1mr7F+3Z3dDlu7+fa+YJRNOTSp4GpQAFqlcP8C5roIISDFD4YAnDCHiwk9DRgwkBEXAsj4mJCrowzjJEjbUpZjDIsYgf07dGuodiA9sIzkWqHTvHojUmpY4c0IeXFhMVpuoyn0lmwv3mPpKe425D+tvLyieXoE/uXbpL5X52ohaOLY1mDSzVFkhHVOcollV0RN9e/VMXJISJO4A7FY8KOVE76rEtNImsXvbVk/E1mClbsHZWb4l3ckgZs/hznNKjuFc3DonF+UCidqFFnsYVt7NI8j1DCGcqokPc1HvGEZ+1Cu9XutPvPVC2jNJv4trSHD2LQlDY=</latexit> S 3 m BERT <latexit sha1_base64="Qlqi4XLct23vEt6KFfLWQ5kyG1Q=">AAAC13icjVHLSsNAFD2Nr1pfsS7dBItQNyXxgS6LblxWsA9pa0nSaRvMi2QillLciVt/wK3+kfgH+hfeGVNQi+iEJGfOvefM3Hut0HViruuvGWVmdm5+IbuYW1peWV1T1/O1OEgim1XtwA2ihmXGzHV8VuUOd1kjjJjpWS6rW1cnIl6/ZlHsBP45H4as7Zl93+k5tsmJ6qj5lmfygdUbDcadvctRke+MO2pBL+lyadPASEEB6aoE6gta6CKAjQQeGHxwwi5MxPQ0YUBHSFwbI+IiQo6MM4yRI21CWYwyTGKv6NunXTNlfdoLz1iqbTrFpTcipYZt0gSUFxEWp2kynkhnwf7mPZKe4m5D+lupl0csx4DYv3STzP/qRC0cPRzJGhyqKZSMqM5OXRLZFXFz7UtVnBxC4gTuUjwibEvlpM+a1MSydtFbU8bfZKZgxd5OcxO8i1vSgI2f45wGtd2ScVDSz/YL5eN01FlsYgtFmuchyjhFBVXyvsEjnvCsXCi3yp1y/5mqZFLNBr4t5eED6suWvw==</latexit> h ( t ) 3 <latexit sha1_base64="IWXHYIZx2iANgtFqjdtPQCqAOSk=">AAAC13icjVHLSsNAFD2Nr1pfsS7dBItQNyURRZdFNy4r2Ie0tSTptA3Ni2QillLciVt/wK3+kfgH+hfeGVNQi+iEJGfOvefM3Hut0HViruuvGWVufmFxKbucW1ldW99QN/O1OEgim1XtwA2ihmXGzHV8VuUOd1kjjJjpWS6rW8NTEa9fsyh2Av+Cj0LW9sy+7/Qc2+REddR8yzP5wOqNB5OOcTUu8r1JRy3oJV0ubRYYKSggXZVAfUELXQSwkcADgw9O2IWJmJ4mDOgIiWtjTFxEyJFxhglypE0oi1GGSeyQvn3aNVPWp73wjKXaplNceiNSatglTUB5EWFxmibjiXQW7G/eY+kp7jaiv5V6ecRyDIj9SzfN/K9O1MLRw7GswaGaQsmI6uzUJZFdETfXvlTFySEkTuAuxSPCtlRO+6xJTSxrF701ZfxNZgpW7O00N8G7uCUN2Pg5zllQ2y8ZhyX9/KBQPklHncU2dlCkeR6hjDNUUCXvGzziCc/KpXKr3Cn3n6lKJtVs4dtSHj4A5f+WvQ==</latexit> h ( t ) 1 <latexit sha1_base64="luQOQwQcADXkR56QgKizH+9WUn4=">AAAC13icjVHLSsNAFD2Nr/qOdekmWIS6KUlRdFl047KCfUhbS5JO29C8SCZiKcWduPUH3OofiX+gf+GdMQW1iE5Icubce87MvdcKXSfmuv6aUebmFxaXsssrq2vrG5vqVq4WB0lks6oduEHUsMyYuY7PqtzhLmuEETM9y2V1a3gq4vVrFsVO4F/wUcjantn3nZ5jm5yojppreSYfWL3xYNIpXY0LfH/SUfN6UZdLmwVGCvJIVyVQX9BCFwFsJPDA4IMTdmEipqcJAzpC4toYExcRcmScYYIV0iaUxSjDJHZI3z7tminr0154xlJt0ykuvREpNeyRJqC8iLA4TZPxRDoL9jfvsfQUdxvR30q9PGI5BsT+pZtm/lcnauHo4VjW4FBNoWREdXbqksiuiJtrX6ri5BASJ3CX4hFhWyqnfdakJpa1i96aMv4mMwUr9naam+Bd3JIGbPwc5yyolYrGYVE/P8iXT9JRZ7GDXRRonkco4wwVVMn7Bo94wrNyqdwqd8r9Z6qSSTXb+LaUhw/oZZa+</latexit> h ( t ) 2 <latexit sha1_base64="pHpKYg8HJw3R9FgPq82Ooetu3nU=">AAAC13icjVHLSsNAFD2Nr1pfsS7dBItQNyXxgS6LblxWsA9pa0nSaRvMi2QillLciVt/wK3+kfgH+hfeGVNQi+iEJGfOvefM3Hut0HViruuvGWVmdm5+IbuYW1peWV1T1/O1OEgim1XtwA2ihmXGzHV8VuUOd1kjjJjpWS6rW1cnIl6/ZlHsBP45H4as7Zl93+k5tsmJ6qj5lmfygdUbDcadvctR0dgZd9SCXtLl0qaBkYIC0lUJ1Be00EUAGwk8MPjghF2YiOlpwoCOkLg2RsRFhBwZZxgjR9qEshhlmMRe0bdPu2bK+rQXnrFU23SKS29ESg3bpAkoLyIsTtNkPJHOgv3NeyQ9xd2G9LdSL49YjgGxf+kmmf/ViVo4ejiSNThUUygZUZ2duiSyK+Lm2peqODmExAncpXhE2JbKSZ81qYll7aK3poy/yUzBir2d5iZ4F7ekARs/xzkNarsl46Ckn+0XysfpqLPYxBaKNM9DlHGKCqrkfYNHPOFZuVBulTvl/jNVyaSaDXxbysMHSyWWfA==</latexit> h (1)3 <latexit sha1_base64="1a8461c+6aJQu/PDwaVswGEIclU=">AAAC13icjVHLSsNAFD2Nr/qOdekmWIS6KUlRdFl047KCfUhbS5JO29C8SCZiKcWduPUH3OofiX+gf+GdMQW1iE5Icubce87MvdcKXSfmuv6aUebmFxaXsssrq2vrG5vqVq4WB0lks6oduEHUsMyYuY7PqtzhLmuEETM9y2V1a3gq4vVrFsVO4F/wUcjantn3nZ5jm5yojppreSYfWL3xYNIpXY0Lxv6ko+b1oi6XNguMFOSRrkqgvqCFLgLYSOCBwQcn7MJETE8TBnSExLUxJi4i5Mg4wwQrpE0oi1GGSeyQvn3aNVPWp73wjKXaplNceiNSatgjTUB5EWFxmibjiXQW7G/eY+kp7jaiv5V6ecRyDIj9SzfN/K9O1MLRw7GswaGaQsmI6uzUJZFdETfXvlTFySEkTuAuxSPCtlRO+6xJTSxrF701ZfxNZgpW7O00N8G7uCUN2Pg5zllQKxWNw6J+fpAvn6SjzmIHuyjQPI9QxhkqqJL3DR7xhGflUrlV7pT7z1Qlk2q28W0pDx9Iv5Z7</latexit> h (1)2 <latexit sha1_base64="Rp/JJGeoNz+ZHtejK025RLIVmmE=">AAAC13icjVHLSsNAFD2Nr1pfsS7dBItQNyURRZdFNy4r2Ie0tSTptA3Ni2QillLciVt/wK3+kfgH+hfeGVNQi+iEJGfOvefM3Hut0HViruuvGWVufmFxKbucW1ldW99QN/O1OEgim1XtwA2ihmXGzHV8VuUOd1kjjJjpWS6rW8NTEa9fsyh2Av+Cj0LW9sy+7/Qc2+REddR8yzP5wOqNB5OOcTUuGnuTjlrQS7pc2iwwUlBAuiqB+oIWughgI4EHBh+csAsTMT1NGNAREtfGmLiIkCPjDBPkSJtQFqMMk9ghffu0a6asT3vhGUu1Tae49Eak1LBLmoDyIsLiNE3GE+ks2N+8x9JT3G1Efyv18ojlGBD7l26a+V+dqIWjh2NZg0M1hZIR1dmpSyK7Im6ufamKk0NInMBdikeEbamc9lmTmljWLnpryvibzBSs2NtpboJ3cUsasPFznLOgtl8yDkv6+UGhfJKOOott7KBI8zxCGWeooEreN3jEE56VS+VWuVPuP1OVTKrZwrelPHwARlmWeg==</latexit> h (1)1 <latexit sha1_base64="fZKlWfEVrB7jZjefHn/Ds4cqWcA=">AAAC0HicjVHLTsJAFD3UF+ILdemmkZi4Ii3R6JLoxiU+eCRASFsGaOjLdmokhBi3/oBb/SrjH+hfeGccEpUYnabtmXPvOTP3Xjvy3IQbxmtGm5tfWFzKLudWVtfWN/KbW7UkTGOHVZ3QC+OGbSXMcwNW5S73WCOKmeXbHqvbw1MRr9+wOHHD4IqPItb2rX7g9lzH4kS1W77FB3ZvfDnplHKdfMEoGnLps8BUoAC1KmH+BS10EcJBCh8MAThhDxYSepowYSAiro0xcTEhV8YZJsiRNqUsRhkWsUP69mnXVGxAe+GZSLVDp3j0xqTUsUeakPJiwuI0XcZT6SzY37zH0lPcbUR/W3n5xHIMiP1LN838r07UwtHDsazBpZoiyYjqHOWSyq6Im+tfquLkEBEncJfiMWFHKqd91qUmkbWL3loy/iYzBSv2jspN8S5uSQM2f45zFtRKRfOwaJwfFMonatRZ7GAX+zTPI5Rxhgqq5H2NRzzhWbvQbrU77f4zVcsozTa+Le3hA2BvlDU=</latexit> S 2 source <latexit sha1_base64="e0eyKfHFEzba81YesYB38G1Irsg=">AAAC2nicjVHLSsNAFD2Nr1pfVXHlJlgEVyUVRZdFN4IuKtgHtEUm09GG5kUyEUvoxp249Qfc6geJf6B/4Z0xBR+ITkhy5tx7zsy91w5dJ5aW9ZIzJianpmfys4W5+YXFpeLySiMOkoiLOg/cIGrZLBau44u6dKQrWmEkmGe7omkPDlW8eSWi2An8MzkMRddjl75z4XAmiTovrnU8JvucuenJ6LwjxbVMjwkVS1bZ0sv8CSoZKCFbtaD4jA56CMCRwIOAD0nYBUNMTxsVWAiJ6yIlLiLk6LjACAXSJpQlKIMRO6DvJe3aGevTXnnGWs3pFJfeiJQmNkkTUF5EWJ1m6niinRX7m3eqPdXdhvS3My+PWIk+sX/pxpn/1alaJC6wr2twqKZQM6o6nrkkuivq5uanqiQ5hMQp3KN4RJhr5bjPptbEunbVW6bjrzpTsWrPs9wEb+qWNODK93H+BI3tcmW3bJ3ulKoH2ajzWMcGtmiee6jiCDXUyTvFAx7xZHSMG+PWuPtINXKZZhVflnH/DnLAmIs=</latexit> LKL Source Models <latexit sha1_base64="n0M5zFPPvV8VUTiczPfTxdqbNho=">AAACynicjVHLSsNAFD2Nr1pfVZdugkUQFyURRZdFNy5cVLAPqLUk02kNTZMwmQghdOcPuNUPE/9A/8I7YwpqEZ2Q5My559yZe68b+V4sLeu1YMzNLywuFZdLK6tr6xvlza1mHCaC8QYL/VC0XSfmvhfwhvSkz9uR4M7Y9XnLHZ2reOuei9gLg2uZRrw7doaBN/CYI4lqpT37NjuY9MoVq2rpZc4COwcV5Ksell9wgz5CMCQYgyOAJOzDQUxPBzYsRMR1kREnCHk6zjFBibwJqTgpHGJH9B3SrpOzAe1Vzli7GZ3i0yvIaWKPPCHpBGF1mqnjic6s2N9yZzqnultKfzfPNSZW4o7Yv3xT5X99qhaJAU51DR7VFGlGVcfyLInuirq5+aUqSRki4hTuU1wQZto57bOpPbGuXfXW0fE3rVSs2rNcm+Bd3ZIGbP8c5yxoHlbt46p1dVSpneWjLmIHu9ineZ6ghgvU0dBVPuIJz8alIYzUyD6lRiH3bOPbMh4+AD91kc4=</latexit> y  1 <latexit sha1_base64="UFWMgmnfJh22xDaw7oyqQ2xgG1s=">AAACynicjVHLSsNAFD2Nr1pfVZdugkUQFyUVRZdFNy5cVLAPqLUk6bQOzYvJRAihO3/ArX6Y+Af6F94ZU1CL6IQkZ849587ce53I47G0rNeCMTe/sLhUXC6trK6tb5Q3t1pxmAiXNd3QC0XHsWPm8YA1JZce60SC2b7jsbYzPlfx9j0TMQ+Da5lGrOfbo4APuWtLotpp//A2O5j0yxWraullzoJaDirIVyMsv+AGA4RwkcAHQwBJ2IONmJ4uarAQEddDRpwgxHWcYYISeRNSMVLYxI7pO6JdN2cD2qucsXa7dIpHryCniT3yhKQThNVppo4nOrNif8ud6Zzqbin9nTyXT6zEHbF/+abK//pULRJDnOoaONUUaUZV5+ZZEt0VdXPzS1WSMkTEKTyguCDsaue0z6b2xLp21Vtbx9+0UrFq7+baBO/qljTg2s9xzoLWYbV2XLWujir1s3zURexgF/s0zxPUcYEGmrrKRzzh2bg0hJEa2afUKOSebXxbxsMHQdmRzw==</latexit> y  2 <latexit sha1_base64="aGsLqJJQEOIzP/KOC2aqFVmA/dg=">AAACynicjVHLSsNAFD2Nr1pfVZdugkUQFyX1gS6Lbly4qGBbodaSTKc1NC8mEyGE7vwBt/ph4h/oX3hnTEEtohOSnDn3nDtz73Uiz42lZb0WjJnZufmF4mJpaXllda28vtGKw0Qw3mShF4prx4655wa8KV3p8etIcNt3PN52Rmcq3r7nInbD4EqmEe/69jBwBy6zJVHttHdwm+2Ne+WKVbX0MqdBLQcV5KsRll9wgz5CMCTwwRFAEvZgI6angxosRMR1kREnCLk6zjFGibwJqTgpbGJH9B3SrpOzAe1Vzli7GZ3i0SvIaWKHPCHpBGF1mqnjic6s2N9yZzqnultKfyfP5RMrcUfsX76J8r8+VYvEACe6BpdqijSjqmN5lkR3Rd3c/FKVpAwRcQr3KS4IM+2c9NnUnljXrnpr6/ibVipW7VmuTfCubkkDrv0c5zRo7VdrR1Xr8rBSP81HXcQWtrFL8zxGHedooKmrfMQTno0LQxipkX1KjULu2cS3ZTx8AEQ9kdA=</latexit> y  3 Attention <latexit sha1_base64="pfkd67KrF2+niklga1mBnBBChIM=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkXQTUlE0aXoxmVF2wpWZRJHGzp5MJkIpQji1h9wqz8l/oH+hXfGKfhAdEKSM+fec2buvUEmolx53kvJGRkdG58oT1ampmdm56rzC608LWTIm2EqUnkcsJyLKOFNFSnBjzPJWRwI3g56ezrevuYyj9LkSPUzfhqzqyS6jEKmiDrrMJF12bl/Nlj1127OqzWv7pnl/gS+BTXY1Uirz+jgAilCFIjBkUARFmDI6TmBDw8ZcacYECcJRSbOcYMKaQvK4pTBiO3R94p2J5ZNaK89c6MO6RRBrySlixXSpJQnCevTXBMvjLNmf/MeGE99tz79A+sVE6vQJfYv3TDzvzpdi8Iltk0NEdWUGUZXF1qXwnRF39z9VJUih4w4jS8oLgmHRjnss2s0uald95aZ+KvJ1Kzehza3wJu+JQ3Y/z7On6C1Xvc3697BRm1n1466jCUsY5XmuYUd7KOBJnlLPOART86h03dunbuPVKdkNYv4spz7d7yulFU=</latexit>  (1)1 <latexit sha1_base64="5FMYB+DVAUDCStaXWNggFMl4QuI=">AAAC0XicjVHLSsNAFD2N7/qqunQTLELdlKQouhTduFS0WqhtmYxjG5oXk4lQiiBu/QG3+lPiH+hfeGdMQS2iE5KcOfeeM3Pv9ZLAT5XjvBasicmp6ZnZueL8wuLScmll9TyNM8lFncdBLBseS0XgR6KufBWIRiIFC71AXHj9Qx2/uBEy9ePoTA0S0QpZN/Kvfc4UUe1LFiQ91qm1hxV367ZTKjtVxyx7HLg5KCNfx3HpBZe4QgyODCEEIijCARhSeppw4SAhroUhcZKQb+ICtyiSNqMsQRmM2D59u7Rr5mxEe+2ZGjWnUwJ6JSltbJImpjxJWJ9mm3hmnDX7m/fQeOq7Dejv5V4hsQo9Yv/SjTL/q9O1KFxjz9TgU02JYXR1PHfJTFf0ze0vVSlySIjT+IrikjA3ylGfbaNJTe26t8zE30ymZvWe57kZ3vUtacDuz3GOg/Na1d2pOifb5f2DfNSzWMcGKjTPXezjCMeok7fEI57wbJ1aA+vOuv9MtQq5Zg3flvXwAb8UlFY=</latexit>  (1)2 <latexit sha1_base64="phBLuH3Lpg5nUXKxxCXP3lfb0DE=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkWom5L4QJdFNy4r2lqoDyZx1ME0CZOJUEpB3PoDbvWnxD/Qv/DOOAUfiE5Icubce87MvTdII5Epz3spOCOjY+MTxcnS1PTM7Fx5fqGVJbkMeTNMokS2A5bxSMS8qYSKeDuVnHWDiB8F17s6fnTDZSaS+FD1Un7SZZexuBAhU0SdHrMovWJn66f9qr86OCtXvJpnlvsT+BZUYFcjKT/jGOdIECJHFxwxFOEIDBk9HfjwkBJ3gj5xkpAwcY4BSqTNKYtTBiP2mr6XtOtYNqa99syMOqRTInolKV2skCahPElYn+aaeG6cNfubd9946rv16B9Yry6xClfE/qUbZv5Xp2tRuMC2qUFQTalhdHWhdclNV/TN3U9VKXJIidP4nOKScGiUwz67RpOZ2nVvmYm/mkzN6n1oc3O86VvSgP3v4/wJWms1f7Pm7W9U6jt21EUsYRlVmucW6thDA03ylnjAI56cA6fn3Dp3H6lOwWoW8WU59+/BepRX</latexit>  (1)3 <latexit sha1_base64="pfkd67KrF2+niklga1mBnBBChIM=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkXQTUlE0aXoxmVF2wpWZRJHGzp5MJkIpQji1h9wqz8l/oH+hXfGKfhAdEKSM+fec2buvUEmolx53kvJGRkdG58oT1ampmdm56rzC608LWTIm2EqUnkcsJyLKOFNFSnBjzPJWRwI3g56ezrevuYyj9LkSPUzfhqzqyS6jEKmiDrrMJF12bl/Nlj1127OqzWv7pnl/gS+BTXY1Uirz+jgAilCFIjBkUARFmDI6TmBDw8ZcacYECcJRSbOcYMKaQvK4pTBiO3R94p2J5ZNaK89c6MO6RRBrySlixXSpJQnCevTXBMvjLNmf/MeGE99tz79A+sVE6vQJfYv3TDzvzpdi8Iltk0NEdWUGUZXF1qXwnRF39z9VJUih4w4jS8oLgmHRjnss2s0uald95aZ+KvJ1Kzehza3wJu+JQ3Y/z7On6C1Xvc3697BRm1n1466jCUsY5XmuYUd7KOBJnlLPOART86h03dunbuPVKdkNYv4spz7d7yulFU=</latexit>  (1)1 <latexit sha1_base64="5FMYB+DVAUDCStaXWNggFMl4QuI=">AAAC0XicjVHLSsNAFD2N7/qqunQTLELdlKQouhTduFS0WqhtmYxjG5oXk4lQiiBu/QG3+lPiH+hfeGdMQS2iE5KcOfeeM3Pv9ZLAT5XjvBasicmp6ZnZueL8wuLScmll9TyNM8lFncdBLBseS0XgR6KufBWIRiIFC71AXHj9Qx2/uBEy9ePoTA0S0QpZN/Kvfc4UUe1LFiQ91qm1hxV367ZTKjtVxyx7HLg5KCNfx3HpBZe4QgyODCEEIijCARhSeppw4SAhroUhcZKQb+ICtyiSNqMsQRmM2D59u7Rr5mxEe+2ZGjWnUwJ6JSltbJImpjxJWJ9mm3hmnDX7m/fQeOq7Dejv5V4hsQo9Yv/SjTL/q9O1KFxjz9TgU02JYXR1PHfJTFf0ze0vVSlySIjT+IrikjA3ylGfbaNJTe26t8zE30ymZvWe57kZ3vUtacDuz3GOg/Na1d2pOifb5f2DfNSzWMcGKjTPXezjCMeok7fEI57wbJ1aA+vOuv9MtQq5Zg3flvXwAb8UlFY=</latexit>  (1)2 <latexit sha1_base64="pfkd67KrF2+niklga1mBnBBChIM=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkXQTUlE0aXoxmVF2wpWZRJHGzp5MJkIpQji1h9wqz8l/oH+hXfGKfhAdEKSM+fec2buvUEmolx53kvJGRkdG58oT1ampmdm56rzC608LWTIm2EqUnkcsJyLKOFNFSnBjzPJWRwI3g56ezrevuYyj9LkSPUzfhqzqyS6jEKmiDrrMJF12bl/Nlj1127OqzWv7pnl/gS+BTXY1Uirz+jgAilCFIjBkUARFmDI6TmBDw8ZcacYECcJRSbOcYMKaQvK4pTBiO3R94p2J5ZNaK89c6MO6RRBrySlixXSpJQnCevTXBMvjLNmf/MeGE99tz79A+sVE6vQJfYv3TDzvzpdi8Iltk0NEdWUGUZXF1qXwnRF39z9VJUih4w4jS8oLgmHRjnss2s0uald95aZ+KvJ1Kzehza3wJu+JQ3Y/z7On6C1Xvc3697BRm1n1466jCUsY5XmuYUd7KOBJnlLPOART86h03dunbuPVKdkNYv4spz7d7yulFU=</latexit>  (1)1 <latexit sha1_base64="5FMYB+DVAUDCStaXWNggFMl4QuI=">AAAC0XicjVHLSsNAFD2N7/qqunQTLELdlKQouhTduFS0WqhtmYxjG5oXk4lQiiBu/QG3+lPiH+hfeGdMQS2iE5KcOfeeM3Pv9ZLAT5XjvBasicmp6ZnZueL8wuLScmll9TyNM8lFncdBLBseS0XgR6KufBWIRiIFC71AXHj9Qx2/uBEy9ePoTA0S0QpZN/Kvfc4UUe1LFiQ91qm1hxV367ZTKjtVxyx7HLg5KCNfx3HpBZe4QgyODCEEIijCARhSeppw4SAhroUhcZKQb+ICtyiSNqMsQRmM2D59u7Rr5mxEe+2ZGjWnUwJ6JSltbJImpjxJWJ9mm3hmnDX7m/fQeOq7Dejv5V4hsQo9Yv/SjTL/q9O1KFxjz9TgU02JYXR1PHfJTFf0ze0vVSlySIjT+IrikjA3ylGfbaNJTe26t8zE30ymZvWe57kZ3vUtacDuz3GOg/Na1d2pOifb5f2DfNSzWMcGKjTPXezjCMeok7fEI57wbJ1aA+vOuv9MtQq5Zg3flvXwAb8UlFY=</latexit>  (1)2 <latexit sha1_base64="phBLuH3Lpg5nUXKxxCXP3lfb0DE=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkWom5L4QJdFNy4r2lqoDyZx1ME0CZOJUEpB3PoDbvWnxD/Qv/DOOAUfiE5Icubce87MvTdII5Epz3spOCOjY+MTxcnS1PTM7Fx5fqGVJbkMeTNMokS2A5bxSMS8qYSKeDuVnHWDiB8F17s6fnTDZSaS+FD1Un7SZZexuBAhU0SdHrMovWJn66f9qr86OCtXvJpnlvsT+BZUYFcjKT/jGOdIECJHFxwxFOEIDBk9HfjwkBJ3gj5xkpAwcY4BSqTNKYtTBiP2mr6XtOtYNqa99syMOqRTInolKV2skCahPElYn+aaeG6cNfubd9946rv16B9Yry6xClfE/qUbZv5Xp2tRuMC2qUFQTalhdHWhdclNV/TN3U9VKXJIidP4nOKScGiUwz67RpOZ2nVvmYm/mkzN6n1oc3O86VvSgP3v4/wJWms1f7Pm7W9U6jt21EUsYRlVmucW6thDA03ylnjAI56cA6fn3Dp3H6lOwWoW8WU59+/BepRX</latexit>  (1)3 <latexit sha1_base64="ShObPZKpqBQPNMb86OA/KHmtqSY=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkWom5KIosuiG1dSwT6k1pKk0xqaF5mJUEq3/oBb/S7xD/QvvDNOQS2iE5KcOfeeM3PvdZPA58KyXnPG3PzC4lJ+ubCyura+UdzcavA4Sz1W9+IgTluuw1ngR6wufBGwVpIyJ3QD1nSHZzLevGcp9+PoSowS1gmdQeT3fc8RRF0nXX47Ltv7k26xZFUstcxZYGtQgl61uPiCG/QQw0OGEAwRBOEADjg9bdiwkBDXwZi4lJCv4gwTFEibURajDIfYIX0HtGtrNqK99ORK7dEpAb0pKU3skSamvJSwPM1U8Uw5S/Y377HylHcb0d/VXiGxAnfE/qWbZv5XJ2sR6ONE1eBTTYliZHWedslUV+TNzS9VCXJIiJO4R/GUsKeU0z6bSsNV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFjYOKfVSxLg9L1VM96jx2sIsyzfMYVZyjhjp5h3jEE56NC0MYY2PymWrktGYb35bx8AHpX5Jz</latexit> p (1) s <latexit sha1_base64="cmkpwKj0JpyvyJ80MKu96oaVZDk=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl0484KfaEtkqTTdmheTCZCqXXrD7jV3xL/QP/CO2MKahGdkOTMufecmXuvG/s8kZb1mjPm5hcWl/LLhZXVtfWN4uZWI4lS4bG6F/mRaLlOwnwesrrk0metWDAncH3WdIdnKt68ZSLhUViTo5h1Aqcf8h73HEnUVTtw5MDtjWuTm2LJKlt6mbPAzkAJ2apGxRe00UUEDykCMISQhH04SOi5hg0LMXEdjIkThLiOM0xQIG1KWYwyHGKH9O3T7jpjQ9orz0SrPTrFp1eQ0sQeaSLKE4TVaaaOp9pZsb95j7WnutuI/m7mFRArMSD2L9008786VYtEDye6Bk41xZpR1XmZS6q7om5ufqlKkkNMnMJdigvCnlZO+2xqTaJrV711dPxNZypW7b0sN8W7uiUN2P45zlnQOCjbR2Xr8rBUOc1GnccOdrFP8zxGBeeook7eIR7xhGfjwkiNO+P+M9XIZZptfFvGwwdz25N9</latexit> T <latexit sha1_base64="9g3FCpPsTkZIiOI+nH2HgF2v/g8=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl047I+2gq1lEk6bUPzcjIRSyni1h9wq18l/oH+hXfGFHwgOiHJmXPvOTP3Xif2vURa1kvOmJqemZ3LzxcWFpeWV4qra/UkSoXLa27kR+LCYQn3vZDXpCd9fhELzgLH5w1ncKTijWsuEi8Kz+Uw5q2A9UKv67lMEtW6DJjsO93R2bhtF9rFklW29DJ/AjsDJWSrGhWfcYkOIrhIEYAjhCTsgyGhpwkbFmLiWhgRJwh5Os4xRoG0KWVxymDEDujbo10zY0PaK89Eq106xadXkNLEFmkiyhOE1WmmjqfaWbG/eY+0p7rbkP5O5hUQK9En9i/dJPO/OlWLRBcHugaPaoo1o6pzM5dUd0Xd3PxUlSSHmDiFOxQXhF2tnPTZ1JpE1656y3T8VWcqVu3dLDfFm7olDdj+Ps6foL5TtvfK1sluqXKYjTqPDWxim+a5jwqOUUWNvK/wgEc8GafGjXFr3H2kGrlMs44vy7h/B14OlDQ=</latexit> S 1 Gold labels Target Model <latexit sha1_base64="pfkd67KrF2+niklga1mBnBBChIM=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkXQTUlE0aXoxmVF2wpWZRJHGzp5MJkIpQji1h9wqz8l/oH+hXfGKfhAdEKSM+fec2buvUEmolx53kvJGRkdG58oT1ampmdm56rzC608LWTIm2EqUnkcsJyLKOFNFSnBjzPJWRwI3g56ezrevuYyj9LkSPUzfhqzqyS6jEKmiDrrMJF12bl/Nlj1127OqzWv7pnl/gS+BTXY1Uirz+jgAilCFIjBkUARFmDI6TmBDw8ZcacYECcJRSbOcYMKaQvK4pTBiO3R94p2J5ZNaK89c6MO6RRBrySlixXSpJQnCevTXBMvjLNmf/MeGE99tz79A+sVE6vQJfYv3TDzvzpdi8Iltk0NEdWUGUZXF1qXwnRF39z9VJUih4w4jS8oLgmHRjnss2s0uald95aZ+KvJ1Kzehza3wJu+JQ3Y/z7On6C1Xvc3697BRm1n1466jCUsY5XmuYUd7KOBJnlLPOART86h03dunbuPVKdkNYv4spz7d7yulFU=</latexit>  (1)1 <latexit sha1_base64="xxAx7b5j53uB6CbGK+Vza+p/rgo=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkXQTUlE0aXoxmVF2wpWZRJHGzp5MJkIpQji1h9wqz8l/oH+hXfGKfhAdEKSM+fec2buvUEmolx53kvJGRkdG58oT1ampmdm56rzC608LWTIm2EqUnkcsJyLKOFNFSnBjzPJWRwI3g56ezrevuYyj9LkSPUzfhqzqyS6jEKmiDrrMJF12bl3Nlj1127OqzWv7pnl/gS+BTXY1Uirz+jgAilCFIjBkUARFmDI6TmBDw8ZcacYECcJRSbOcYMKaQvK4pTBiO3R94p2J5ZNaK89c6MO6RRBrySlixXSpJQnCevTXBMvjLNmf/MeGE99tz79A+sVE6vQJfYv3TDzvzpdi8Iltk0NEdWUGUZXF1qXwnRF39z9VJUih4w4jS8oLgmHRjnss2s0uald95aZ+KvJ1Kzehza3wJu+JQ3Y/z7On6C1Xvc3697BRm1n1466jCUsY5XmuYUd7KOBJnlLPOART86h03dunbuPVKdkNYv4spz7d7pIlFQ=</latexit>  (1)0 <latexit sha1_base64="LYrDxAUqlje+Cxmds6PL7ukPxv8=">AAAC13icjVHLSsNAFD2Nr1pfsS7dBItQNyUVRZdFNy4r2Ie0tSTptA3Ni2QillDciVt/wK3+kfgH+hfeGVNQi+iEJGfOvefM3HvNwLEjruuvGWVufmFxKbucW1ldW99QN/P1yI9Di9Us3/HDpmlEzLE9VuM2d1gzCJnhmg5rmKNTEW9cszCyfe+CjwPWcY2BZ/dty+BEddV82zX40Ownw0lXv0qKfG/SVQt6SZdLmwXlFBSQrqqvvqCNHnxYiOGCwQMn7MBARE8LZegIiOsgIS4kZMs4wwQ50saUxSjDIHZE3wHtWinr0V54RlJt0SkOvSEpNeySxqe8kLA4TZPxWDoL9jfvRHqKu43pb6ZeLrEcQ2L/0k0z/6sTtXD0cSxrsKmmQDKiOit1iWVXxM21L1VxcgiIE7hH8ZCwJZXTPmtSE8naRW8NGX+TmYIVeyvNjfEubkkDLv8c5yyo75fKhyX9/KBQOUlHncU2dlCkeR6hgjNUUSPvGzziCc/KpXKr3Cn3n6lKJtVs4dtSHj4A45mWvA==</latexit> h ( t ) 0 <latexit sha1_base64="I/arKfw4XoVvdgQBqOe4q/gbhgo=">AAAC13icjVHLSsNAFD2Nr1pfsS7dBItQNyURRZdFNy4r2Ie0tSTptA3Ni2QillLciVt/wK3+kfgH+hfeGVNQi+iEJGfOvefM3Hut0HViruuvGWVufmFxKbucW1ldW99QN/O1OEgim1XtwA2ihmXGzHV8VuUOd1kjjJjpWS6rW8NTEa9fsyh2Av+Cj0LW9sy+7/Qc2+REddR8yzP5wOqNB5OOfjUuGnuTjlrQS7pc2iwwUlBAuiqB+oIWughgI4EHBh+csAsTMT1NGNAREtfGmLiIkCPjDBPkSJtQFqMMk9ghffu0a6asT3vhGUu1Tae49Eak1LBLmoDyIsLiNE3GE+ks2N+8x9JT3G1Efyv18ojlGBD7l26a+V+dqIWjh2NZg0M1hZIR1dmpSyK7Im6ufamKk0NInMBdikeEbamc9lmTmljWLnpryvibzBSs2NtpboJ3cUsasPFznLOgtl8yDkv6+UGhfJKOOott7KBI8zxCGWeooEreN3jEE56VS+VWuVPuP1OVTKrZwrelPHwAQ/OWeQ==</latexit> h (1)0 <latexit sha1_base64="pfkd67KrF2+niklga1mBnBBChIM=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkXQTUlE0aXoxmVF2wpWZRJHGzp5MJkIpQji1h9wqz8l/oH+hXfGKfhAdEKSM+fec2buvUEmolx53kvJGRkdG58oT1ampmdm56rzC608LWTIm2EqUnkcsJyLKOFNFSnBjzPJWRwI3g56ezrevuYyj9LkSPUzfhqzqyS6jEKmiDrrMJF12bl/Nlj1127OqzWv7pnl/gS+BTXY1Uirz+jgAilCFIjBkUARFmDI6TmBDw8ZcacYECcJRSbOcYMKaQvK4pTBiO3R94p2J5ZNaK89c6MO6RRBrySlixXSpJQnCevTXBMvjLNmf/MeGE99tz79A+sVE6vQJfYv3TDzvzpdi8Iltk0NEdWUGUZXF1qXwnRF39z9VJUih4w4jS8oLgmHRjnss2s0uald95aZ+KvJ1Kzehza3wJu+JQ3Y/z7On6C1Xvc3697BRm1n1466jCUsY5XmuYUd7KOBJnlLPOART86h03dunbuPVKdkNYv4spz7d7yulFU=</latexit>  (1)1 <latexit sha1_base64="sLMepVTze8Enhuz/xIhZ1VqCQxU=">AAACz3icjVHLTsJAFD3UF+ILdemmkZjghrRGo0uiG5eQCJgAmmkZoKG0zXSqIQTj1h9wq39l/AP9C++MJVGJ0Wnanjn3njNz73Ui34ulZb1mjLn5hcWl7HJuZXVtfSO/uVWPw0S4vOaGfiguHRZz3wt4TXrS55eR4Gzo+LzhDM5UvHHDReyFwYUcRbw9ZL3A63ouk0S1WsyP+uxqXLT3J9f5glWy9DJngZ2CAtJVCfMvaKGDEC4SDMERQBL2wRDT04QNCxFxbYyJE4Q8HeeYIEfahLI4ZTBiB/Tt0a6ZsgHtlWes1S6d4tMrSGlijzQh5QnC6jRTxxPtrNjfvMfaU91tRH8n9RoSK9En9i/dNPO/OlWLRBcnugaPaoo0o6pzU5dEd0Xd3PxSlSSHiDiFOxQXhF2tnPbZ1JpY1656y3T8TWcqVu3dNDfBu7olDdj+Oc5ZUD8o2Uclq3pYKJ+mo85iB7so0jyPUcY5KqiRd4RHPOHZqBq3xp1x/5lqZFLNNr4t4+EDEm6TsQ==</latexit>  (1) Attention <latexit sha1_base64="ShObPZKpqBQPNMb86OA/KHmtqSY=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkWom5KIosuiG1dSwT6k1pKk0xqaF5mJUEq3/oBb/S7xD/QvvDNOQS2iE5KcOfeeM3PvdZPA58KyXnPG3PzC4lJ+ubCyura+UdzcavA4Sz1W9+IgTluuw1ngR6wufBGwVpIyJ3QD1nSHZzLevGcp9+PoSowS1gmdQeT3fc8RRF0nXX47Ltv7k26xZFUstcxZYGtQgl61uPiCG/QQw0OGEAwRBOEADjg9bdiwkBDXwZi4lJCv4gwTFEibURajDIfYIX0HtGtrNqK99ORK7dEpAb0pKU3skSamvJSwPM1U8Uw5S/Y377HylHcb0d/VXiGxAnfE/qWbZv5XJ2sR6ONE1eBTTYliZHWedslUV+TNzS9VCXJIiJO4R/GUsKeU0z6bSsNV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFjYOKfVSxLg9L1VM96jx2sIsyzfMYVZyjhjp5h3jEE56NC0MYY2PymWrktGYb35bx8AHpX5Jz</latexit> p (1) s <latexit sha1_base64="ShObPZKpqBQPNMb86OA/KHmtqSY=">AAACzHicjVHLSsNAFD2Nr1pfVZdugkWom5KIosuiG1dSwT6k1pKk0xqaF5mJUEq3/oBb/S7xD/QvvDNOQS2iE5KcOfeeM3PvdZPA58KyXnPG3PzC4lJ+ubCyura+UdzcavA4Sz1W9+IgTluuw1ngR6wufBGwVpIyJ3QD1nSHZzLevGcp9+PoSowS1gmdQeT3fc8RRF0nXX47Ltv7k26xZFUstcxZYGtQgl61uPiCG/QQw0OGEAwRBOEADjg9bdiwkBDXwZi4lJCv4gwTFEibURajDIfYIX0HtGtrNqK99ORK7dEpAb0pKU3skSamvJSwPM1U8Uw5S/Y377HylHcb0d/VXiGxAnfE/qWbZv5XJ2sR6ONE1eBTTYliZHWedslUV+TNzS9VCXJIiJO4R/GUsKeU0z6bSsNV7bK3joq/qUzJyr2nczO8y1vSgO2f45wFjYOKfVSxLg9L1VM96jx2sIsyzfMYVZyjhjp5h3jEE56NC0MYY2PymWrktGYb35bx8AHpX5Jz</latexit> p (1) s Sentence-level aggregation Substructure-level aggregation Language-level aggregation <latexit sha1_base64="Z8Cxx/Hd3mVswf+lW6GlLMaVrcE=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl002VF+4BaSjKd1tA0CZOJWorgD7jVTxP/QP/CO+MU1CI6IcmZc+85M/dePwmDVDrOa86am19YXMovF1ZW19Y3iptbjTTOBON1FoexaPleysMg4nUZyJC3EsG9kR/ypj88U/HmDRdpEEeXcpzwzsgbREE/YJ4k6uKu63aLJafs6GXPAteAEsyqxcUXXKGHGAwZRuCIIAmH8JDS04YLBwlxHUyIE4QCHee4R4G0GWVxyvCIHdJ3QLu2YSPaK89UqxmdEtIrSGljjzQx5QnC6jRbxzPtrNjfvCfaU91tTH/feI2Ilbgm9i/dNPO/OlWLRB8nuoaAako0o6pjxiXTXVE3t79UJckhIU7hHsUFYaaV0z7bWpPq2lVvPR1/05mKVXtmcjO8q1vSgN2f45wFjYOye1R2zg9LlVMz6jx2sIt9mucxKqiihjp5D/CIJzxbVSuyMuv2M9XKGc02vi3r4QMPXZAl</latexit> x 1 <latexit sha1_base64="Gx72PyhmZwwXmHsmqKjIYRGqdIs=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl002VF+4BaSjKd1qFpEiYTtRTBH3Crnyb+gf6Fd8YU1CI6IcmZc+85M/dePw5EohznNWfNzS8sLuWXCyura+sbxc2tRhKlkvE6i4JItnwv4YEIeV0JFfBWLLk38gPe9IdnOt684TIRUXipxjHvjLxBKPqCeYqoi7vuQbdYcsqOWfYscDNQQrZqUfEFV+ghAkOKEThCKMIBPCT0tOHCQUxcBxPiJCFh4hz3KJA2pSxOGR6xQ/oOaNfO2JD22jMxakanBPRKUtrYI01EeZKwPs028dQ4a/Y374nx1Hcb09/PvEbEKlwT+5dumvlfna5FoY8TU4OgmmLD6OpY5pKaruib21+qUuQQE6dxj+KSMDPKaZ9to0lM7bq3nom/mUzN6j3LclO861vSgN2f45wFjYOye1R2zg9LldNs1HnsYBf7NM9jVFBFDXXyHuART3i2qlZopdbtZ6qVyzTb+Lashw8RvZAm</latexit> x 2 <latexit sha1_base64="I2fS8C4pYfLOmSco7JCnmoClNp8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRIf6LLopsuK9gG1lGQ6rYNpEiYTtRTBH3Crnyb+gf6Fd8YU1CI6IcmZc+85M/dePw5EohznNWfNzM7NL+QXC0vLK6trxfWNRhKlkvE6i4JItnwv4YEIeV0JFfBWLLk39APe9K9Pdbx5w2UiovBCjWLeGXqDUPQF8xRR53fd/W6x5JQds+xp4GaghGzVouILLtFDBIYUQ3CEUIQDeEjoacOFg5i4DsbESULCxDnuUSBtSlmcMjxir+k7oF07Y0Paa8/EqBmdEtArSWljhzQR5UnC+jTbxFPjrNnfvMfGU99tRH8/8xoSq3BF7F+6SeZ/dboWhT6OTQ2CaooNo6tjmUtquqJvbn+pSpFDTJzGPYpLwswoJ322jSYxteveeib+ZjI1q/csy03xrm9JA3Z/jnMaNPbK7mHZOTsoVU6yUeexhW3s0jyPUEEVNdTJe4BHPOHZqlqhlVq3n6lWLtNs4tuyHj4AFB2QJw==</latexit> x 3 The Aggregated Source View The Target View <latexit sha1_base64="/uQTlZSR96coa4Cn7QeEsTEm1dY=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl002Wl9gG1lGQ6rUPTJEwmSimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y8udLx1y2UiovBKTWLeHXvDUAwE8xRR9bhX7xVLTtkxy54HbgZKyFYtKr7gGn1EYEgxBkcIRTiAh4SeDlw4iInrYkqcJCRMnOMeBdKmlMUpwyN2RN8h7ToZG9JeeyZGzeiUgF5JShsHpIkoTxLWp9kmnhpnzf7mPTWe+m4T+vuZ15hYhRti/9LNMv+r07UoDHBmahBUU2wYXR3LXFLTFX1z+0tVihxi4jTuU1wSZkY567NtNImpXffWM/E3k6lZvWdZbop3fUsasPtznPOgeVR2T8rO5XGpcp6NOo897OOQ5nmKCqqooUHeQzziCc9W1Qqt1Lr7TLVymWYX35b18AFNDZA/</latexit> p S <latexit sha1_base64="/uQTlZSR96coa4Cn7QeEsTEm1dY=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl002Wl9gG1lGQ6rUPTJEwmSimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y8udLx1y2UiovBKTWLeHXvDUAwE8xRR9bhX7xVLTtkxy54HbgZKyFYtKr7gGn1EYEgxBkcIRTiAh4SeDlw4iInrYkqcJCRMnOMeBdKmlMUpwyN2RN8h7ToZG9JeeyZGzeiUgF5JShsHpIkoTxLWp9kmnhpnzf7mPTWe+m4T+vuZ15hYhRti/9LNMv+r07UoDHBmahBUU2wYXR3LXFLTFX1z+0tVihxi4jTuU1wSZkY567NtNImpXffWM/E3k6lZvWdZbop3fUsasPtznPOgeVR2T8rO5XGpcp6NOo897OOQ5nmKCqqooUHeQzziCc9W1Qqt1Lr7TLVymWYX35b18AFNDZA/</latexit> p S <latexit sha1_base64="/uQTlZSR96coa4Cn7QeEsTEm1dY=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl002Wl9gG1lGQ6rUPTJEwmSimCP+BWP038A/0L74wpqEV0QpIz595zZu69fhyIRDnOa85aWFxaXsmvFtbWNza3its7zSRKJeMNFgWRbPtewgMR8oYSKuDtWHJv7Ae85Y8udLx1y2UiovBKTWLeHXvDUAwE8xRR9bhX7xVLTtkxy54HbgZKyFYtKr7gGn1EYEgxBkcIRTiAh4SeDlw4iInrYkqcJCRMnOMeBdKmlMUpwyN2RN8h7ToZG9JeeyZGzeiUgF5JShsHpIkoTxLWp9kmnhpnzf7mPTWe+m4T+vuZ15hYhRti/9LNMv+r07UoDHBmahBUU2wYXR3LXFLTFX1z+0tVihxi4jTuU1wSZkY567NtNImpXffWM/E3k6lZvWdZbop3fUsasPtznPOgeVR2T8rO5XGpcp6NOo897OOQ5nmKCqqooUHeQzziCc9W1Qqt1Lr7TLVymWYX35b18AFNDZA/</latexit> p S <latexit sha1_base64="Pxi6dlL8Jsu3+MY6AUGwYaV8hT0=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl002XFvqCWkkyndWiahMlEKUXwB9zqp4l/oH/hnTEFtYhOSHLm3HvOzL3XjwORKMd5zVkLi0vLK/nVwtr6xuZWcXunmUSpZLzBoiCSbd9LeCBC3lBCBbwdS+6N/YC3/NGFjrduuUxEFNbVJObdsTcMxUAwTxF1FffqvWLJKTtm2fPAzUAJ2apFxRdco48IDCnG4AihCAfwkNDTgQsHMXFdTImThISJc9yjQNqUsjhleMSO6DukXSdjQ9prz8SoGZ0S0CtJaeOANBHlScL6NNvEU+Os2d+8p8ZT321Cfz/zGhOrcEPsX7pZ5n91uhaFAc5MDYJqig2jq2OZS2q6om9uf6lKkUNMnMZ9ikvCzChnfbaNJjG16956Jv5mMjWr9yzLTfGub0kDdn+Ocx40j8ruSdm5PC5VzrNR57GHfRzSPE9RQRU1NMh7iEc84dmqWqGVWnefqVYu0+zi27IePgBPbZBA</latexit> p T <latexit sha1_base64="aEswHmbf3N0nb7qrHP1xFzZhLds=">AAAC3HicjVHLSsNAFD2Nr1pfURcu3ASL4KqkouiyKIILFxX6glbLZBzbYF4kE1FKd+7ErT/gVr9H/AP9C++MEdQiOiHJmXPvOXPvXCfy3ETa9kvOGBufmJzKTxdmZufmF8zFpUYSpjEXdR56YdxyWCI8NxB16UpPtKJYMN/xRNO52Ffx5qWIEzcMavI6Eic+6wXuucuZJKprrnR8JvuceYOjYbcjxZUc7B8MT2tds2iXbL2sUVDOQBHZqobmMzo4QwiOFD4EAkjCHhgSetoow0ZE3AkGxMWEXB0XGKJA2pSyBGUwYi/o26NdO2MD2ivPRKs5neLRG5PSwjppQsqLCavTLB1PtbNif/MeaE9V2zX9nczLJ1aiT+xfus/M/+pULxLn2NU9uNRTpBnVHc9cUn0rqnLrS1eSHCLiFD6jeEyYa+XnPVtak+je1d0yHX/VmYpVe57lpnhTVdKAyz/HOQoam6Xydsk+3ipW9rJR57GKNWzQPHdQwSGqqOv6H/CIJ+PUuDFujbuPVCOXaZbxbRn371NJmUI=</latexit> LT CE <latexit sha1_base64="Jw3iAdZ0MSmKoAZ5KsxtbvCK60A=">AAAC3HicjVHLSsNAFD2Nr1pfURcu3ASL4KqkouhSLIILFxXtA6yWyTjaYF4kE7GU7tyJW3/ArX6P+Af6F94ZU1CL6IQkZ86958y9c53IcxNp2685Y2R0bHwiP1mYmp6ZnTPnF+pJmMZc1HjohXHTYYnw3EDUpCs90YxiwXzHEw3nqqLijWsRJ24YHMtuJE59dhm4Fy5nkqi2udTymexw5vUO+u2WFDeyV9nrnx21zaJdsvWyhkE5A0VkqxqaL2jhHCE4UvgQCCAJe2BI6DlBGTYi4k7RIy4m5Oq4QB8F0qaUJSiDEXtF30vanWRsQHvlmWg1p1M8emNSWlglTUh5MWF1mqXjqXZW7G/ePe2pauvS38m8fGIlOsT+pRtk/lenepG4wLbuwaWeIs2o7njmkupbUZVbX7qS5BARp/A5xWPCXCsH92xpTaJ7V3fLdPxNZypW7XmWm+JdVUkDLv8c5zCor5fKmyX7cKO4s5uNOo9lrGCN5rmFHeyjipqu/xFPeDbOjFvjzrj/TDVymWYR35bx8AFQ6ZlB</latexit> LS CE Figure 1: The proposed multi-view framework with K source models ( K = 4 in this case). soft predictions 2 , LCE =  log p ( y  | x ) =  n (cid:88) i =1 log p ( y  i | x ) where y  is the gold label sequence. In dependency parsing, we use the biaffine parser (Dozat and Manning, 2016) which is one of the state-of-the-art parsers. Following Wu and Dredze (2019a), we replace the BiLSTM encoder with mBERT. Similar to sequence labeling, the biaffine parser models the dependency head separately for each token. Following Anderson and Gomez-Rodrguez (2020), it has two independent distributions, one for head prediction and one for label prediction. The cross-entropy loss for dependency head is, LCE ( head ) =  log p ( t  | x ) =  n (cid:88) i =1 log p ( h  i | x ) where h  i is the gold head for i -th word of the gold tree t  . Together with the similar cross-entropy 2 This is a common way in the BERT-fintuning setup (Wu and Dredze, 2019a; Wu et al., 2020). loss of predicted edge labels, the dependency parsing objective function is LCE = LCE ( head ) + LCE ( label ) . In this section, we take the sequence labeling tasks as an example to introduce our aggregated source view. The source models have the same model structure as the task-specific model of the target view in section 2.1. As presented in fig-ure 1, for a K -source setup, we have K pre-trained source models S k , k  { 1 , . . . , K } and the target structured model T . Given a sentence x = { x 0 , . . . , x n } , where x 0 represents the [ CLS ] token, we feed it to these models and get the internal states { h (1) , . . . , h ( K ) } and the probability distributions { p (1) s , . . . , p ( K ) s } over the structured output of K source models S k , and h ( t ) and p T of the target model. To aggregate all source models, we propose three novel coarse-to-fine approaches. the Figure 1. The final output distribution of the aggregated source view can be computed as, p S ( y | x ) = K (cid:88) k =1  ( k ) lang  p ( k ) s ( y | x ) We use superscript to represent the index of vector  lang . Note that we use lowercase s , uppercase S , and uppercase T to differentiate the final outputs of the source model, aggregated source view, and target view respectively. In this approach, the k -th source model has the same weight  ( k ) lang over all sentences. In this section, we leverage an attention mechanism (Luong et al., 2015; Vaswani et al., 2017) to learn the weight of each source model on an input sentence, as shown on the top right part of Figure 1. Firstly, we use the internal states of the [ CLS ] token as sentence representation. Secondly, h ( t ) 0 from the target model T is used as a query to attend h ( k ) 0 from the k -th source model S k to produce the probabilities  sent ( x )  RK . where K 0 is the concatenation of sentence representations from K source models, and W  R d  d is the bilinear weight matrix. Then the probabilities are utilized to compute the aggregation distribution p S ( y | x ) as follows, p S ( y | x ) = K (cid:88) k =1  ( k ) sent ( x )  p ( k ) s ( y | x ) In sentence-level aggregation approach, k -th source model has the same weight  ( k ) sent ( x ) over each substructure of a sentence, but different weights over different sentences and thus can capture the diverse strengths of each source on different sentences. We further propose a fine-grained aggregation approach on sub-structure level, which is also based on the attention mechanism. As shown in the left part of Figure 1, for token x i in a given sentence x , we use its representation h ( t ) i as the query to attend the corresponding representation from each source model. We compute the probabilities  sub ( x i ) for i-th sub-structure as follows, K i = [ h (1) i ; . . . ; h ( K ) i ]  sub ( x i ) = Softmax ( h ( t ) i WK Ti ) Then the aggregation distribution becomes, p S ( y | x ) = n (cid:89) i =1 K (cid:88) k =1  ( k ) sub ( x i )  p ( k ) s ( y i | x ) In this approach, our target model acts as a selector to dynamically assess the multiple source models on sub-structure level. To achieve a good trade-off between the target view and the aggregation view during training, inspired by Clark et al. (2018), we utilize the KL divergence 3 as the metric to encourage the similarity between the two views. For sequence labeling, the objective is, LKL ( x ) = KL ( p S ( y | x ) || p T ( y | x )) 2.4 Overall Training Objective In the model training, for the unlabeled sentences, we only calculate the KL-divergence loss LU = LKL . For the labeled sentences, we train the model with two supervised cross-entropy loss in addition to the KL-divergence loss, LL =  1 LSCE +  2 LTCE +  3 LKL where  1 ,  2 and  3 are the interpolation factors. Finally, we introduce an interpolation  to balance the labeled and unlabeled sentences and the overall learning objective is L =  LL + (1   ) LU . Connections to KD There are mainly four differences between KD (Wu et al., 2020) and our approach: 1. Unlike our approach, KD only utilizes the target unlabeled data, from which it cannot well learn the strength and weakness of different source models (see Sec.1 for more discus-sion.). 3 We also try many metrics of measuring the similarity between two probability distributions, e.g., mean squared error (MSE) (Wu et al., 2020), Cosine, and Jensen-Shannon divergence (JS) (Ruder and Plank, 2017), and we find KL perform best. 2. KD assigns equal importance to multiple source models, which can be seen as a fixed uniform vector in our language-level aggregation approach. 3. Besides language-level aggregation, we propose two fine-grained aggregation strategies to dynamically balance the information from source models. 4. To achieve the previously described goal, our approach has trainable parameters in the aggregation component and our multi-view learning framework can jointly learn the parameters of two views. Following previous work on cross-lingual transfer (Rahimi et al., 2019; Wu et al., 2020), the source models are previously trained on their corresponding labeled training data. During training, we freeze the parameters of the pre-trained source models and only update the parameters of calculating weights  in the aggregated source view, and update all parameters of the target view. In every iteration, we randomly sample a batch of data from the labeled dataset and unlabeled dataset according to the interpolation  . In the experiments, our model can significantly benefit from this training strategy by controlling the ratio of labeled data and unlabeled data. During the inference phase, we have two options to obtain the predictions: utilizing the aggregated source view or the target view. In our experiments, we use the second one as the main result for its simplicity and better performance. We experiment on three structured prediction tasks: NER, POS tagging, and dependency parsing. Following previous work (Rahimi et al., 2019; Wu et al., 2020), we conduct the experiments in a leave-one-out setting in which we hold out one language as the target language and the others as the source languages. To simulate the low-resources scenario, for each training set in a specific target language, we randomly select fifty sentences 4 with the gold annotations and discard the annotations of the remaining sentences to construct the training set. We 4 We explore the effects of randomness on labeled data in the Appendix C.1 and the results show that our approach is robust to randomness in the selection of labeled data. randomly select six languages from Universal Dependencies Treebanks (v2.2) 5 for dependency parsing and POS tagging tasks. We use the datasets from CoNLL 2002 and CoNLL 2003 shared tasks (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003) for NER tasks. We utilize the base cased multilingual BERT (Devlin et al., 2019) as base model for all approaches. We run each approach five times and report the averaged accuracy for POS tagging, f1-score for NER, and unlabelled attachment score (UAS) and labeled attachment score (LAS) for dependency parsing. More details can be found in the Appendix B.1. We compare the results of the target view of our language/sentence/sub-structure-level approaches which are denoted as Ours-lang/sent/sub respectively, with a large amount of previous state-of-the-art cross-lingual baselines: direct fine-tuning (DT-finetuning), direct transfer (DT), hard knowledge distillation (hard-KD) (Liu et al., 2017), soft knowledge distillation (soft-KD) (Hinton et al., 2015; Wu et al., 2020), unified multilingual model (UMM) which is similar to (Yasunaga et al., 2018; Ak-bik et al., 2019), and bootstrapping approaches (Yarowsky, 1995; Zhou and Li, 2005; McClosky et al., 2006; Ruder and Plank, 2018) based on UMM. DT In DT, there is only test data in the target language. Therefore, we evaluate this approach in three ways: 1) using the mean probability distribution of source models (DT-mean); 2) using the maximal probability distribution of source models over the sub-structure level (DT-max). 3) evaluating each source model and voting on the sub-structure level (DT-vote). We also provide the maximal results of DT on language level (DT-Max(lang)) 6 . Hard-KD The hard knowledge distillation approaches first predict the pseudo labels on target unlabeled training set by using pre-trained source models and then train a new model on the pseudo labeled data (Liu et al., 2017; Rahimi et al., 2019). 5 https://universaldependencies.org/ 6 We separately evaluate the source language models on the target test data and choose the best score. Since we don't know which source model is the best for DT in practice, the DT-Max(lang) results are only for reference. We obtain the pseudo labels in four ways: 1) using DT-mean (hard-KD-mean); 2) using DT-max (hard-KD-max); 3) using DT-vote (hard-KD-vote); 4) concatenating all predictions of source models instead of voting (hard-KD-concat). For fairly comparison, we also concatenate the fifty target labeled data into the pseudo labeled data. Soft-KD Instead of leveraging hard predictions of source models in hard-KD, the soft-KD leverages soft probability distribution of source models. The original Soft-KD (Wu et al., 2020) only focuses on zero-shot NER tasks. Instead, we modify their training objective to leverage fifty target labeled data and adapt it to POS tagging and dependency parsing tasks. (Refer to section 2.4 for details.) We re-implement their two proposed approaches: 1) uniformly aggregating multiple source models (KD-avg); 2) aggregating source models by fixed weights pre-trained on source unlabeled data based on language similarity (KD-sim) 7 . UMM The UMM is trained on the concatenation of all source languages labeled data and fifty labeled data of target language. Bootstrapping Bootstrapping approaches firstly train a UMM and then add the most confident sen-7 For more details of the two approaches, please refer to the original paper. tences of target unlabeled data into the training set every iteration during training. We compare our approaches to Self-Training (Yarowsky, 1995; McClosky et al., 2006) and Tri-Training (Ruder and Plank, 2018). We provide the upper bound results of DT (DT-gold). We construct the upper bound using the gold label set in test data by selecting the gold label if any prediction of source models appears in the gold set. Besides, unlike UMM, self-training, tri-training, and KD-sim, our approaches do not require extra resources like source language training data. We report the results in Table 1 for NER and POS tagging, and 2 for dependency parsing. Common Results on All Tasks As shown in Table 1 and 2, our three proposed approaches outperform most of the baselines on all tasks, which demonstrates the effectiveness of the proposed multi-view learning framework. When trained on only fifty labeled data, the task-specific model shows significantly poor results especially on dependency parsing which verifies our intuition that the task-specific model is easily over-fitted and only training the task-specific model is not sufficient. Notably, UMM, self-training, and tri-training With 50 labeled data ENCAIDHIFIRU Avg. do not yield improvements compared to hard-KD-*, soft-KD-*, and Ours-*, verifying our motivation that simply concatenating all training data is not sufficient to model the difference between multiple sources. We also observe that our three approaches outperform the two KD approaches consistently, indicating that their simple or heuristic-based aggregation strategies are difficult to assess the diverse quality of source models. It is also worth noticing that with a more fine-grained aggregated source view, the target view has stronger performance, especially for Ours-sub 8 . Even though UMM, self-training, tri-training, and soft-KD-sim all utilize source language training data during training, Ours-sub achieves remarkable advantage over these baselines without the extra resources, especially for dependency parsing. Other results Although Tri-training achieves the highest score and UAS on De of NER and En of parsing respectively, it is not statistically significant compared to Ours-sub and the gap is very marginal ( < 0 . 1% ). For NER task, it is probably due to the difference of the capitalization style between De and other languages on CoNLL NER (Chen et al., 2019), which may lead to the negative transfer problem 9 . Besides, the gaps between the 8 This is mainly due to the stronger cross-lingual ability of the aggregated source view. We further analyze this in section 4.1. DT-gold and the best transfer approaches suggest the large potential space on multi-source transfer tasks. In this section, we study the reason why the proposed framework works. We show the performance of the aggregated source view in Figure 2. It can be seen that with a more fine-grained strategy, the performance of the aggregated source view becomes stronger. It demonstrates the effectiveness of more fine-grained aggregation strategies in the multi-source transfer. The only counter case is language and sentence level on NL, and the performance of the target view drops accordingly. Connecting to Table 1, the target view has the same trends. The reason is probably that the stronger aggregated source view can lead to a stronger target view and vice versa, and the framework achieves a good trade-off to make them both improved. To further understand the proposed framework we investigate the component contributions. We gradually remove some components of our sub-structure-level model, i.e., LS CE , LT CE and LKL , and evaluate approach is the second-best system in this case, indicating that it can alleviate this problem by better leveraging labeled data to access the confidence level of source models on more fine-grained-level property. CoNLL02/03 NER task. w/o denotes without'. on the NER task. We report the average results of twenty-five runs 10 in Table 3. Without LKL the approach degenerates into supervised training with only fifty labeled data and it leads to the largest drop in performance. It is because the model is easily over-fitted. Though the performance drops without one of LS CE and LT CE , it still outperform KD-* baselines of Table 1. w/o LS CE leads to less drops than w/o LT CE , which suggest that the labeled data influence more in the target model. Besides, without both cross-entropy loss of labeled data, the approach degenerates into a zero-shot manner and results in inferior performance. In this section, we study the impact of the sizes of labeled data and unlabeled data on the target language for the ours-sub model. We randomly select { 10 , 50 , 200 , 1000 } labeled data and { 1000 , 2000 , 4000 , All } unlabeled data. We repeat each experiment five times and report the average results of both two views 11 . It can be seen that with more labeled data or unlabeled data, the results both become higher and the labeled data shows higher influence than the unlabeled data. Unlike the aggregated source view, the target view gains significantly larger boosts when the size of 10 We randomly select five different copies of labeled data and run five times for each copy. 11 We only show the De results due to the space limitation. The results of the other three languages can be found in the Appendix C.2. unlabeled data or labeled data increases (the aggregation view generally shows comparable or even superior results to the target view with fewer data). This verifies our motivation that there exists a tradeoff between two views. With #0 unlabeled data, the task-specific model is over-fitted when only trained on #200 or less labeled data. Target labeled data: #10 #50 #200 #1000 T a r g e t un l a b e l e d d a t a #0 1.22  49.13  68.53  77.01  #1000 68.95 70.38 73.42 75.59 75.75 76.11 77.47 77.18 #2000 70.94 71.09 75.18 76.15 76.54 76.37 78.48 77.66 #4000 71.78 71.89 76.49 76.41 77.52 76.59 78.77 77.69 All 74.61 74.66 76.56 76.44 78.26 77.07 79.18 77.76 Table 4: Results on different sizes of target unlabeled data and labeled data on De of NER tasks. Cross-lingual Structured Prediction Comparing to single-source transfer, the multi-source transfer shows superior performance by leveraging multi-source language knowledge (McDonald et al., 2011; Rahimi et al., 2019; Hu et al., 2021). However, the diverse quality of source models sorely hurt the target model. To tackle this challenging problem, Ammar et al. (2016) leverage language embeddings to model language topological similarities. Rahimi et al. (2019) utilize truth inference to obtain the best labeling over multiple unreliable predictors. Hu et al. (2021) models the relations between the predicted labels from the source models and the true labels.Approaches based on the similarity of source and target data are widely studied (Chen et al., 2019; Wu et al., 2020). Multi/Cross-view Learning Multi-view learning learns multiple representations for the target data. Tri-training approaches (Zhou and Li, 2005; Ruder and Plank, 2018) leverage voting on three separate models to select confident sentences. Jiang et al. (2019); Cai and Lapata (2020) utilize similarity metrics to regularize source-target language pairs. Multi-view learning can also be utilized in training NER models with different kinds of input components (Wang et al., 2021). Cross-view learning (Clark et al., 2018) is a semi-supervised approach that aims to boost the monolingual model's performance. It learns only one model with several auxiliary prediction modules which are treated as different views. In contrast to it, we focus on the cross-lingual scenario and our two views are a target task-specific model and the aggregation of multiple pre-trained source models. Contextual Multilingual Language Model Trained on massive unlabeled data of hundreds of monolingual corpus, the contextual multilingual models (Devlin et al., 2019; Conneau et al., 2020) learn common representations for multiple languages. Though cross-lingual transfer learning significantly benefits from these models (Pires et al., 2019; Wu and Dredze, 2019b), large gaps still remain between low and high-resources setups (Hu et al., 2020a; Wu and Dredze, 2020). We propose a novel multi-view framework to selectively transfer knowledge from multiple sources by utilizing a small amount of labeled dataset. Experimental results show that our approaches achieve state-of-the-art performances on all tasks. Moreover, even compared to approaches with extra resources like source language data, our sub-structure-level approach still shows significant improvements. This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program. We thank Yuting Zhen for her support in processing datasets and conducting sig-nificance tests.
abstain abstain abstain objective result result abstain other abstain abstain abstain abstain method abstain abstain abstain objective result abstain abstain abstain abstain abstain abstain objective objective abstain abstain objective objective objective abstain objective result method objective abstain method abstain abstain abstain abstain method other abstain other abstain method method abstain abstain abstain abstain abstain method abstain method other abstain abstain method method abstain abstain method abstain abstain other abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain other result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other objective other other objective result result other other