Recent years have seen the paradigm shift of Named Entity Recognition (NER) systems from sequence labeling to span prediction . Despite its preliminary effectiveness, the span prediction model's architectural bias has not been fully understood. In this paper, we first investigate the strengths and weaknesses when the span prediction model is used for named entity recognition compared with the sequence labeling framework and how to further improve it, which motivates us to make complementary advantages of systems based on different paradigms. We then reveal that span prediction, simultaneously, can serve as a system combiner to re-recognize named entities from different systems' outputs. We experimentally implement 154 systems on 11 datasets, covering three languages, comprehensive results show the effectiveness of span prediction models that both serve as base NER systems and system combiners. We make all code and datasets available: https:// github.com/neulab/spanner , as well as an online system demo: http://spanner. sh . Our model also has been deployed into the EXPLAINABOARD (Liu et al., 2021) platform, which allows users to flexibly perform the system combination of top-scoring systems in an interactive way: http://explainaboard. nlpedia.ai/leaderboard/task-ner/ . The rapid evolution of neural architectures (Kalch-brenner et al., 2014a; Kim, 2014; Hochreiter and Schmidhuber, 1997) and large pre-trained models (Devlin et al., 2019; Lewis et al., 2020) not only drive the state-of-the-art performance of many NLP tasks (Devlin et al., 2019; Liu and Lapata, 2019) to a new level but also change the way how researchers formulate the task. For example, recent years have seen frequent paradigm shifts for the task of named entity recognition (NER) from token-level tagging , which conceptualize NER as a sequence labeling (SEQLAB ) task (Chiu and Nichols, 2015; Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016; Akbik et al., 2018; Peters et al., 2018; Devlin et al., 2018; Xia et al., 2019; Luo et al., 2020; Lin et al., 2020; Fu et al., 2021), to span-level prediction (SPANNER ) (Li et al., 2020; Mengge et al., 2020; Jiang et al., 2020; Ouchi et al., 2020; Yu et al., 2020), which regards NER either as question answering (Li et al., 2020; Mengge et al., 2020), span classification (Jiang et al., 2020; Ouchi et al., 2020; Yamada et al., 2020), and dependency parsing tasks (Yu et al., 2020). However, despite the success of span prediction-based systems, as a relatively newly-explored framework, the understanding of its architectural bias has not been fully understood so far. For example, what are the complementary advantages compared with SEQLAB frameworks and how to make full use of them? Motivated by this, in this paper, we make two scientific contributions. We first investigate what strengths and weaknesses are when NER is conceptualized as a span prediction task . To achieve this goal, we perform a fine-grained evaluation of SPANNER systems against SEQLAB systems and find there are clear complementary advantages between these two frameworks. For example, SEQLAB -based models are better at dealing with those entities that are long and with low label consistency. By contrast, SPANNER systems do better in sentences with more Out-of-Vocabulary (OOV) words and entities with medium length (3.3). Secondly, we reveal the unique advantage brought by the architectural bias of the span prediction framework : it can not only be used as a base system for named entity recognition but also serve as a meta-system to combine multiple NER systems' outputs . In other words, the span prediction model play two roles showing in Fig. 1: (i) as a base NER system; and (ii) as a system combiner of multiple base systems. We claim that compared with traditional ensemble learning of the NER task, SPANNER combiners are advantageous in the following aspects: 1. Most of the existing NER combiners rely on heavy feature engineering and external knowledge (Florian et al., 2003; Wu et al., 2003; Saha and Ekbal, 2013). Instead, the SPANNER models we proposed for system combination train in an end-to-end fashion. 2. Combining complementarities of different paradigms: most previous works perform NER system combination solely focusing on the sequence labeling framework. It is still an understudied topic how systems from different frameworks help each other. 3. No extra training overhead and flexibility of use: Existing ensemble learning algorithms are expensive, which usually need to collect training samples by k-fold cross-validation for system combiner (Speck and Ngomo, 2014), reducing their practicality. 4. Connecting two separated training processes: previously, the optimization of base NER systems and ensemble learning for combiner are two independent processes. Our work builds their connection and the same set of parameters shared over these two processes. Experimentally, we first implement 154 systems on 11 datasets, on which we comprehensively evaluate the effectiveness of our proposed span prediction-based system combiner. Empirical results show its superior performance against several typical ensemble learning algorithms. that benefits from the practicality of our proposed methods . Specifically, we developed an online demo system based on our proposed method, and integrate it into the NER Leaderboard, which is very convenient for researchers to find the complementarities among different combinations of systems, and search for a new state-of-the-art system. NER is frequently formulated as a sequence labeling (SEQLAB ) problem (Chiu and Nichols, 2015; Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016), where X = { x 1 , x 2 , . . . , x T } is an input sequence and Y = { y 1 , y 2 , . . . , y T } is the output label (e.g., B-PER, I-LOC, O) sequence. The goal of this task is to accurately predict entities by assigning output label y t for each token x t . We take the F1-score 1 as the evaluation metric for the NER task. To make a comprehensive evaluation, in this paper, we use multiple NER datasets that cover different domains and languages. CoNLL-2003 2 (Sang and De Meulder, 2003) covers two different languages: English and German. Here, we only consider the English ( EN ) dataset collected from the Reuters Corpus. CoNLL-2002 3 (Sang, 2002) contains annotated corpus in Dutch ( NL ) collected from De Morgen news, and Spanish ( ES ) collected from Spanish EFE News Agency. We evaluate both languages. OntoNotes 5.0 4 (Weischedel et al., 2013) is a large corpus consisting of three different languages: English, Chinese, and Arabic, involving six genres: newswire ( NW ), broadcast news ( BN ), broadcast conversation ( BC ), magazine ( MZ ), web data ( WB ), and telephone conversation ( TC ). Following previous works (Durrett and Klein, 2014; Ghaddar and Langlais, 2018), we utilize different domains in English to test the robustness of proposed models. 1 http://www.clips.uantwerpen.be/ conll2000/chunking/conlleval.txt 2 https://www.clips.uantwerpen.be/ conll2003/ner/ 3 https://www.clips.uantwerpen.be/ conll2002/ner/ 4 https://catalog.ldc.upenn.edu/ LDC2013T19 WNUT-2016 5 and WNUT-2017 6 (Strauss et al., 2016; Derczynski et al., 2017) are social media data from Twitter, which were public as a shared task at WNUT-2016 ( W16 ) and WNUT-2017 ( W17 ). Although this is not the first work that formulates NER as a span prediction problem (Jiang et al., 2020; Ouchi et al., 2020; Yu et al., 2020; Li et al., 2020; Mengge et al., 2020), we contribute by (1) exploring how different design choices influence the performance of SPANNER and (2) interpreting complementary strengths between SEQLAB and SPANNER with different design choices. In what follows, we first detail span prediction-based NER systems with the vanilla configuration and proposed advanced featurization. Overall, the span prediction-based framework for NER consists of three major modules: token representation layer, span representation layer, and span prediction layer. Given a sentence X = { x 1 ,    , x n } with n tokens, the token representation h i is as follows: where EMB (  ) is the pre-trained embeddings, such as non-contextualized embeddings GloVe (Pen-nington et al., 2014) or contextualized pre-trained embeddings BERT (Devlin et al., 2018). BILSTM is the bidirectional LSTM (Hochreiter and Schmid-huber, 1997). First, we enumerate all the possible m spans S = { s 1 ,    , s i ,    , s m } for sentence X = { x 1 ,    , x n } and then re-assign a label y  Y for each span s . For example, for sentence: London 1 is 2 beautiful 3 , the possible span's (start, end) indices are { (1 , 1) , (2 , 2) , (3 , 3) , (1 , 2) , (2 , 3) , (1 , 3) } , and the labels of these spans are all O except (1 , 1) (London) is LOC. We use b i and e i to denote the startand endindex of the span s i , respectively, 5 http://noisy-text.github.io/2016/ ner-shared-task.html 6 http://noisy-text.github.io/2017/ emerging-rare-entities.html and 1  b i  e i  n . Then each span can be represented as s i = { x b i , x b i +1 ,    , x e i } . The vectorial representation of each span could be calculated based on the following parts: Boundary embedding: span representation is calculated by the concatenation of the start and end tokens' representations z bi = [ h b i ; h e i ] Span length embedding: we additionally featur-ize each span representation by introducing its length embedding z li , which can be obtained by a learnable look-up table. where score(  ) is a function that measures the compatibility between a specified label and a span: Heuristic Decoding Regarding the flat NER task without nested entities, we present a heuristic decoding method to avoid the prediction of overlapped spans. Specifically, for those overlapped spans, we keep the span with the highest prediction probability and drop the others. Setup To explore how different mechanisms influence the performance of span prediction models, We design four specific model variants (i) generic SPANNER : only using boundary embedding (ii) boundary embedding + span length embedding, (iii) boundary embedding + heuristic decoding, (iv) heuristic decoding + (ii). Results As shown in Tab. 1, we can observe that: (i) heuristic decoding is an effective method that can boost the generic model's performance over all the datasets. (ii) span length feature works most of the time. The performances on 10 of the 11 datasets have improved against the generic model. (iii) By combining two mechanisms together, significant improvements were achieved on all datasets. The holistic results in Tab. 1 make it hard for us to interpret the relative advantages of NER systems with different structural biases. To address this problem, we follow the interpretable evaluation idea (Fu et al., 2020a,c) that proposes to breakdown the holistic performance into different buckets from different perspectives and use a performance heatmap to illustrate relative advantages between two systems, i.e., system-pair diagnosis. Setup As a comparison, we replicate five top-scoring SEQLAB -based NER systems, which are sq 1 : 92 . 41 , sq 2 : 92 . 01 , sq 3 : 92 . 46 , sq 4 : 92 . 11 , sq 5 : 91 . 99 . Notably, to make a fair comparison, all five SEQLAB s are with closed performance comparing to the above SPANNER s. Although we will detail configurations of these systems later (to reduce content redundancy) in 5.1 Tab. 3 , it would not influence our analysis in this section. Regarding interpretable evaluation, we choose the CoNLL-2003 (EN) dataset as a case study and breakdown the holistic performance into four groups based on different attributes. Specifically, given an entity e that belongs to a sentence S , the following attribute feature functions can be defined:   eLen = len( e ) : entity length   sLen = len( S ) : sentence length   oDen = | OOVs | len( S ) : density of OOVs   eCon = |{  | label(  )=label( e ) ,   E}| |E| : entity label consistency where len(  ) counts the number of words, label( e ) gets the label for span e , E denotes all spans in the training set. | OOVs | is the number of OOV words in the sentence. We additionally use a training set dependent attribute: entity label consistency ( eCon ), which measures how consistently a particular entity is labeled with a particular label. For example, if an entity with the label LOC has a higher eCon , it means that the entity is frequently labeled as LOC in the training set. Based on the attribute value of entities, we partition test entities into four buckets: extra-small (XS), small (S), large (L), and extra-large (XL). 7 . For each bucket, we calculate a 7 we show detailed bucket intervals in the appendix bucket-wise F1. Analysis As shown in Tab. 2, the green area indicates SEQLAB performs better while the red area implies the span model is better. We observe that: (1) The generic SPANNER shows clear complementary advantages with SEQLAB -based systems. Specifically, almost all SEQLAB -based models outperform generic SPANNER when (i) entities are long and with lower label consistency (ii) sentences are long and with fewer OOV words. By contrast, SPANNER is better at dealing with entities locating on sentences with more OOV words and entities with medium length. (2) By introducing heuristic decoding and span length features, SPANNER s do slightly better in long sentences and long entities, but are still under-performing on entities with lower label consistency. The complementary advantages presented by SEQLAB s and SPANNER s motivate us to search for an effective framework to utilize them. The development of ensemble learning for NER systems, so far, lags behind the architectural evolution of the NER task. Based on our evidence from 3.3, we propose a new ensemble learning framework for NER systems. SPANNER as System Combiner The basic idea is that each span prediction NER (SPANNER ) system itself can also conceptualize as a system combiner to re-recognize named entities from different systems' outputs. Specifically, Fig. 2 illustrates the general workflow. Here, SPANNER plays two roles, (1) as a base model to identify potential named entities; (2) as a meta-model (combiner) to calculate the score for each potential named entity. Given a test span s and prediction label set  L from m base systems ( |  L| = m ). Let L be NER label set where |L| = c and c is the number of pre-defined NER classes (i.e., LOC, ORG, PER, MISC, O in CoNLL 2003 (EN) .) For each l  L we define P ( s, l ) as the combined probability that span s can be assigned as label l , which can be calculated as: P ( s, l ) = (cid:88)  l   L  l = l score( s,  l ) , (5) where score(  ) is defined as Eq.4. Intuitively, Fig. 2 gives an example of how SPANNER re-recognizes the entity  New York  based on outputs from four base systems. As a base system, SPANNER predicts this span as LOC, and the label will be considered as one input of the combiner model. The prediction labels of the other three base models are LOC, ORG, and O, respectively. Then, as a combiner, SPANNER calculates the score for each predicted label. We sum weights (scores) of the same label that are predicted by the base models and select the label that achieves the maximum score as the output of the combiner. To make a thorough evaluation of SPANNER as a system combiner, as illustrated in Tab. 3, we first implement 10 SEQLAB based systems that cover rich combinations of popular neural components. To be fair for other system combination methods, we also include two SPANNER s as base systems. To reduce the uncertainty, we run experiments with multiple trials and also perform the significant test with Wilcoxon Signed-Rank Test (Wilcoxon et al., 1970) at p < 0 . 05 . We keep the testing result from the model with the best performance on the development set, terminating training when the performance of the development set is not improved in 20 epochs. We extensively explore six system combination methods as competitors, which involves supervised and unsupervised fashions. Weighted voting base on overall F1-score (VOF1): The taggers are combined according to 9 We view BERT as the subword-sensitive representation because we get the representation of each subword. testing set. Weighted voting base on class F1-score (VCF1): Also weighted voting, the weights are the cate-gories' F1-score. Stacking (a.k.a, Stacked Generalization) is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy (Ting and Witten, 1997). To make a comprehensive evaluation, we investigated three popular methods that are supervised learning , thereby requiring additional training samples. Specifically, there are: Support Vector Machines (SVM) (Hearst et al., 1998) is a supervised machine learning algorithm, which can train quickly over large datasets. Therefore, the ensemble classifier is usually SVM. Random Forest (RF) (Breiman, 2001) is a common ensemble classifier that randomly selects a subset of training samples and variables to make multiple decision trees. Extreme Gradient Boosting (XGB) (Chen and Guestrin, 2016) is also an ensemble machine learning algorithm. It is based on the decision-tree and the gradient boosting decision (Friedman et al., 2000). Notably, compared to our model, these methods are computationally expensive since they require external training samples for system combiners, Cm Best SpNER Voting-based Stacking-based Best SpNER Voting-based Stacking-based VM  VOF1  VCF1  SVM  RF  XGB  VM  VOF1  VCF1  SVM  RF  XGB  CoNLL-2003 (EN) OntoNotes5.0-BN (BN) all 93.02 93.80 93.62 93.57 93.60 93.28 93.04 92.93 90.93 91.14 90.92 91.29 91.12 89.67 90.95 90.50 m[:10] 93.02 93.78 93.48 93.55 93.54 93.21 93.03 93.18 90.93 91.48 91.03 91.41 91.27 89.97 90.92 90.91 m[:9] 93.02 93.81 93.57 93.59 93.51 93.33 93.26 93.35 90.93 91.64 91.16 91.24 91.22 90.16 90.75 90.76 m[:8] 93.02 93.81 93.41 93.52 93.54 93.28 93.17 93.14 90.93 91.74 91.17 91.59 91.39 90.16 90.69 90.81 m[:7] 93.02 93.72 93.41 93.47 93.41 93.26 92.98 93.00 90.93 91.86 91.60 91.66 91.57 90.16 90.80 90.73 m[:6] 93.02 93.71 93.21 93.63 93.53 93.20 93.27 93.21 90.93 91.95 91.42 91.74 91.67 90.34 91.09 91.04 m[:5] 93.02 93.80 93.46 93.54 93.52 93.33 93.19 93.28 90.93 90.65 91.69 91.77 91.97 90.54 90.72 90.69 m[:4] 93.02 93.70 93.29 93.69 93.61 93.47 93.20 93.28 90.93 90.30 91.18 91.13 90.32 90.02 90.93 90.77 m[:3] 93.02 93.75 93.66 93.75 93.61 93.30 93.38 93.43 90.93 91.13 91.07 91.13 91.13 90.89 90.98 91.08 m[:2] 93.02 93.01 93.02 92.99 92.95 92.74 92.86 92.86 90.93 89.81 90.70 89.78 90.01 90.61 90.98 90.81 m[2:4] 92.41 93.66 92.41 92.46 92.78 92.32 92.37 92.51 90.53 90.23 88.54 90.53 89.10 89.38 90.26 90.18 m[4:6] 92.11 93.39 92.01 92.11 92.31 92.01 92.15 92.17 89.43 90.80 89.33 89.43 89.77 89.66 89.49 89.99 m[3:6] 92.28 93.04 92.97 92.92 92.95 92.18 92.52 92.46 89.66 90.98 90.82 90.96 90.90 89.48 90.59 90.56 m[1:] 92.46 93.68 93.59 93.54 93.55 93.07 92.83 93.00 90.70 91.21 90.81 90.94 90.91 89.50 90.90 90.56 m[2:] 92.41 93.58 93.43 93.40 93.34 93.06 92.96 92.89 90.53 90.97 90.54 90.86 90.74 89.29 90.72 90.53 m[3:] 92.28 93.58 93.35 93.41 93.35 93.09 92.81 92.81 89.66 90.71 90.25 90.38 90.31 89.05 90.26 90.10 m[4:] 92.11 93.50 92.86 93.21 93.10 92.88 92.79 92.68 89.43 90.70 89.46 89.89 89.84 89.10 89.20 89.05 m[5:] 92.01 93.54 92.67 92.84 92.78 92.81 92.85 92.65 89.33 90.39 89.46 89.42 89.58 88.61 89.32 88.93 m[6:] 91.99 93.32 91.85 92.51 92.34 92.16 92.58 92.51 89.21 89.94 88.51 89.27 89.08 88.56 88.75 88.57 m[7:] 91.57 92.66 90.92 91.55 91.29 91.93 92.20 92.02 89.03 89.42 88.33 88.33 88.62 88.00 87.87 87.86 m[8:] 90.88 91.29 87.39 90.65 90.32 90.98 90.90 90.83 86.84 88.52 86.50 87.61 87.19 86.35 87.14 87.10 m[9:] 89.71 91.21 85.97 87.31 86.27 89.68 89.50 89.56 86.18 88.36 85.87 86.27 86.20 85.34 86.20 86.01 m[10:] 83.03 85.65 78.49 83.03 81.83 83.06 83.17 83.17 83.87 86.52 81.05 83.87 83.39 82.25 83.94 83.86 Std. which is achieved by (i) collecting training data by performing five-fold cross-validation (Wu et al., 2003; Florian et al., 2003) on the original training samples of each dataset (ii) training a system combiner based on collected samples. Setup Most previous works on system combination only consider one combination case where all base systems are put together. In this setting, we aim to explore more fine-grained combination cases. Specifically, we first sort systems based on their performance in a descending order to get a list m . We refer to m [ i : k ] as one combination case, dubbed combined interval , which represents systems whose ranks are between i and k . In practice, we consider 23 combination cases showing in Tab. 4. To examine whether the SPANNNER is significantly better than the other baseline methods, we conduct the significance test with Wilcoxon Signed-RankTest (Wilcoxon et al., 1970) at p < 0 . 05 . Results Tab. 4 shows results of our SPANNER against six baseline combiner methods on CoNLL-2003 and OntoNotes5.0-BN under a nuanced view. We can observe that: (1) Overall, our proposed SPANNER outperforms all other competitors significantly (p-value < 0.05) on most of the combination cases include the one ( all ) that most previous works have explored. (2) As more base systems are introduced in descending order, the combined performance will be improved gradually. The combination performance will decrease with the reduction of the best single system, which holds for all the combiners. (3) The best performance is always achieved on the combination case with more models, instead of the one with a small number of top-scoring base models. This suggests that introducing more base models with diverse structures will provide richer complementary information. Setup To also explore the effectiveness of SPANNER on the other datasets, we calculate the average performance of each system combination method over 23 combination cases. Results Tab. 5 shows the results, and we can observe that: comparing with the three voting combiner, SPANNER achieves the best average combination performance with the lowest standard deviation, which holds for seven of nine testing datasets with statistical significance p < 0.05. Specifically, the performance gap between SPANNER and other combiners is larger on datasets from web domain: WB and Twitter: W16 , W17 . Setup The above experiments have shown the superior performance of SPANNER on system combination. To further investigate where the gains of the SPANNER come from, similar to 3.3, we perform fine-grained evaluation on CoNLL-2003 dataset using one combination case to interpret how SPANNER outperform other (i) base systems and (ii) other baseline combiners . The combination case contains base systems: sq 0 5 together with sp 1 , sp 2 (model's detail can refer to Tab.3). Results As shown in Tab. 6, we can find: (1) SPANNER v.s. Base systems : the improvements of all base systems largely come from entities with low label consistency ( eCon : XS, S). Particularly, base systems with SEQLAB benefit a lot from short entities while base systems with SPANNER gain mainly from long entities. (2) SPANNER v.s. Other combiners : as a system combiner, the improvement of SPANNER against other baselines mainly comes from low label consistency ( eCon : XS, S). By contrast, traditional combiners surpass SPANNER when dealing with long sentences ( sLen : XL). NER as Different Tasks Although NER is commonly formulated as a sequence labeling task (Chiu and Nichols, 2015; Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016; Akbik et al., 2018; Peters et al., 2018; Devlin et al., 2018; Xia et al., 2019; Akbik et al., 2019; Luo et al., 2020; Lin et al., 2020), recently other new forms of frameworks have been explored and have shown impressive results. For example, (Jiang et al., 2020; Ouchi et al., 2020; Yu et al., 2020) shift NER from token-level tagging to span-level prediction task while (Li et al., 2020; Mengge et al., 2020) conceptualize it as reading comprehension task. In this work we aim to interpret the complementarity between sequence labeling and span prediction. System Combination Traditionally, system combination was used to improve the performance of statistical MT systems (Gonzalez-Rubio et al., 2011; Watanabe and Sumita, 2011; Duh et al., 2011; Mizumoto and Matsumoto, 2016). Some recent work (Zhou et al., 2017; Huang et al., 2020) also extended this method to neural MT where the meta-model and base systems are all neural models. There is a handful of works about system combination for NER. (Wu et al., 2003; Florian et al., 2003) investigated stacking and voting methods for combining strong classifiers. Ekbal and Saha (2011); Zhang et al. (2014) proposes a weighted voting approach based on differential evolution. These works commonly require training samples and rely heavily on feature engineering. Co-evolution of NLP Systems and their combiners Systems for NLP tasks (e.g., NER model) and their combiners (e.g., ensemble learning for NER) are developing in two parallel directions. This paper builds the connection between them and proposes a model that can be utilized as both a base NER system and a system combiner. Our work opens up a direction toward making the algorithms of NLP models and system combination co-evolved. The unified idea can be applied to other NLP tasks, and some traditional methods like reranking in syntactic parsing can be re-visited . For example, we can formulate constituency parsing (Jiang et al., 2020) as well as its re-ranking (Collins and Koo, 2005; Huang, 2008) as a span prediction (Stern et al., 2017) problem, which is be unified and parameterized with the same form. CombinaBoard It has become a trend to use a Leaderboard (e.g., paperwithcode 10 ) to track current progress in a particular field, especially with the rapid emergence of a plethora of models. Leaderboard makes us pay more attention to and even obsess over the state-of-the-art systems (Ethayarajh and Jurafsky, 2020). We argue that Leaderboard with an effective system combination (dubbed COMBINABOARD ) feature would allow researchers to quickly find the complementarities among different systems. As a result, the value of a worse-ranked model still could be observed through its combined results. In this paper, we make the first step towards this by releasing a preliminary COMBINABOARD for the NER task http://spanner.sh . Our model also has been deployed into the EXPLAINABOARD (Liu et al., 2021) platform, which allows users to flexibly perform system combination of top-scoring systems in an interactive way: http://explainaboard. nlpedia.ai/leaderboard/task-ner/ 10 https://paperswithcode.com/ Acknowledgements We thank Professor Graham Neubig and anonymous reviewers for valuable feedback and helpful suggestions. This work was supported by the Air Force Research Laboratory under agreement number FA8750-19-2-0200. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.
abstain abstain objective result result other abstain other abstain other abstain other abstain abstain objective objective objective abstain abstain result abstain abstain abstain method objective objective objective objective objective method objective abstain objective objective other abstain method method abstain abstain method abstain method abstain abstain other other abstain abstain abstain abstain abstain abstain other other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain other other objective other other other other other other abstain abstain abstain abstain method abstain abstain abstain abstain other other other abstain abstain abstain