This work aims to tackle the challenging heterogeneous graph encoding problem in the text-to-SQL task. Previous methods are typically node-centric and merely utilize different weight matrices to parameterize edge types, which 1) ignore the rich semantics embedded in the topological structure of edges, and 2) fail to distinguish local and nonlocal relations for each node. To this end, we propose a L ine G raph E nhanced Text-to-SQL (LGESQL) model to mine the underlying relational features without constructing meta-paths. By virtue of the line graph, messages propagate more efficiently through not only connections between nodes, but also the topology of directed edges. Furthermore, both local and non-local relations are integrated distinctively during the graph iteration. We also design an auxiliary task called graph pruning to improve the discriminative capability of the encoder. Our framework achieves state-of-the-art results ( 62 . 8% with GLOVE , 72 . 0% with ELECTRA ) on the cross-domain text-to-SQL benchmark Spider at the time of writing. The text-to-SQL task (Zhong et al., 2017; Xu et al., 2017) aims to convert a natural language question into a SQL query, given the corresponding database schema. It has been widely studied in both academic and industrial communities to build natural language interfaces to databases (NLIDB, Androut-sopoulos et al., 1995). One daunting problem is how to jointly encode the question words and database schema items (in-cluding tables and columns), as well as various relations among these heterogeneous inputs. Typically, previous literature utilizes a node-centric graph neural network (GNN, Scarselli et al., 2008)  The corresponding authors are Lu Chen and Kai Yu. to aggregate information from neighboring nodes. GNNSQL (Bogin et al., 2019a) adopts a relational graph convolution network (RGCN, Schlichtkrull et al., 2018) to take into account different edge types between schema items, such as T-HAS-C relationship 1 , primary key and foreign key constraints. However, these edge features are directly retrieved from a fixed-size parameter matrix and may suffer from the drawback: unaware of con-textualized information, especially the structural topology of edges. Meta-path is defined as a composite relation linking two objects, which can be used to capture multi-hop semantics. For example, in Figure 1(a), relation Q-EXACTMATCH-C and C-BELONGSTO-T can form a 2-hop meta-path indicating that some table t has one column exactly mentioned in the question. 1 For abbreviation, Q represents QUESTION node, while T and C represent TABLE and COLUMN nodes. or multi-hop, in the same manner (relative position embedding, Shaw et al., 2018) in a complete graph. Without distinguishing local and non-local neighbors, see Figure 1(b), each node will attend to all the other nodes equally, which may lead to the notorious over-smoothing problem (Chen et al., 2020a). Besides, meta-paths are currently constructed by domain experts or explored by breadth-first search (Kong et al., 2012). Unfortunately, the number of possible meta-paths increases exponentially with the path length, and selecting the most important subset among them is an NP-complete problem (Lao and Cohen, 2010). To address the above limitations, we propose a L ine G raph E nhanced Text-toSQL model (LGESQL), which explicitly considers the topological structure of edges. According to the definition of a line graph (Gross and Yellen, 2005), we firstly construct an edge-centric graph from the original node-centric graph. These two graphs capture the structural topology of nodes and edges, respectively. Iteratively, each node in either graph gathers information from its neighborhood and incorporates edge features from the dual graph to update its representation. As for the node-centric graph, we combine both local and non-local edge features into the computation. Local edge features denote 1 -hop relations and are dynamically provided by node embeddings in the line graph, while non-local edge features are directly extracted from a parameter matrix. This distinction encourages the model to pay more attention to local edge features while maintaining information from multihop neighbors. Additionally, we propose an auxiliary task called graph pruning . It introduces an inductive bias that the heterogeneous graph encoder of text-to-SQL should be intelligent to extract the golden schema items related to the question from the entire database schema graph. Experimental results on benchmark Spider (Yu et al., 2018b) demonstrate that our LGESQL model promotes the exact set match accuracy to 62 . 8% (with GLOVE , Pennington et al. 2014) and 72 . 0% (with pretrained language model ELECTRA , Clark et al. 2020). Our main contributions are summarized as follows:  We propose to model the 1 -hop edge features with a line graph in text-to-SQL. Both nonlocal and local features are integrated during the iteration process of node embeddings. We design an auxiliary task called graph pruning , which aims to determine whether each node in the database schema graph is relevant to the given question. Empirical results on dataset Spider demonstrate that our model is effective, and we achieve state-of-the-art performances both without and with pre-trained language models. Problem definition Given a natural language question Q = ( q 1 , q 2 ,    , q | Q | ) with length | Q | and the corresponding database schema S = T  C , the target is to generate a SQL query y . The database schema S contains multiple tables T = { t 1 , t 2 ,    } and columns C = { c t 1 1 , c t 1 2 ,    , c t 2 1 , c t 2 2 ,    } . Each table t i is described by its name and is further composed of several words ( t i 1 , t i 2 ,    ) . Similarly, we use word phrase ( c t i j 1 , c t i j 2 ,    ) to represent column c t i j  t i . Besides, each column c t i j also has a type field c t i j 0 to constrain its cell values (e.g. TEXT and NUMBER ). The entire input node-centric heterogeneous graph G n = ( V n , R n ) consists of all three types of nodes mentioned above, that is V n = Q  T  C with the number of nodes | V n | = | Q | + | T | + | C | , where | T | and | C | are the number of tables and columns respectively. Meta-path As shown in Figure 1(a), a meta-path represents a path  1 r 1   2 r 2     r l   l +1 , where the target vertex type of previous relation r i  1 equals to the source vertex type  i of the current relation r i . It describes a composite relation r = r 1  r 2    r l between nodes with type  1 and  l +1 . In this work,  i  { QUESTION ,T ABLE ,C OLUMN } . Throughout our discussion, we use the term local to denote relations with path length 1 , while nonlocal relations refer to meta-paths longer than 1 . The relational adjacency matrix R n contains both local and non-local relations, see Appendix A for enumeration. Line Graph Each vertex v ei , i = 1 , 2 ,    , | V e | in the line graph G e = ( V e , R e ) can be uniquely mapped to a directed edge r nst  R n , or v ns  v nt , in the original node-centric graph G n = ( V n , R n ) . Function f maps the source and target node index tuple ( s, t ) into the edge index i = f ( s, t ) in G e . The reverse mapping is f 1 . In the line graph G e , a directed edge r eij  R e exists from node v ei to v ej , iff the target node of edge r nf 1 ( i ) and the source node of edge r nf 1 ( j ) in G n are exactly the same node. Actually, r eij captures the information flow in meta-path r nf 1 ( i )  r nf 1 ( j ) . We prevent backtracking cases where two reverse edges will not be connected in G e , illustrated in Figure 2. We only utilize local relations in R n as the node set V e to avoid creating too many nodes in the line graph G e . Symmetrically, each edge in R e can be uniquely identified by the node in V n . For example, in the upper right part of Figure 2, the edge between nodes e1 and e2 in the line graph can be represented by the middle node with double solid borderlines in the original graph. After constructing the line graph, we utilize the classic encoder-decoder architecture (Sutskever et al., 2014; Bahdanau et al., 2015) as the backbone of our model. LGESQL consists of three parts: a graph input module, a line graph enhanced hidden module, and a graph output module (see Figure 3 for an overview). The first two modules aim to map the input heterogeneous graph G n into node embeddings X  R | V n | d , where d is the graph hidden size. The graph output module retrieves and transforms X into the target SQL query y . This module aims to provide the initial embeddings for both nodes and edges. Initial local edge features Z 0  R | V e | d and non-local edge features Z nlc  R ( | R n || V e | )  d are directly retrieved from a parameter matrix. For nodes, we can obtain their representations from either word vectors GLOVE (Pennington et al., 2014) or a pre-trained language model (PLM) such as BERT (Devlin et al., 2019). GLOVE Each word q i in the question Q or schema item t i  T or c t i j  C can be initialized by looking up the embedding dictionary without considering the context. Then, these vectors are passed into three type-ware bidirectional LSTMs (BiL-STM, Hochreiter and Schmidhuber, 1997) respectively to attain contextual information. We concatenate the forward and backward hidden states for each question word q i as the graph input x 0 q i . As for table t i , after feeding ( t i 0 , t i 1 , t i 2 ,    ) into the BiLSTM (special type t i 0 =  table  ,  i ), we concatenate the last hidden states in both directions as the graph input x 0 t i (similarly for column c t i j ). These node representations are stacked together to form the initial node embeddings matrix X 0  R | V n | d . PLM Firstly, we flatten all question words and schema items into a sequence, where columns belong to the same table are clustered together 2 : [CLS] q 1 q 2    q | Q | [SEP] t 10 t 1 c t 1 10 c t 1 1 c t 1 20 c t 1 2    t 20 t 2 c t 2 10 c t 2 1 c t 2 20 c t 2 2    [SEP] . The type information t i 0 or c t i j 0 is inserted before each schema item. Since each word w is tokenized into sub-words, we append a subword attentive pooling layer after PLM to obtain word-level representations. Concretely, given the output sequence of subword features w s 1 , w s 2 ,    , w s | w | for each subword w si in w , the word-level representation w is 3 a i = softmax i tanh ( w si W s ) v T s , w = (cid:88) i a i w si , where v s and W s are trainable parameters. After obtaining the word vectors, we also feed them into three BiLSTMs according to the node types and get the graph inputs X 0 for all nodes. It contains a stack of L dual relational graph attention network (Dual RGAT) layers. In each layer l , two RGATs (Wang et al., 2020b) capture the structure of the original graph and line graph, respectively. Node embeddings in one graph play the role of edge features in another graph. For example, the edge features used in graph G n are provided by the node embeddings in graph G e . We use X l  R | V n | d to denote the input node embedding matrix of graph G n in the l -th 2 We randomly shuffle the order of tables and columns in different mini-batches to discourage over-fitting. layer, l  { 0 , 1 ,    , L  1 } . As for each specific node v ni  V n , we use x li . Similarly, matrix Z l  R | V e | d and vector z li are used to denote node embeddings in the line graph. Following RATSQL (Wang et al., 2020a), we use multi-head scaled dot-product (Vaswani et al., 2017) to calculate the attention weights. For brevity, we formulate the entire computation in one layer as two basic modules: X l +1 = RGAT n ( X l , [ Z l ; Z nlc ] , G n ) , Z l +1 = RGAT e ( Z l , X l , G e ) , where Z nlc is the aforementioned non-local edge features in the original graph G n . Given the node-centric graph G n , the output representation x l +1 i of the l -th layer is computed by hji =( x li W hq )( x lj W hk + [  ( r nji )] Hh ) T ,  hji = softmax j (  hji / (cid:112) d/H ) ,  x li = H (cid:110) h =1 (cid:88) v nj N ni  hji ( x lj W hv + [  ( r nji )] Hh ) ,  x l +1 i = LayerNorm ( x li +  x li W o ) , x l +1 i = LayerNorm ( x l +1 i + FFN ( x l +1 i )) , where (cid:107) represents vector concatenation, matrices W hq , W hk , W hv  R d  d/H , W o  R d  d are trainable parameters, H is the number of heads and FFN (  ) denotes a feedforward neural network. N ni represents the receptive field of node v n i and function  ( r nji ) returns a d -dim feature vector of relation r nji . Operator [  ] Hh first evenly splits the vector into H parts and returns the h -th partition. Since there are two genres of relations (local and non-local), we design two schemes to integrate them: Mixed Static and Dynamic Embeddings If r nji is a local relation,  ( r nji ) returns the node embedding z lf ( j,i ) from the line graph 4 . Otherwise,  ( r nji ) directly retrieves the vector from the non-local embedding matrix Z nlc , see Figure 4. The neighborhood function N ni for node v ni returns the entire node set V n and is shared across different heads. Multi-head Multi-view Concatenation An alternative is to split the muli-head attention module into two parts. In half of the heads, the neighborhood function N ni of node v ni only contains nodes that are reachable within 1 -hop. In this case,  ( r n ji ) returns the layer-wise updated feature z lf ( j,i ) from 4 Function f maps the tuple of source and target node indices in G n into the corresponding node index in G e . Z l . In the other heads, each node has access to both local and non-local neighbors, and  (  ) always returns static entries in the embedding matrix Z nlc  Z 0 , see Figure 5 for illustration. Symmetrically, given edge-centric graph G e , the updated node representation z l +1 i from z li is calculated similarly with little modifications: hji =( z li U hq + [  ( r eji )] Hh )( z lj U hk ) T ,  hji = softmax j (   hji / (cid:112) d/H ) ,  z li = H (cid:110) h =1 (cid:88) v ej N ei  hji ( z lj U hv + [  ( r eji )] Hh ) ,  z l +1 i = LayerNorm ( z li +  z li U o ) , z l +1 i = LayerNorm ( z l +1 i + FFN ( z l +1 i )) . Here  ( r eji ) returns the feature vector of relation r eji in G e . Since we only consider local relations in the line graph, N ei only includes 1 -hop neighbous and  ( r eji ) equals to the source node embedding in X l of edge v ei . Attention that the relational feature is added on the query side instead of the key side when computing attention logits   h ji cause it is irrelevant to the incoming edges. For example, in Figure 3, the connecting nodes of two edge pairs (1 4 , 4 5) and (2 4 , 4 5) are the same node with index 4 . U hq , U hk , U hv  R d  d/H , U o  R d  d are trainable parameters. The output matrices of the final layer L are the desired outputs of the encoder: X = XL , Z = ZL . This module includes two tasks: one decoder for the main focus text-to-SQL and the other one to perform an auxiliary task called graph pruning . We use the subscript to denote the collection of node embeddings with a specific type, e.g., X q is the matrix of all question node embeddings. We adopt the grammar-based syntactic neural decoder (Yin and Neubig, 2017) to generate the abstract syntax tree (AST) of the target query y in depth-first-search order. The output at each decoding timestep is either 1) an APPLYRULE action that expands the current non-terminal node in the partially generated AST, or 2) SELECTTABLE or SELECTCOLUMN action that chooses one schema item x s i from the encoded memory X s = X t  X c . Mathematically, P ( y | X ) = (cid:81) j P ( a j | a <j , X ) , where a j is the action at the j -th timestep. For more implementation details, see Appendix B. 3.3.2 Graph Pruning We hypothesize that a powerful encoder should distinguish irrelevant schema items from golden schema items used in the target query. In Figure 6, the question-oriented schema sub-graph (above the shadow region) can be easily extracted. The intent c 2 and the constraint c 5 are usually explicitly mentioned in the question, identified by dot-product attention mechanism or schema linking. The linking nodes such as t 1 , c 3 , c 4 , t 2 can be inferred by the 1 -hop connections of the schema graph to form a connected component. To introduce this inductive bias, we design an auxiliary task that aims to classify each schema node s i  S = T  C based on its relevance with the question and the sparse structure of the schema graph. Firstly, we compute the context vector  x s i from the question node embeddings X q for each schema node s i via multi-head attention. hji = softmax j ( x s i W hsq )( x q j W hsk ) T (cid:112) d/H ,  x s i =( H (cid:110) h =1 (cid:88) j  hji x q j W hsv ) W so , where W hsq , W hsk , W hsv  R d  d/H and W so  R d  d are network parameters. Then, a biaffine (Dozat and Manning, 2017) binary classifier is used to determine whether the compressed context vector  x s i and the schema node embedding x s i are correlated. Biaffine ( x 1 , x 2 ) = x 1 U s x T2 + [ x 1 ; x 2 ] W s + b s , P gp ( y s i | x s i , X q ) =  ( Biaffine ( x s i ,  x s i )) . The ground truth label y gs i of a schema item is 1 iff s i appears in the target SQL query. The training object can be formulated as L gp =  (cid:88) s i [ y gs i log P gp ( y s i | x s i , X q ) + (1  y gs i ) log(1  P gp ( y s i | x s i , X q ))] . This auxiliary task is combined with the main text-to-SQL task in a multitasking way. Similar ideas (Bogin et al., 2019b; Yu et al., 2020) and other association schemes are discussed in Appendix C. 4 Experiments In this section, we evaluate our LGESQL model in different settings. Codes are public available 5 . Dataset Spider (Yu et al., 2018b) is a large-scale cross-domain zero-shot text-to-SQL benchmark 6 . It contains 8659 training examples across 146 databases in total, and covers several domains from other datasets such as Restaurants (Popescu et al., 2003), GeoQuery (Zelle and Mooney, 1996), Scholar (Iyer et al., 2017), Academic (Li and Ja-gadish, 2014), Yelp and IMDB (Yaghmazadeh et al., 2017) datasets. The detailed statistics are shown in Table 1. We follow the common practice to report the exact set match accuracy on the validation and test dataset. The test dataset contains 2147 samples with 40 unseen databases but is not public available. We submit our model to the organizer of the challenge for evaluation. Implementations We preprocess the questions, table names, and column names with toolkit Stanza (Qi et al., 2020) for tokenization and lemma-tization. Our model is implemented with Py-torch (Paszke et al., 2019), and the original and line graphs are constructed with library DGL (Wang et al., 2019a). Within the encoder, we use GLOVE (Pennington et al., 2014) word embeddings with dimension 300 or pretrained language models (PLMs), BERT (Devlin et al., 2019) or ELECTRA (Clark et al., 2020), to leverage contextual information. With GLOVE , embeddings of the most frequent 50 words in the training set are fixed during training while the remaining will be fine-tuned. The schema linking strategy is borrowed from RATSQL (Wang et al., 2020a), which is also our baseline system. During evaluation, we adopt beam search decoding with beam size 5 . Hyper-parameters In the encoder, the GNN hidden size d is set to 256 for GLOVE and 512 for PLMs. The number of GNN layers L is 8 . In the decoder, the dimension of hidden state, action embedding and node type embedding are set to 512 , 128 and 128 respectively. The recurrent dropout rate (Gal and Ghahramani, 2016) is 0 . 2 for decoder LSTM. The number of heads in multi-head attention is 8 and the dropout rate of features is set to 0 . 2 in both the encoder and decoder. Throughout the experiments, we use AdamW (Loshchilov and Hutter, 2019) optimizer with linear warmup scheduler. The warmup ratio of total training steps is 0 . 1 . For GLOVE , the learning rate is 5 e 4 and the weight decay coefficient is 1 e 4 ; For PLMs, we use smaller leaning rate 2 e 5 ( base ) or 1 e 5 ( large ), and larger weight decay rate 0 . 1 . The optimization of the PLM encoder is carried out more carefully with layer-wise learning rate decay coefficient 0 . 8 . Batch size is 20 and the maximum gradient norm is 5 . The number of training epochs is 100 for GLOVE , and 200 for PLMs respectively. The main results of the test set are provided in Table 2. Our proposed line graph enhanced text-to-SQL (LGESQL) model achieves state-of-the-art results in all configurations at the time of writing. With word vectors GLOVE , the performance increases from 57 . 2% to 62 . 8% , 5 . 6% absolute improvements. With PLM bert-large-wwm , LGESQL also surpasses all previous methods, including the ensemble model, and attains 68 . 3% accuracy. Recently, more advanced approaches all leverage the benefits of larger PLMs, more task adaptive data (text-table pairs), and tailored pretraining tasks. For example, GAP (Shi et al., 2020) designs some task adaptive self-supervised tasks such as column prediction and column recovery to better address the downstream joint encoding problem. We utilize electra-large for its compatibility with our model and achieves 72 . 0% accuracy. Taking one step further, we compare more fine-grained performances of our model to the baseline system RATSQL (Wang et al., 2020a) classified by the level of difficulty in Table 3. We observe that LGESQL surpasses RATSQL across all subdivisions in both the validation and test datasets regardless of the application of a PLM, especially at the Medium and Extra Hard levels. This validates the superiority of our model by exploiting the structural relations among edges in the line graph. In this section, we investigate the contribution of each design choice. We report the average accuracy on the validation dataset with 5 random seeds. Different Components of LGESQL RGATSQL is our baseline system where the line graph is not utilized. It can be viewed as a variant of RATSQL with our tailored grammar-based decoder. From Table 4, we can discover that: 1) if non-local relations or meta-paths are removed (w/o NLC), the performance will decrease roughly by 2 points in LGESQL, while 3 points drop in RGATSQL. However, our LGESQL with merely local relations is still competitive. It consolidates our motivation that by exploiting the structure among edges, the line graph can capturing long-range relations to some extent. 2) graph pruning task contributes more in LGESQL ( +1 . 2% ) than RGATSQL ( +0 . 7% ) on account of the fact that local relations are more critical to structural inference. 3) Two strategies of combining local and non-local relations introduced in  3.2.1 (w/ MSDE or MMC) are both beneficial to the eventual performances of LGESQL ( 2 . 0% and 2 . 1% gains, respectively). It corroborates the assumption that local and nonlocal relations should be treated with distinction. However, the performance remains unchanged in RGATSQL, when merging a different view of the graph (w/ MMC) into multi-head attention. This may be caused by the over-smoothing problem of a complete graph. In this part, we analyze the effects of different pre-trained language models in Table 5. From the overall results, we can see that: 1) by involving the line graph into computation, LGESQL outperforms the baseline model RGATSQL with different PLMs, further demonstrating the effectiveness of explicitly modeling edge features. 2) large series PLMs consistently perform better than base models on account of their model capacity and generalization capability to unseen domains. 3) Task adaptive PLMs especially ELECTRA are su-perior to vanilla BERT irrespective of the upper GNN architecture. We hypothesize the reason is that ELECTRA is pre-trained with a tailored binary classification task, which aims to individually distinguish whether each input word is substituted given the context. Essentially, this self-supervised task is similar to our proposed graph pruning task, which focuses on enhancing the discriminative capability of the encoder. In Figure 7, we compare the SQL queries generated by our LGESQL model with those created by the baseline model RGATSQL. We notice that Figure 7: Case study: the first three cases are positive samples while the last one is negative. LGESQL performs better than the baseline system, especially on examples that involve the JOIN operation of multiple tables. For instance, in the second case where the connection of three tables are included, RGATSQL fails to identify the existence of table flights . Thus, it is unable to predict the WHERE condition about the destination city and does repeat work. In the third case, our LGESQL still successfully constructs a connected schema sub-graph by linking table template to documents. Sadly, the RGATSQL model neglects the occurrence of documents again. However, in the last case, our LGESQL is stupid to introduce an unnecessary table airports. It ignores the situation that table flights has one column source airport which already satisfies the requirement. Encoding Problem for Text-to-SQL To tackle the joint encoding problem of the question and database schema, Xu et al. (2017) proposes col-umn attention strategy to gather information from columns for each question word. TypeSQL (Yu et al., 2018a) incorporates prior knowledge of column types and schema linking as additional input features. Bogin et al. (2019a) and Chen et al. (2021) deal with the graph structure of database schema via GNN. EditSQL (Zhang et al., 2019b) considers co-attention between question words and database schema nodes similar to the common practice in text matching (Chen et al., 2017). BRIDGE (Lin et al., 2020) further leverages the database content to augment the column representation. The most advanced method RATSQL (Wang et al., 2020a), utilizes a complete relational graph attention neural network to handle various pre-defined relations. In this work, we further consider both local and non-local, dynamic and static edge features among different types of nodes with a line graph. Heterogeneous Graph Neural Network Apart from the structural topology, a heterogeneous graph (Shi et al., 2016) also contains multiple types of nodes and edges. To address the heterogeneity of node attributes, Zhang et al. (2019a) designs a type-based content encoder and Fu et al. (2020) utilizes a type-specific linear transformation. For edges, relational graph convolution network (RGCN, Schlichtkrull et al., 2018) and relational graph attention network (RGAT, Wang et al., 2020b) have been proposed to parameterize different relations. HAN (Wang et al., 2019b) converts the original heterogeneous graph into multiple homogeneous graphs and applies a hierarchical attention mechanism to the meta-path-based sub-graphs. Similar ideas have been adopted in dialogue state tracking (Chen et al., 2020b, 2019a), dialogue policy learning (Chen et al., 2018) and text matching (Chen et al., 2020c; Lyu et al., 2021) to handle heterogeneous inputs. In another branch, Chen et al. (2019b), Zhu et al. (2019) and Zhao et al. (2020) construct the line graph of the original graph and explicitly model the computation over edge features. In this work, we borrow the idea of a line graph and update both node and edge features via iteration over dual graphs. In this work, we utilize the line graph to update the edge features in the heterogeneous graph for the text-to-SQL task. Through the iteration over the structural connections in the line graph, local edges can incorporate multi-hop relational features and capture significant meta-paths. By further integrating non-local relations, the encoder can learn from multiple views and attend to remote nodes with shortcuts. In the future, we will investigate more useful meta-paths and explore more effective methods to deal with different meta-path-based neighbors. We thank Tao Yu, Yusen Zhang and Bo Pang for their careful assistance with the evaluation. We also thank the anonymous reviewers for their thoughtful comments. This work has been supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), No.SKLMCPTS2020003 Project and Startup Fund for Youngman Research at SJTU (SFYR at SJTU).
objective abstain objective abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain method abstain abstain objective abstain abstain abstain abstain objective objective objective objective abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method abstain method abstain method abstain method abstain abstain abstain abstain method abstain method abstain method method abstain other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain abstain method method other abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method method method method abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain other other other other other other method other other other other other other method method abstain abstain objective other other other