Knowledge-based visual question answering (QA) aims to answer a question which requires visually-grounded external knowledge beyond image content itself. Answering complex questions that require multi-hop reasoning under weak supervision is considered as a challenging problem since i) no supervision is given to the reasoning process and ii) high-order semantics of multi-hop knowledge facts need to be captured. In this paper, we introduce a concept of hypergraph to encode high-level semantics of a question and a knowledge base, and to learn high-order associations between them. The proposed model, Hypergraph Transformer, constructs a question hypergraph and a query-aware knowledge hypergraph, and infers an answer by encoding inter-associations between two hypergraphs and intra-associations in both hypergraph itself. Extensive experiments on two knowledge-based visual QA and two knowledge-based textual QA demonstrate the effectiveness of our method, especially for multi-hop reasoning problem. Our source code is available at https://github.com/yujungheo/ kbvqa-public . Visual question answering (VQA) is a semantic reasoning task that aims to answer questions about visual content depicted in images (Antol et al., 2015; Zhu et al., 2016; Hudson and Manning, 2019), and has become one of the most active areas of research with advances in natural language processing and computer vision. Recently, researches for VQA have advanced, from inferring visual properties on entities in a given image, to inferring commonsense or world knowledge about those entities (Wang et al., 2017, 2018; Marino et al., 2019; Shah et al., 2019; Zellers et al., 2019). In this paper, we focus on the task which is called knowledge-based visual question answering, Figure 1: An example of knowledge-based visual question answering. where a massive number of knowledge facts from a general knowledge base (KB) is given with an image-question pair. To answer the given question as shown in Figure 1, a model should understand the semantics of the given question, link visual entities appearing in the given image to the KB, extract a number of evidences from the KB and predict an answer by aggregating semantics of both the question and the extracted evidences. Following these, there are two fundamental challenges in this task. i) To answer a complex question, multi-hop reasoning over multiple knowledge evidences is necessary. ii) Learning a complex reasoning process is difficult especially in a condition where only QA is provided without extra supervision on how to capture any evidence from the KB and infer based on them. That is, the model should learn which knowledge facts to be attended to and how to combine them to infer the correct answer on its own. Following the previous work (Zhou et al., 2018), we call this setting under weak supervision . Under weak supervision, previous studies proposed memory-based methods (Narasimhan and Schwing, 2018; Shah et al., 2019) and graph-based 373 methods (Narasimhan et al., 2018; Zhu et al., 2020) to learn to selectively focus on necessary pieces of knowledge. The memory-based methods represent knowledge facts in a form of memory and calculate soft attention scores of each memory with respect to a question. Then, it infers an answer by attending to knowledge evidence with high attention scores. On the other hand, to explicitly consider relational structure between knowledge facts, graph-based methods construct a query-aware knowledge graph by retrieving facts from KB and perform graph reasoning for a question. These methods mainly adopt an iterative message passing process to propagate information between adjacent nodes in the graph. However, it is difficult to capture multi-hop relationships containing long-distance nodes from the graph due to the well-known over-smoothing problem, where repetitive message passing process to propagate information across long distance makes features of connected nodes too similar and undiscriminating (Li et al., 2018; Wang et al., 2020). To address the above limitation, we propose a novel method, Hypergraph Transformer, which exploits hypergraph structure to encode multi-hop relationships and transformer-based attention mechanism to learn to pay attention to important knowledge evidences for a question. We construct a question hypergraph and a knowledge hypergraph to explicitly encode high-order semantics present in the question and each knowledge fact, and capture multi-hop relational knowledge facts effectively. Then, we perform hyperedge matching between the two hypergraphs by leveraging transformer-based attention mechanism. We argue that introducing the concept of hypergraph is powerful for multi-hop reasoning problem in that it can encode high-order semantics without the constraint of length and learn cross-modal high-order associations. The main contributions of this paper can be summarized as follows. i) We propose Hypergraph Transformer which enhances multi-hop reasoning ability by encoding high-order semantics in the form of a hypergraph and learning interand intra-high-order associations in hypergraphs using the attention mechanism. ii) We conduct extensive experiments on two knowledge-based VQA datasets (KVQA and FVQA) and two knowledge-based textual QA datasets (PQ and PQL) and show superior performances on all datasets, especially multi-hop reasoning problem. iii) We qualitatively observe that Hypergraph Transformer performs robust inference by focusing on correct reasoning evidences under weak supervision. Knowledge-based visual question answering (Wang et al., 2017, 2018; Shah et al., 2019; Marino et al., 2019; Sampat et al., 2020) proposed benchmark datasets for knowledge-based visual question answering that requires reasoning about an image on the basis of facts from a large-scale knowledge base (KB) such as Freebase (Bollacker et al., 2008) or DBPedia (Auer et al., 2007). To solve the task, two pioneering studies (Wang et al., 2017, 2018) suggested logical parsing-based methods which convert a question to a KB logic query using predefined query templates and execute the generated query on KB for searching an answer. Since then information retrieval-based methods which retrieve knowledge facts associated with a question and conduct semantic matching between the facts and the question are introduced. (Narasimhan and Schwing, 2018; Shah et al., 2019) proposed memory-based methods that represent knowledge facts in the form of memory and calculate soft attention scores of the memory with a question. (Narasimhan et al., 2018; Zhu et al., 2020) represented the retrieved facts as a graph and performed graph reasoning through message passing scheme utilizing graph convolution. However, these methods are complicated to encode inherent high-order semantics and multi-hop relationships present in the knowledge graph. Therefore, we introduce a concept of hypergraph and propose transformer-based attention mechanism over hypergraphs. Multi-hop knowledge graph reasoning is a process of sequential reasoning based on multiple evidences of a knowledge graph, and has been broadly used in various downstream tasks such as question answering (Lin et al., 2019; Saxena et al., 2020; Han et al., 2020b,a; Yadati et al., 2021), or knowledge-enhanced text generation (Liu et al., 2019; Moon et al., 2019; Ji et al., 2020). Recent researches have introduced the concept of hypergraph for multi-hop graph reasoning (Kim et al., 2020; Han et al., 2020b,a; Yadati et al., 2019, 2021; Sun et al., 2020). These models have a similar motivation to the Hypergraph Transformer proposed in this paper, but core operations are vastly different. These models mainly update node representations in the hypergraph through a message passing process using graph convolution operation. On the 374 Figure 2: The overview of Hypergraph Transformer. contrary, our method update node representations via hyperedge matching of hypergraphs instead of message passing scheme. We argue that this update process effectively learns the high-order semantics inherent in each hypergraph and the high-order associations between two hypergraphs. To capture high-order semantics inherent in the knowledge sources, we adopt the concept of hypergraph. Formally, directed hypergraph H = {V , E} is defined by a set of nodes V = { v 1 , ..., v |V| } and a set of hyperedges E = { h 1 , ..., h |E| } . Each node is represented as a w -dimensional embedding vector, i.e., v i  R w . Each hyperedge connects an arbitrary number of nodes and has partial order itself, i.e., h i = { v  1  ...  v  l } where V  = { v  1 , ..., v  l } is a subset of V and  is a binary relation which denotes an element ( v  i ) precedes the other ( v  j ) in the ordering when v  i  v  j . A hyperedge is flexible to encode different kinds of semantics in the underlying graph without the constraint of length. As shown in Figure 2(a), entity linking module first links concepts from query (a given image-question pair) to knowledge base. We detect visual concepts (e.g., objects, attributes, person names) in a given image and named entities in a given question. The semantic labels of visual concepts or named entities are then linked with knowledge entities in the knowledge base using exact keyword matching. Query-aware knowledge hypergraph A knowledge base (KB), a vast amount of general knowledge facts, contains not only knowledge facts required to answer a given question but also unnecessary knowledge facts. Thus, we construct a query-aware knowledge hypergraph H k = {V k , E k } to extract related information for answering a given question. It consists of a node set V k and hyperedge set E k , which represent a set of entities in knowledge facts and a set of hyperedges, respectively. Each hyperedge connects the subset of vertices V  k  V k . We consider a huge number of knowledge facts in the KB as a huge knowledge graph, and construct a hypergraph by traversing the knowledge graph. Such traversal, called graph walk, starts from the node linked from the previous module (see section 3.2) and considers all entity nodes associated with the start node. We define a triplet as a basic unit of graph walk to preserve high-order semantics inherent in knowledge graph, i.e., every single graph walk contains three nodes {head, predicate, tail} , rather than having only one of these three nodes. In addition to the triplet-based graph walks, a multihop graph walk is proposed to encode multiple relational facts that are interconnected. Multi-hop graph walk connects multiple facts by setting the arrival node ( tail ) of the preceding walk as the starting ( head ) node of the next walk, thus, n -hop graph walk combines n facts as a hyperedge. Question hypergraph We transform a question sentence into a question hypergraph H q consisting of a node set V q and a hyperedge set E q . We assume that each word unit (a word or named entity) of the question is defined as a node, and has edges to adjacent nodes. For question hypergraph, each word unit is used as a start node of a graph walk. The multi-hop graph walk is conducted in the same manner as the knowledge hypergraph. A n -gram phrase is considered as a hyperedge in the question hypergraph (see Figure 2(b)). To consider high-order associations between knowledge and question, we devise structural semantic matching between the query-aware knowledge hypergraph and the question hypergraph. We introduce an attention mechanism over two hypergraphs based on guided-attention (Tsai et al., 2019) and self-attention (Vaswani et al., 2017). As shown in Figure 2(c), the guided-attention blocks are introduced to learn correlations between knowledge hyperedges and question hyperedges by inter-attention mechanism, and then intra-relationships of in knowledge or question hyperedges are trained with the following self-attention blocks. The details of two modules, guided-attention blocks and self-attention blocks, are described as below. Note that we use Q , K , and V for query, key, value, and q , k as subscripts to represent question and knowledge, respectively. Guided-attention To learn inter-association between two hypergraphs, we first embed a knowledge hyperedge and a question hyperedge as follows: e k =  k  f k ( h k )  R d , e q =  q  f q ( h q )  R d where h [  ] is a hyperedge in E [  ] . Here, f [  ] is a hyperedge embedding function and  [  ] is a linear projection function. The design and implementation of f [  ] are not constrained (e.g., any pooling operation or any learnable neural networks), but we use a simple concatenation operation of node representations in a hyperedge as f [  ] . The representations of hyperedges in the same hypergraph (e.g., e k , e q ) are packed together into a matrix E k and E q . We define the knowledge hyperedges E k and the question hyperedges E q as a query and key-value pairs, respectively. We set a query Q k = E k WQ k , a key K q = E q WK q , and a value V q = E q WV q , where all projection matrices W [  ]  R d  d v are learnable parameters. Then, scaled dot product attention using the query, key, and value is calculated as Attention( Q k , K q , V q ) = softmax( Q k K Tq  d v ) V q where d v is the dimension of the query and the key vector. In addition, the guided-attention which uses the question hyperedges as query and the knowledge hyperedges as key-value pairs is performed in a similar manner: Attention( Q q , K k , V k ) . Self-attention The only difference between guided-attention and self-attention is that the same input is used for both query and key-value within self-attention. For example, we set query, key, and value based on the knowledge hyperedges E k , and the self-attention for knowledge hyperedges is conducted by Attention( Q k , K k , V k ) . For question hyperedges E q , self-attention is performed in a similar manner: Attention( Q q , K q , V q ) . Following the standard structure of the transformer, we build up guided-attention block and self-attention block where each block consists of each attention operation with layer normalization, residual connection, and a single feed-forward layer. By passing the guided-attention blocks and self-attention blocks sequentially, representations of knowledge hyperedges and question hyperedges are updated and finally aggregated to single vector representation as z k  R d v and z q  R d v , respectively. To predict an answer, we first concatenate the representation z k and z q obtained from the attention blocks and feed into a single feed-forward layer (i.e., R 2 d v (cid:55) R w ) to make a joint representation z . We then consider two types of answer predictor: multi-layer perceptron and similarity-based answer predictor. Multi-layer perceptron as an answer classifier p =  ( z ) is a prevalent for visual question answering problems. For similarity-based answer, we calculate a dot product similarity p = zC T between z and answer candidate set C  R |A| w where |A| is a number of candidate answers and w is a dimension of representation for each answer. The most similar answer to the joint representation is selected as an answer among the answer candidates. For training, we use only supervision from QA pairs without annotations for ground-truth reasoning paths. To this end, cross-entropy between prediction p and ground-truth t is utilized as a loss function. 4.1 Datasets In this paper, we evaluate our model across various benchmark datasets: Knowledge-aware VQA (KVQA) (Shah et al., 2019), Fact-based VQA (FVQA) (Wang et al., 2018), PathQuestion (PQ) and PathQuestion-Large (PQL) (Zhou et al., 2018). KVQA, a large-scale benchmark dataset for complex VQA, contains 183,007 pairs for 24,602 images from Wikipedia and corresponding captions, and provides 174,006 knowledge facts for 39,414 unique named entities based on Wikidata (Vrandecic and Krtzsch, 2014) since it requires world knowledge beyond visual content. KVQA consists of two types of questions: original (ORG) and paraphrased (PRP) question generated from the original question via the online paraphrasing tool. FVQA, a representative dataset for commonsense-enabled VQA, considers external knowledge about common nouns depicted in a given image, and contains 5,826 QA pairs for 2,190 images and 4,216 unique knowledge facts from DBPedia (Auer et al., 2007), ConceptNet (Liu and Singh, 2004), and WebChild (Tandon et al., 2014). The last two datasets, PQ and PQL, focus on evaluating multi-hop reasoning ability in the knowledge-based textual QA task. PQ and PQL contain 7,106 and 2,625 QA pairs on 4,050 and 9,844 knowledge facts from the subset of Freebase (Bollacker et al., 2008), respectively. The detailed statistics of the datasets are shown in Appendix A. 4.2 Implementation details Each node in the knowledge hypergraph and the question hypergraph is represented as a 300-dimensional vector (i.e., w = 300 ) initialized using GloVe (Pennington et al., 2014). Random initialization is applied when a word for a node does not exist in the vocabulary of GloVe. Mean pooling is applied when a node consists of multiple words. For entity linking for KVQA, we apply the wellknown pre-trained models for face identification: RetinaFace (Deng et al., 2020) for face detection and ArcFace (Deng et al., 2019) for face feature extraction. For all datasets, we follow the experimental settings as in previous works. We use the similarity-based answer predictor for KVQA, and MLP for the others. We adopt Adam (Kingma and Ba, 2015) to optimize all learnable parameters in the model. We describe details of the experimental settings and the tuned hyperparameters for each dataset in Appendix D. 5 Quantitative Results 5.1 Knowledge-aware visual question answering We compare the proposed model, Hypergraph Transformer, with other comparative state-of-the-art methods. We report performances on original (ORG) and paraphrased (PRP) questions according to the number of graph walk. For comparative models, three kinds of methods are considered, which are graph-based, memory-based and attention-based networks. The detailed description about the comparative models is described in Appendix E. To evaluate a pure reasoning ability of the models regardless of the performance of entity linking, we first conduct experiments in the oracle setting which ground-truth named entities in an image are given. all settings. From the results, we find that the attention mechanism between question and knowledge is crucial for complex QA. Since GCN (Kipf and Welling, 2017) and GGNN (Li et al., 2016) encode question and knowledge graph separately, they do not learn interactions between question and knowledge. Thus, GCN and GGNN show quite low performance under 74% mean accuracy. On the other hand, MemNN  (Weston et al., 2015), HAN (Kim et al., 2020), and BAN (Kim et al., 2018) achieve comparatively high performance because MemNN  adopts question-guided soft attention over knowledge memories. HAN and BAN utilize multi-head co-attention between question and knowledge. Entity linking setting We also present the experimental results on the entity linking setting where the named entities are not provided as the oracle setting, but detected by the module as described in Section 3.2. As shown in Table 7 of Appendix E, our model shows the best performances for both original and paraphrased questions. For all comparative models, we use the same knowledge hypergraph extracted by the 3-hop graph walk. In entity linking setting, the constructed knowledge hypergraph can be incomplete and quite noisy due to the undetected entities or misclassified entity labels. However, Hypergraph Transformer shows robust reasoning capacity over the noisy inputs. Here, we remark that the upper bound of QA performance is 72.8% due to the error rate of entity linking module. We expect that the performance will be improved when the entity linking module is enhanced. We conduct experiments on Fact-based Visual Question Answering (FVQA) as an additional benchmark dataset for knowledge-based VQA. Different from KVQA focusing on world knowledge for named entities, FVQA considers commonsense knowledge about common nouns in a given image. Here, we assume that the performance of entity linking is perfect, and evaluate the pure reasoning ability of our model. As shown in Table 8 of Appendix D, Hypergraph Transformer shows comparable performance in both top-1 and top-3 accuracy in comparison with the state-of-the-art methods. We confirm that our model works effectively as a general reasoning framework without considering characteristics of different knowledge sources (i.e., Wikidata for KVQA, DBpedia, ConceptNet, WebChild for FVQA). To verify multi-hop reasoning ability of our model, we conduct experiments on PathQuestion (PQ) and PathQuestion-Large (PQL). PQ and PQL datasets have annotations of a ground-truth reasoning path to answer a given question. Specifically, {PQ, PQL}-{2H, 3H} denotes a split of PQ and PQL with respect to the number of hops in ground-truth reasoning paths (i.e., 2-hop or 3-hop). {PQ, PQL}-M is a mixture of the 2-hop and 3-hop questions in both dataset, and used to evaluate the more general scenario where the number of reasoning path 378 Model Inputs Original (ORG) Paraphrased (PRP) Mean Knowledge Question 1-hop 2-hop 3-hop 1-hop 2-hop 3-hop required to answer a given question is unknown. The experimental results on diverse split of PQ and PQL datasets are provided in Table 2. The first section in the table includes fully-supervised models which require a ground-truth path annotation as an additional supervision. The second section contains weakly-supervised models learning to infer the multi-hop reasoning paths without the ground-truth path annotation. Hypergraph Transformer is involved in the weakly-supervised models because it only exploits an answer as a supervision. Our model shows comparable performances on PQ-{2H, 3H, M} to the state-of-the-art weakly-supervised model, SRN. Especially, Hypergraph Transformer shows significant performance improvement (78.6%  90.5% for PQL-2H, 78.3%  94.5% for PQL-M) on PQL. We highlight that PQL is more challenging dataset than PQ in that PQL not only covers more knowledge facts but also has fewer QA instances. We observe that the accuracy on PQL-3H is relatively lower than the other splits. This is due to the insufficient number of training QA pairs in PQL-3H. When we use PQL-3H-More which has twice more QA pairs (1031  2062) on the same knowledge base as PQL-3H, our model achieves 95.4% accuracy. We verify the effectiveness of each module in Hypergraph Transformer. To analyze the performances of the variants in our model, we use KVQA which is a representative and large-scale dataset for knowledge-based VQA. Here, we mainly focus on two aspects: i) effect of hypergraph and ii) effect of attention mechanism. To evaluate a pure reasoning ability of the models, we conduct experiments in the oracle setting. To analyze the effectiveness of hypergraph-based input representation, we conduct comparative experiments on the different types of input formats for Transformer architecture. Here, we consider the two types of input format, which are single-word-unit and hyperedge-based representations. Compared to hyperedge-based inputs considering multiple relational facts as a input token, single-word-unit takes every entity and relation tokens as separate input tokens. We note that using single-word-unit-based input format for both knowledge and question is the standard settings for the Transformer network and using hyperedge-based input format for both is the proposed model, Hypergraph Transformer. We set the Transformer (SA+GA) as a backbone model, and present the results in Table 3(b-e). When hypergraph-based representations are used for both knowledge and question, the results show the best performance across all settings over question types (ORG and PRP) and a number of graph walk (1-hop, 2-hop, and 3-hop). As shown in Table 3, the mean accuracy of QA achieves 89.7% when both are encoded using hyperedges, while using single-word-unit-based representation causes performance to drop to 81.6%. Especially, when we convert the one of both hyperedge-level representation to single-word-unit-based representation, the mean accuracy of QA is 82.7% and 88.7%, respectively. These results validate that it is meaningful to consider not only knowledge but also question as hypergraphs. Effect of multi-hop graph walk We compare the performances with different number of graph walks used to construct a knowledge hypergraph (i.e., 1-hop, 2-hop, and 3-hop). All models except ours show slightly lower performance on the 3-hop graph than on the 2-hop graph. We observe that the number of extracted knowledge facts increases when the number of graph walk increases, and unnecessary facts for answering a given question are usually included. Nonetheless, our model shows robust reasoning performance when a large and noisy knowledge facts are given. To investigate the impacts of each attention block (i.e., GA and SA), ablation studies are shown in Table 3(e-g). The scores across all settings drop when GA or SA is removed. Particularly, the mean accuracy of QA is decreased by 6.0% (89.7%  83.7%), 2.6% (89.7%  87.1%) for cutting out the GA and the SA block, respectively. Based on the two experiments, we identify that not only the guided-attention which captures inter-relationships between question and knowledge but also the self-attention which learns intra-relationship in them are crucial to the complex QA. To sum up, Hypergraph Transformer takes graph-level inputs, i.e., hyperedge, and conducts semantic matching between hyperedges by the attention mechanism. Due to the two characteristics, the model shows better reasoning performance focusing on the evidences necessary for reasoning under weak supervision. Figure 3 provides the qualitative analysis on effectiveness of using a hypergraph as an input format to Transformer architecture. We present the attention map from the guided-attention block, and visualize topk attended knowledge facts or entities with the attention scores. In the first example, both model, Hypergraph Transformer and Transformer (SA+GA), infer the correct answer, Q5075293 . Our model responds by focusing on { second  from  left } phrase of the question and four facts having a left relation among 86 knowledge hyperedges. In comparison, Transformer (SA+GA) strongly attends to the knowledge entities which appear repetitive in the knowledge facts. Especially, the model attends to Q3476753 , Q290666 and Ireland with the high attention score 0.237, 0.221, and 0.202. In the second example, our model attends to the correct knowledge hyperedges considering the multi-hop facts about place of birth of the people shown in the given image, and infers 380 the correct answer. On the other hand, Transformer (SA+GA) strongly attends to the knowledge entity of person ( Q2439789 ) presented in the image with undesired attention score 0.788. The second and third attended knowledge entities are the other person ( Q7141361 ) and Iran . Transformer (SA+GA) fails to focus on the multi-hop facts required to answer the given question and predicts the answer with the wrong number at the end. In this paper, we proposed Hypergraph Transformer for multi-hop reasoning over knowledge graph under weak supervision. Hypergraph Transformer adopts hypergraph-based representation to encode high-order semantics of knowledge and questions and considers associations between a knowledge hypergraph and a question hypergraph. Here, each node representation in the hypergraphs is updated by interand intra-attention mechanisms in two hypergraphs, rather than by iterative message passing scheme. Thus, Hypergraph Transformer can mitigate the well-known over-smoothing problem in the previous graph-based methods exploiting the message passing scheme. Extensive experiments on various datasets, KVQA, FVQA, PQ, and PQL validated that Hypergraph Transformer conducts accurate inference by focusing on knowledge evidences necessary for question from a large knowledge graph. Although not covered in this paper, an interesting future work is to construct heterogeneous knowledge graph that includes more diverse knowledge sources (e.g. documents on web). We would like to thank Woo Young Kang, Kyoung-Woon On, Seonil Son, Gi-Cheon Kang, Christina Baek, Junseok Park, Min Whoo Lee, Hwiyeol Jo and Sang-Woo Lee for their helpful comments and discussion. This work was partly supported by the IITP (2015-0-00310-SW.StarLab/20%, 2017-0-01772-VTT/20%, 2019-0-01371-BabyMind/10%, 2021-0-02068-AIHub/10%, 2021-0-01343-GSAI/10%, 2020-0-01373/10%) grants, the KIAT (P0006720-ILIAS/10%) grant funded by the Korean government, and the Hanyang University (HY-202100000003160/10%).
abstain abstain abstain abstain abstain abstain objective other abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain objective method method method objective objective result result other other other other other other objective other other objective other other method abstain method abstain abstain other abstain abstain abstain method abstain abstain method abstain abstain method abstain method abstain abstain method method abstain abstain abstain abstain method method abstain abstain abstain method method abstain method abstain method method abstain abstain abstain method abstain method abstain method method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain result method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method method method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain method other other