We consider the task of linking social media accounts that belong to the same author in an automated fashion on the basis of the content and metadata of their corresponding document streams. We focus on learning an embedding that maps variable-sized samples of user activityranging from single posts to entire months of activityto a vector space, where samples by the same author map to nearby points. The approach does not require human-annotated data for training purposes, which allows us to leverage large amounts of social media content. The proposed model outperforms several competitive baselines under a novel evaluation framework modeled after established recognition benchmarks in other domains. Our method achieves high linking accuracy, even with small samples from accounts not seen at training time, a prerequisite for practical applications of the proposed linking framework. The scale and anonymity of social media pose systematic challenges for manual moderation efforts (Pennycook et al., 2020; Broniatowski et al., 2018). These challenges have motivated the development of automated methods to identify abusive content, such as Davidson et al. (2017), which considers automatically classifying hate speech, or Alvari et al. (2019), which deals with detecting violent extremism, both in the Twitter domain. However, automatic moderation remains a difficult problem. Indeed, existing methods based on hand-constructed resources such as keyword lists may fail to adapt to novel trends (Corbett-Davies and Goel, 2018) whereas automatic methods based on statistics of large corpora may exhibit harmful biases (Caliskan et al., 2017). Additionally, individual posts may fail to contain sufficient information to reliably identify them as harmful. This work considers account-level moderation. Specifically, we consider the problem of determining whether two document streams share the same author, based on samples from those streams rather than individual documents. This capability has numerous applications, such as detecting users attempting to circumvent account bans, identifying sockpuppet accounts, and detecting coordinated disinformation campaigns involving multiple authors controlling multiple accounts. As a motivating application, we consider the enforcement of account bans on anonymous platforms, such as Reddit. Given a new account, the problem is to automatically identify whether it matches any previously banned account, which amounts to making binary decisions about whether pairs of accounts share the same author. Variations of this problem have been studied before. For example, Schwartz et al. (2013) learn a classifier to determine whether the author of a Twitter comment belongs to a small, closed set of authors. In contrast, we are interested in an open-world setting, requiring binary decisions about arbitrary pairs of accounts. This introduces a number of challenges. First, any individual comment may be too short to serve as the basis for linking accounts. Figure 1a illustrates this empirically using a variation of our model, where embeddings of individual comments from the same account fail to coalesce, making it difficult to assert that an account has the same author as another account. See 4 for further experimental details. Therefore, we focus on aggregating information across contiguous sequences of documents. Figure 1b illustrates the impact of aggregation using our full model, where aggregations of contiguous sequences of documents from the same account exhibit an approximate convergence behavior as the number of documents aggregated increases. In fact, the motivating application above requires linking accounts on the basis of samples of widely varying sizes. Indeed, banned accounts (a) Embeddings of single posts by a model trained on single posts. (b) Embeddings of samples by a model trained on variablesized samples. For each author and each 1  j  16 , the j th point shows the embedding of the author's first j posts. typically have many documents, all of which we would like to consider, while new accounts generally have few documents, which nevertheless must be linked to banned accounts as quickly as possible to mitigate abusive behavior. The second challenge is that of spurious associations. For example, over a short period of time, an author may discuss only a single, narrow topic. While a nave model based on word statistics alone might be sufficient to link such an account to another, this approach would fail to generalize over longer periods of time due to topic drift. Both our training procedure and evaluation framework have been designed to ensure that our model learns the appropriate invariances to identify same-authorship, rather than being correct for the wrong reasons (McCoy et al., 2019). Namely, samples from an account are drawn from different time periods in each training iteration (see 2.3), while the evaluation data consists of posts by accounts not seen at training time and is future to all the training data (see 4.2). Finally, the numbers of banned and new accounts may be quite large, requiring a still larger number of pairwise comparisons. For this reason, our proposed approach to account linking consists of embedding variable-sized samples from document streams into a metric space whereby samples likely to have been composed by the same author map to nearby points. Under this embedding, comparisons between document streams amount to pairwise distance calculations, so our approach is highly scalable and amenable to various optimizations, such as approximate nearest neighbor methods. Our primary contributions are the following:  We provide a simple but effective data augmentation strategy which enables embedding variable-sized samples. In addition, we successfully train such an embedding on a large-scale dataset consisting of more than 300 million comments from 1 million distinct accounts using scalable losses. We propose a novel framework to assess account linking performance focused on challenging conditions and minimizing the impact of incidental authorship features, such as topic. In particular, we propose benchmark datasets as well as verification metrics tailored to our application. Our code, data splits, and scripts to reproduce our experiments are available at http:// github.com/noa/naacl2021 . We treat a document stream as a sequence of times-tamped actions a 1 , a 2 , . . . , a L where each a i is a structure containing the data comprising an action. The possible contents of a i are specific to the document stream, but include at least a timestamp t i such that t 1 < t 2 <    < t L . In this work we focus on textual content published on social media platforms, although the approach would easily extend to allow a 1 , a 2 , . . . , a L to contain other modalities such as images or video, which would be handled similarly. In addition, we ( t 1 , r 1 ) E t 1 r 1 x 1 ( t 2 , r 2 ) E t 2 r 2 x 2 ... ( t M , r M ) E t M r M x MA M f  ( a ) P x 1 E C P x 2 E C P ... x ME C P Figure 2: Illustration of the proposed model architecture. also avail of certain categorical features contained in a 1 , a 2 , . . . , a L such as hashtags or the subreddit to which a comment was posted. A sample from a document stream is a contiguous subsequence of its actions. We introduce an embedding f  in 2.1 mapping a variable-sized sample to a point in a vector space, such that the Euclidean distance between the embeddings of two samples quantifies the likelihood that they belong to the same author. We define an embedding f  as follows. This embedding is illustrated in Figure 2. Consider a sample a = ( a 1 , a 2 , . . . , a M ) where each action a i consists of a subreddit feature r i , a timestamp t i , and text content x i . 1 We encode t i as the corresponding hour of the day and r i by lookup in the list of 2048 most common subreddits, resulting in t i  { 0 , 1 , . . . , 23 } and r i  { 0 , 1 , . . . , 2048 } , where r i = 2048 when the subreddit is not among the top 2048. We encode the text feature using the SentencePiece unigram subword model (Kudo, 2018), resulting in x i  Z (cid:96) , where the parameter (cid:96) is defined in 2.2. Note that the chosen vocabulary size impacts the amount of content that can be encoded with (cid:96) integers. Further details on the choice of text encoding are provided in Appendix C. We replace each token of x i with a correspond-1 In addition to r i , t i , x i platforms like Reddit contain further metadata, such as the thread title and the submission and parent comments, all of which might help distinguish users because users read them and chose to respond. Incorporating these features would be interesting to explore in future work. ing learned embedding in RN for all 1  i  M , resulting in M matrices in R (cid:96)  N . We apply one-dimensional convolutions of widths 2, 3, and 4 along the first axis of each, concatenate the convolved matrices along their second axes, max-pool along the first axis, and concatenate the results with learned embeddings of the corresponding subreddit features and one-hot encodings of the corresponding time features, resulting in M vectors, which we aggregate using dot-product attention. The resulting sequence of vectors is projected to a single vector through max-pooling followed by two fully-connected layers with bias, resulting in the encoding f  ( a )  RD of the sample a . The variance in the lengths of documents poses computational challenges when aggregating large samples. Therefore we resort to truncating each document to a fixed number (cid:96) of tokens, padding any documents containing fewer than (cid:96) tokens. We take (cid:96) = 32 after observing that Reddit posts have an average length of approximately 43 tokens (see Appendix D). We also experimented with a more complicated text sampling strategy, namely sampling contiguous segments of (cid:96) tokens from each post uniformly at random during training. While this approach leverages all available textual information by affording slightly different samples of each post in each iteration of training, we found it to yield similar results and also complicates the comparison to our primary baseline model, which uses the prefix x y Unif (0 , 1) Beta (3 , 1) 1 2 3 12 1 Figure 3: In contrast with choosing sample sizes uniformly at random, sampling according to a left skewed Beta distribution results in more samples of sizes closer to the specified maximum. During training we randomly select document stream samples of sizes varying between R = 1 and S = 16 , which we regard as hyperparameters of the model. To select a sample from the stream a 1 , a 2 , . . . , a L we first choose its length M = R + (cid:100) x ( S  R ) (cid:101) where x  Beta (3 , 1) and take the sample a i , a i +1 , . . . , a i + M  1 where 1  i  L  M +1 is chosen uniformly at random. Selecting M according to Beta (3 , 1) provides an expected sample size closer to S than to R , a tradeoff that allows the model to quickly learn features of a document stream by exposing it to larger samples most of the time, while still maintaining the flexibility to handle samples of varying sizes. Indeed, the latter is critical in the evaluation described in 4.3, which requires linking large samples to small samples. The density function of Beta (3 , 1) is shown in Figure 3 together with that of the uniform distribution Unif (0 , 1) for comparison. We explore the benefits of Beta (3 , 1) and other related distributions in 4.5. Deep metric learning methods aim to embed observations into a low-dimensional space such that instances from the same class map to nearby points under a chosen metric, such as Euclidean distance. In our setting, we take the instances to be document stream samples and the classes to be the corresponding accounts, which serve as proxies for latent authorship. Therefore, training the mapping f  defined in 2.1 using metric learning affords an embedding under which samples by the same author map to nearby points. Recent work in deep metric learning has introduced a number of training objectives with state of the art performance on computer vision tasks (Kim et al., 2020; Wang et al., 2019). Unfortunately, many of these objectives scale linearly with the number K of classes considered due to a costly linear projection onto RK . Note that because account names in effect provide labels for the corresponding document streams, we may use raw social media content to fit our model directly, availing of a virtually unlimited source of data. We stipulate that the ability to exploit larger amounts of data may be more important than per-example efficiency, and therefore consider the classical triplet loss (Schroff et al., 2015) in our experiments, whose complexity does not depend on K . In particular, we use semihard negative mining with a fixed margin penalty. We also consider the topk loss recently proposed by Lu et al. (2019), which optimizes precision-at-k as follows. Given targets ranked by similarity to a query, topk arranges for as many matches as possible to be among the top k ranked targets. It accomplishes this by penalizing only those targets that would need to move the smallest amount in order to maximize the number of matching targets among the top k . Like triplet loss, topk also uses an additive margin penalty to separate classes. See Appendix A for further experimental details on both loss functions. The separate but related problem of closed-world author attribution has received considerable attention. For example, the PAN 2019 challenge (Daele-mans et al., 2019) employed a closed-world setting with a small number of authors that are the same at training and test time. That task also considered longer documents, obviating the need for aggregating evidence of authorship across multiple documents. Generic text embedding methods such as the universal sentence encoder (Cer et al., 2018) and BERT (Devlin et al., 2019) are fit using auxiliary tasks, such as conditional language modeling. In the case of BERT, this is usually followed by supervised fine-tuning for a downstream task of interest. In this work, we are interested in learning representations that are immediately useful for our account linking task. However, because a large corpus of task-specific training data may be collected without human supervision, the benefits of generative pretraining are diminished in our setting. Indeed, the parameters of the text encoding are learned from a random initialization in all our experiments. Our approach is further distinguished from generic embedding methods by featuring a multi-document embedding, mapping a sequence of documents to a single vector, where each document may consist of both text and metadata. The most closely related prior work is the Invariant User Representation (IUR) proposed by Andrews and Bishop (2019), whose approach is broadly similar to ours, but only considers samples of a fixed size. Our approach may be viewed as a generalization of that work in support of the account linking task. In addition we use a simpler dot product attention mechanism and introduce the use of scalable metric learning losses in 2.4, which enable us to train our model on an order of magnitude more data than previously considered. We validate these improvements in 4.2 using the ranking task proposed by Andrews and Bishop (2019). We also adapt IUR to serve as a baseline in our primary linking task in 4.3. We believe that our treatment of account linking as a pairwise recognition task between document stream samples and our proposed general-purpose evaluation protocols are both novel. However, in prior work, there have been several platform-specific approaches to account linking. For example Silvestri et al. (2015) explore a heuristic approach to linking accounts across social media platforms. Separately, on platforms with rich social network information, graph-matching methods have been explored (Fan, 2012). Our focus is on content-based account linking, which is more general than prior methods we are aware of. Some other related but distinct problems include detecting deceptive accounts (Van Der Walt and Eloff, 2018) and authorship classification of short messages (Ishihara, 2011). We conduct evaluations on the two primary tasks illustrated in Figure 4. First, our ranking evaluation described in 4.2 is motivated by information retrieval needs. Although ranking is not the focus of this paper, it provides an assessment of the quality of the learned embedding in terms of similarity judgements and facilitates comparison with the baseline model IUR. In addition, we use the ranking evaluation to monitor training using de-Model Features Loss MRR R@4 R@8 Prop TPS Topk 0.637 0.709 0.765 Prop TPS Triplet 0.634 0.702 0.762 Prop TP Topk 0.450 0.522 0.595 Prop TP Triplet 0.452 0.520 0.591 Prop T Triplet 0.372 0.439 0.512 IUR TPS Arcface 0.520 0.590 0.650 IUR T Arcface 0.200 0.240 0.290 Table 1: Ranking results for the proposed model (Prop) and the baseline (IUR), both trained and evaluated using various combinations of text content (T), publication time (P), and subreddit (S). velopment data disjoint from the test data used for the final evaluation. Second, we introduce an account linking evaluation framework in 4.3 inspired by similar evaluations used for speaker recognition (Doddington et al., 2000; Van Leeuwen and Brmmer, 2007). Both evaluations involve setting up two sets of samples as described below, the queries and the targets . For each query, there is exactly one target drawn from the same document stream. Roughly speaking, both evaluations involve matching targets with their corresponding queries. Reddit is currently one of the most popular social media platforms, where anonymous users interact primarily by posting comments to discussion threads. Together with its text content, each comment is labeled by its publication time and the subreddit to which it was posted, a categorical feature roughly indicating its topic. We construct a dataset consisting of 300 million Reddit posts from 1 million users published over an entire year to be used to train our proposed model. This Million User Dataset (MUD) consists of all posts by authors who published at least 100 and at most 1000 posts between July 2015 and June 2016, where the lower bound ensures a sufficiently long history from which to sample, and the upper bound is intended to reduce the impact of bot and spam accounts. We obtained the data by drawing from the existing Pushshift Reddit corpus (Baumgartner et al., 2020). Some further statistics of MUD are shown in Table 8. (a) Ranking evaluation: for each query, the targets are ranked by similarity to the query. (b) Linking evaluation: for each target, the queries are determined to match each query or not. As shown in Figure 4a, the ranking experiment consists of ranking the targets by similarity to each query. For compatibility, we mimic the experimental setup from Andrews and Bishop (2019), which proposes separate sets of queries and targets to be used for training and testing. We adopt the training split for validation and the testing split for evaluation, although we train our model on MUD (see 4.1). We select hyperparameters based on dev split performance (see Appendix A). The test split consists of samples, each of size exactly 16, although we train the proposed model using samples from MUD of varying sizes as described in 2.3. Note that the posts comprising MUD precede those of both IUR splits in publication time, ensuring that our training data is disjoint from IUR's test data. Of the 111,396 authors contributing to the test split, 69,275 or 62% contribute to the IUR training split. In contrast, MUD has only 39,529 users in common with the test split, a significantly smaller overlap than IUR. In principle, the increase in novel users at test time puts the proposed model at a disadvantage because it places more importance on generalization to novel users. We report recall-atk (R@ k ) and mean reciprocal rank (MRR), calulated exactly as in Andrews and Bishop (2019). MRR is the expected value of the reciprocal of the position of the correct target in the ranked list. R@ k is the probability that the unique target composed by the same author as a given query appears in the top k ranked results. We limit ourselves to R@4 and R@8 as proxies for the first page of search results returned to a user issuing a query. The results of this evaluation calculated with the test split are shown in Table 1. Note that the full version of the proposed model significantly outperforms the previously published state-of-the-art. 2 We conclude that although the angular margin loss used by Andrews and Bishop (2019) is considered state-of-the-art, the simpler triplet loss outperforms it, most likely because it admits the use of a considerably larger dataset. We remark that the models trained with topk performed only slightly better than those trained with triplet loss, an observation consistent with recent findings that when matching experimental conditions, the choice of ranking loss is less important than previously believed (Mus-grave et al., 2020). In addition, Figure 5 shows the results of the evaluation performed after every hour of training. 3 Note that after only six hours of training the full model outperforms the baseline. Figure 5 also shows the learning curve for an ablation of our model that eliminates the subreddit feature. We observe that this ablated model performs almost as well as the full-featured baseline, which suggests that the proposed approach may be effective in domains where only text and timestamps are 2 A paired sign test of the differences in ranking between IUR and the proposed model is significant at the p < 10  15 level. While the ranking experiments in 4.2 were designed to measure the quality of the learned embedding, they do not directly measure task performance: moderation applications require decisions rather than rankings . To this end, we propose an account linking benchmark modeled after the problem of enforcing account bans, in which a fixed number of accounts are linked against novel accounts at test-time. Compared to the ranking experiments, the key difference is that we introduce a distinguished subset of authors from which we have accumulated a significant number of previously published documents to serve as queries. The procedure is illustrated in Figure 4b. Because the subreddit feature serves as a proxy for topic, restricting to a single subreddit results in a more challenging problem by increasing the likelihood that the comments considered deal with similar topics. To this end, we repeat the following procedure for each of the five most popular subreddits. Each result of the experiment reported in Tables 2 and 9 is the average over the five subreddits of the corresponding results calculated using those subreddits individually. Given a specified subreddit, we first randomly select 100 distinguished accounts, each publishing at least 100 posts to that subreddit in November 2016. The queries in the experiment consist of the 100 most recently published posts to the subreddit by each of the distinguished accounts in November 2016. In addition, the distinguished accounts must have published at least 16 posts to the subreddit between December 2016 and May 2017 to serve as the corresponding targets, as described below. Next we randomly select 4900 accounts distinct from the distinguished accounts, each publishing at least 16 posts to the subreddit between December 2016 and May 2017. The targets in the experiment consist of the 4 most recently published posts to the subreddit by each of the 5000 accounts. Performance metrics . For every query and target, each model considered returns a score, with smaller scores associated with a higher likelihood that the query and the target have the same author. For example, the proposed model returns the distance between their embeddings under the model. A decision rule to predict an author match is obtained by thresholding this score with respect to a chosen operating point . In production settings, one adjusts the operating point to obtain acceptable rates of false positives and false negatives. In our running application of ban enforcement, these types of errors correspond respectively with mistakenly banning an innocent user and failing to ban a new account of a banned user. Because the severity of these types errors are different, we consider the detection cost function C det = C  P  + (1   ) C + P + proposed by Van Leeuwen and Brmmer (2007), where P  and P + are empirical probabilities of false negatives and false positives, C  and C + are the costs of false negatives and false positives, and  is the a priori probability of a match. We take  = 0 . 05 and we set C  = 1 and C + = 2 , reflecting our presumption that banning an innocent account is more severe than failing to recognize a banned user. Our choices of C  and C + are only meant to reflect the asymmetric nature of the problem, although in practice these costs would be highly platform-specific. We report the minimum value of C det over all operating points (minDCF) and the value of P + at the operating point for which P  = P + , also known as the equal error rate (EER). Baseline models . We compare the proposed method with three baselines. First we consider TF-IDF vector representations of the concatenated text content of a sample, which are compared using cosine similarity. Next, we consider universal sentence encodings (Cer et al., 2018), which Model Training EER minDCF Length TF-IDF  0.341 0.971 Universal  0.363 0.981 IUR 16 0.247 0.999 TP 18 0.169 0.848 TP 116 0.132 0.792 Table 2: Linking evaluation, averaged over 5 subreddits. are compared using angular distance. We experimented with two versions of this baseline, namely embedding the concatenation of the text content of the documents in a sample, and averaging the embeddings of the individual documents. Since we found the concatenated version to perform better, we only report on this variation. Finally, we consider IUR (Andrews and Bishop, 2019). Because this model only embeds samples of size 16, we pad samples containing fewer than 16 posts. To handle samples containing more than 16 posts, we organize the sample into contiguous groups of at most 16 posts, apply the embedding to each group, and average the embeddings. Results . Table 2 compares the linking performance of the three baseline models along with two of variations of the proposed model arising from varying the sizes of the training samples. Note that both variations of the proposed model outperform the baselines. A further variation on this experiment is reported in Table 9 in which the queries are drawn from the training dataset, better reflecting the context of the motivating example. Figure 6 shows the effect of the sizes of the targets on linking performance. Note that performance rapidly improves with larger samples, relative to the baselines. This trend is promising for our motivating application of ban enforcement, where it is desirable to recognize banned users as early as possible. Figure 7 shows receiver operator curves (ROC), which plot false positive rates against true positive rates as the operating points vary. Our experiments in 4.3 show that linking samples from newly created accounts to those of distinguished authors is more successful when using as much historical data from the distinguished authors as possible. However, computational constraints typically inhibit embedding full account histories during training. Instead, in 4.3 we embed large samples of distinguished authors' histories using models trained on samples of sizes up to a maximum tractable length S . We take S = 16 in our experiments as described in 2.3. Here, we examine the ability of a model trained on samples of sizes at most S to generalize to samples of sizes greater than S . We also compare to a further baseline that averages single-post embeddings produced by a variation of the proposed model trained on single posts, which we denote by Avg. This is in contrast with the proposed model, which aggregates embeddings of multiple posts using an attention mechanism. number of variations of the proposed model trained with triplet loss and all features (TPS) on samples of fixed or varying sizes as specified. These results demonstrate that a model trained on variable-sized samples appears to generalize well to much longer samples. We observe substantially better performance compared to the simple averaging baseline, and only a slight decrease in performance compared to the fixed length models as the evaluation sample size increases beyond the lengths seen at training time. As mentioned in 2.3, we use Beta (3 , 1) to select sample sizes during training. We hypothesize that a negatively skewed distribution tends to improve training efficiency by supplying longer samples most of the time, while retaining the ability to handle shorter samples. To evaluate this claim, we investigate several distributions of varying degrees of negative skew. Table 4 shows the ranking performance of variations of the proposed model trained on samples of sizes varying between 1 and 16 posts and evaluated on samples of size 16. These models differ only in the distribution used to select sample sizes. Indeed, the negatively skewed distributions do improve ranking performance over the uniform distribution, although the choice of negatively skewed distribution appears to be mostly immaterial. This work motivates a number of interesting research questions. First, the proposed model makes use of publication times, but only avails of the hour of the day. It would be interesting to examine continuous-time variants of our encoder that incorporate relative time differences between actions when aggregating their embeddings, in light of the fact that patterns of user activity might be highly discriminative. For example, bots and spammers typically post at certain times of day and with particular frequencies. Separately, the proposed data augmentation methods we use to handle variable-sized samples may also be applicable in other settings, such as multi-document summarization (Liu and Lapata, 2019). Finally, the scores we use to determine author matches could be calibrated , providing confidence estimates associated with the account linking decisions. To our knowledge, this work is the first to demonstrate the feasibility of a general-purpose account linking framework at web scale. Indeed, Figure 6 shows that performance improves as the size of the target increases, suggesting a speed-accuracy trade-off that can be tuned for different application settings. Expanding on an idea above, if confidence estimates were available, they could be used to inform the necessary sample sizes to achieve an acceptable level of risk. Finally, we note that the generality of the proposed approach make it potentially applicable to a wide range of applications, including source code attribution (Burrows et al., 2009; Yang et al., 2017; Kalgutkar et al., 2019), plagiarism detection (Pot-thast et al., 2010; Meuschke et al., 2018; Folt`ynek et al., 2019), and authorship attribution in collaborative documents (Flck and Acosta, 2014; Dauber et al., 2017).
method method objective abstain objective abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain objective abstain abstain method abstain method method abstain abstain abstain abstain abstain objective abstain abstain abstain method abstain abstain objective method objective method objective objective other abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method other abstain other abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain other other other other other abstain method abstain method other abstain method abstain abstain objective other other other abstain other abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other