Graph neural networks have triggered a resurgence of graph-based text classification methods, defining today's state of the art. We show that a wide multi-layer perceptron (MLP) using a Bag-of-Words (BoW) outperforms the recent graph-based models TextGCN and HeteGCN in an inductive text classification setting and is comparable with HyperGAT. Moreover, we fine-tune a sequence-based BERT and a lightweight DistilBERT model, which both outperform all state-of-the-art models. These results question the importance of synthetic graphs used in modern text classifiers. In terms of efficiency, DistilBERT is still twice as large as our BoW-based wide MLP, while graph-based models like TextGCN require setting up an O ( N 2 ) graph, where N is the vocabulary plus corpus size. Finally, since Transformers need to compute O ( L 2 ) attention weights with sequence length L , the MLP models show higher training and inference speeds on datasets with long sequences. Text categorization is the task of assigning topical categories to text units such as documents, social media postings, or news articles. Research on text categorization is a very active field as just the sheer amount of new methods in recent surveys shows (Bayer et al., 2021; Li et al., 2020; Zhou et al., 2020; Kowsari et al., 2019; Kadhim, 2019). There are approaches based on a Bag of Words (BoW) that perform text categorization purely on the basis of a multiset of tokens. Among them are Deep Averaging Networks (DAN) (Iyyer et al., 2015), a deep Multi-Layer Perceptron (MLP) model with n layers that relies on averaging the BoW, Simple Word Embedding Models (SWEM) (Shen et al., 2018) that explores different pooling strategies for pretrained word embeddings, and fastText (Bojanowski et al., 2017), which uses a linear layer on top of pretrained word embeddings. These models count the occurrence of all tokens in the input sequence, while disregarding word position and order, and then rely on word embeddings and fully connected feedforward layer(s). We call these BoW-based models . Among the most popular recent methods for text categorization are graph-based models such as TextGCN (Yao et al., 2019) that first induce a synthetic word-document co-occurence graph over the corpus and subsequently apply a graph neural network (GNN) to perform the classification task. Besides TextGCN, there are follow-up works like HeteGCN (Ragesh et al., 2021), TensorGCN (Liu et al., 2020), and HyperGAT (Ding et al., 2020), which we collectively call graph-based models . Finally, there is the well-known Transformer (Vaswani et al., 2017) universe with models such as BERT (Devlin et al., 2019) and its size-reduced variants such as DistilBERT (Sanh et al., 2019). Here, the input is a (fixed-length) sequence of tokens, which is then fed into multiple layers of self-attention. Lightweight versions such as DistilBERT and others (Tay et al., 2020; Fournier et al., 2021) use less parameters but operate on the same type of input. Together with recurrent models such as LSTMs, we call these sequence-based models . In this paper, we hypothesize that text categorization can be very well conducted by simple but effective BoW-based models. We investigate this research question in three steps: First, we conduct an in-depth analysis of the literature. We review the key research in the field of text categorization. From this analysis, we derive the different families of methods, the established benchmark datasets, and identify the top performing methods. We decide for which models we report numbers from the literature and which models we run on our own. Overall, we compare 16 different methods from the families of BoW-based models (8 methods), sequence-based models (3 methods), and graph-based models (5 methods). We run our own experi-4038 ments for 7 of these methods on 5 text categorization datasets, while we report the results from the literature for the remaining methods. The result is surprising: Our own BoW-based MLP, called the WideMLP, with only one wide hidden layer, outperforms many of the recent graph-based models for inductive text categorization (Yao et al., 2019; Liu et al., 2020; Ragesh et al., 2021). Moreover, we did not find any reported scores for BERT-based methods from the sequence-based family. Thus, we fine-tuned our own BERT (Devlin et al., 2019) and DistilBERT (Sanh et al., 2019). These models set a new state of the art. On a meta-level, our study shows that MLPs have largely been ignored as competitor methods in experiments. It seems as if MLPs have been forgotten as baseline in the literature, which instead is focusing mostly on other advanced Deep Learning architectures. Considering strong baselines is, however, an important means to argue about true scientific advancement (Shen et al., 2018; Dacrema et al., 2019). Simple models are also often preferred in industry due to lower operational and maintenance costs. Below, we introduce our methodology and results from the literature study. Subsequently, we introduce the families of models in Section 3. Thereafter, we describe the experimental procedure in Section 4. We present the results of our experiments in Section 5 and discuss our findings in Section 6, before we conclude. Methodology In a first step, we have analyzed recent surveys on text categorization and comparison studies (Minaee et al., 2021; Bayer et al., 2021; Li et al., 2020; Zhou et al., 2020; Kowsari et al., 2019; Kadhim, 2019; Galke et al., 2017; Zhang et al., 2016). These cover the range from shallow to deep classification models. Second, we have screened for literature in key NLP and AI venues. Finally, we have complemented our search by checking results and papers on paperswithcode.com. On the basis of this input, we have determined three families of methods and benchmark datasets (see Table 2). We focus our analysis on identifying models per family showing strong performance and select the methods to include in our study. For all models, we have verified that the same train-test split is used. We check whether modified versions of the datasets have been used (e. g., fewer classes), to avoid bias and wrongfully giving advantages. BoW-based Models Classical machine learning models that operate on a BoW-based input are extensively discussed in two surveys (Kowsari et al., 2019; Kadhim, 2019) and other comparison studies (Galke et al., 2017). Iyyer et al. (2015) proposed DAN, which combine word embeddings and deep feedforward networks. It is an MLP with 1-6 hidden layers, non-linear activation, dropout, and Ada-Grad as optimization method. The results suggest to use pretrained embeddings such as GloVe (Pen-nington et al., 2014) over a randomly initialized neural bag of-words (Kalchbrenner et al., 2014) as input. In fastText (Bojanowski et al., 2017; Joulin et al., 2017) a linear layer on top of pretrained embeddings is used for classification. Furthermore, Shen et al. (2018) explore embedding pooling variants and find that SWEM can rival approaches based on recurrent (RNN) and convolutional neural networks (CNN). We consider fastText, SWEM, and a DAN-like deeper MLP in our comparison. Note that those approaches that rely on logistic regression on top of pretrained word embeddings, e. g., fastText, share a similar architecture as an MLP with one hidden layer. However, the standard training protocol involves pretraining the word embedding on large amounts of unlabeled text and then freezing the word embeddings while training the logistic regression (Mikolov et al., 2013). Graph-based Models Using graphs induced from text for the task of text categorization has a long history in the community. An early work is the term co-occurrence graph of the KeyGraph algorithm (Ohsawa et al., 1998). The graph is split into segments, representing the key concepts in the document. Co-occurence graphs have also been used for automatic keyword extraction such as in RAKE (Rose et al., 2010) and can be also used for classification (Zhang et al., 2021). Modern approaches exploit this idea in combination with graph neural networks (GNN) (Hamil-ton, 2020). Examples of GNN-based methods operating on a word-document co-occurence graph are TextGCN (Yao et al., 2019) and its successor TensorGCN (Liu et al., 2020) as well as HeteGCN (Ragesh et al., 2021), HyperGAT (Ding et al., 2020), and DADGNN (Liu et al., 2020). We briefly discuss these models: In TextGCN, the authors set up a graph based on word-word connections given by window-based pointwise mutual information and word-document TF-IDF scores. They use a one-hot encoding as node features and apply a 4039 two-layer graph convolutional network (Kipf and Welling, 2017) on the graph to carry out the node classification task. HeteGCN combines ideas from Predictive Text Embedding (Tang et al., 2015) and TextGCN and split the adjacency matrix into its word-document and word-word sub-matrices and fuse the different layers' representations when required. TensorGCN uses multiple ways of converting text data into graph data including a semantic graph created with an LSTM, a syntactic graph created by dependency parsing, and a sequential graph based on word co-occurrence. HyperGAT extended the idea of text-induced graphs for text classification to hypergraphs. The model uses graph attention and two kinds of hyperedges. Sequential hyperedges represent the relation between sentences and their words. Semantic hyperedges for word-word connections are derived from topic models (Blei et al., 2001). Finally, DADGNN is a graph-based approach that uses attention diffusion and decoupling techniques to tackle oversmoothing of the GNN and to be able to stack more layers. In TextGCN's original transductive formulation, the entire graph including the test set needs to be known for training. This may be prohibitive in practical applications as each batch of new documents would require retraining the model. When these methods are adapted for inductive learning, where the test set is unseen, they achieve notably lower scores (Ragesh et al., 2021). GNNs for text classification use corpus statistics, e. g., pointwise mutual information (PMI), to connect related words in a graph (Yao et al., 2019). When these were omitted, the GNNs would collapse to bag-of-words MLPs. Thus, GNNs have access to more information than BoW-MLPs. GloVe (Pennington et al., 2014) also captures PMI corpus statistics, which is why we include an MLP on GloVe input representations. Sequence models: RNN and CNN Recurrent neural networks (RNN) are a natural choice for any NLP task. However, it turned out to be challenging to find numbers reported on text categorization in the literature that can be used as references. The bidirectional LSTM with two-dimensional max pooling BLSTM-2DCNN (Zhou et al., 2016) has been applied on a stripped-down to 4 classes version of the 20ng dataset. Thus, the high score of 96 . 5 reported for 4ng cannot be compared with papers applied on the full 20ng dataset. Also Text-RCNN (Lai et al., 2015), a model combining recurrence and convolution uses only the 4 major categories in the 20ng dataset. The results of Text-RCNN are identical with BLSTM-2DCNN. For the MR dataset, BLSTM-2DCNN provides no information on the specific split of the dataset. RNN-Capsule (Wang et al., 2018) is a sentiment analysis method reaching an accuracy of 83 . 8 on the MR dataset, but with a different train-test split. Lyu and Liu (2020) combine a 2D-CNN with bidirectional RNN. Another work applying a combination of a convolutional layer and an LSTM layer is by Wang et al. (2019b). The authors experiment with five English and two Chinese datasets, which are not in the set of representative datasets we identified. The authors report that their approach outperforms existing models like fastText on two of the five English datasets and both Chinese datasets. Sequence models: Transformers Surprisingly, only few works consider Transformer models for text categorization. A recent work shows that BERT outperforms classic TF-IDF BoW approaches on English, Chinese, and Portuguese text classification datasets (Gonzlez-Carvajal and Garrido-Merchn, 2020). We have not found any results of transformer-based models reported on those text categorization datasets that are commonly used in the graph-based approaches. Therefore, we fine-tune BERT (Devlin et al., 2019) and DistilBERT (Sanh et al., 2019) on those datasets ourselves. BERT is a large pretrained language model on the basis of Transformers. DistilBERT (Sanh et al., 2019) is a distilled version of BERT with 40% reduced parameters while retaining 97% of BERT's language understanding capabilities. TinyBERT (Jiao et al., 2020) and Mo-bileBERT (Sun et al., 2020) would be similarly suitable alternatives, among others. We chose DistilBERT because it can be fine-tuned independently from the BERT teacher. Its inference times are 60% faster than BERT, which makes it more likely to be reusable by labs with limited resources. Summary From our literature survey, we see that all recent methods are based on graphs. BoW-based methods are hardly found in experiments, while, likewise surprisingly, Transformer-based sequence models are extremely scarce in the literature on topical text categorization. The recent surveys on text categorization include both classical and Deep Learning models, but none considered a simple MLP except for the inclusion of DAN (Iyyer et al., 2015) in Li et al. (2020). We formally introduce the three families of models for text categorization, namely the BoW-based, graph-based, and sequence-based models. Table 1 summarizes the key properties of the approaches: whether they require a synthetic graph, whether word position is reflected in the model, whether the model can deal with arbitrary length text, and whether the model is capable of inductive learning. Under pure BoW-based text categorization, we denote approaches that are not order-aware and operate only on the multiset of words from the input document. Given paired training examples ( x , y )  D , each consisting of a bag-of-words x  R n vocab and a class label y  Y , the goal is to learn a generalizable function  y = f (BoW)  ( x ) with parameters  such that arg max( y ) preferably equals the true label y for input x . As BoW-based model, we consider a one hidden layer WideMLP (i. e., two layers in total). We experiment with pure BoW, TF-IDF weighted, and averaged GloVe input representations. We also use a two hidden layers WideMLP-2. We list the numbers for fastText, SWEM, and logistic regression from Ding et al. (2020) in our comparison. Graph-based text categorization approaches first set up a synthetic graph on the basis of the text corpus D in the form of an adjacency matrix  A := make-graph( D ) . For instance, in TextGCN the graph is set up in two parts: word-word connections are modeled by pointwise mutual information and word-document edges resemble that the word occurs in the document. Then, a parameterized function f (graph)  ( X ,  A ) is learned that uses the graph as input, where X are the node features. The graph is composed of word and document nodes, each receiving its own embedding (by setting X = I ). In inductive learning, however, there is no embedding of the test documents. Note that the graph-based approaches from the current literature such as TextGCN also disregard word order, similar to the BoW-based models described above. A detailed discussion of the connection between TextGCN and MLP is provided in Appendix B. We consider top performing graph-based models from the literature, namely TextGCN along with its successors HeteGCN, TensorGCN, HyperGAT, DADGNN, as well as simplified GCN (SGC) (Wu et al., 2019). We do not run our own experiments for the graph-based models but rely on the original work and extensive studies by Ding et al. (2020) and Ragesh et al. (2021). We consider RNNs, LSTMs, and Transformers as sequence-based models. These models are aware of the order of the words in the input text in the sense that they are able to exploit word order information. Thus, the key difference to the BoW-based and graph-based families is that the word order is reflected by sequence-based model. The model signature is  y = f (sequence)  ( (cid:104) x 1 , x 2 , . . . , x k (cid:105) ) , where k is the (maximum) sequence length. Word position is modeled by a dedicated positional encoding. For instance, in BERT each position is associated with an embedding vector that is added to the word embedding at input level. For the sequence-based models, we run our own experiments with BERT and DistilBERT, while reporting the scores of a pretrained LSTM from Ding et al. (2020) for comparison. We use the same datasets and train-test split as in TextGCN (Yao et al., 2019). Those datasets are 20ng, R8, R52, ohsumed, and MR. Twenty 4041 Newsgroups (20ng) 1 (bydate version) contains long posts categorized into 20 newsgroups. The mean sequence length is 551 words with a standard deviation (SD) of 2,047. R8 and R52 are subsets of the Reuters 21578 news dataset with 8 and 52 classes, respectively. The mean sequence length and SD is 119  128 words for R8, and 126  133 words for R52. Ohsumed 2 is a corpus of medical abstracts from the MEDLINE database that are categorized into diseases (one per abstract). The mean sequence length is 285  123 words. Movie Reviews (MR) 3 (Pang and Lee, 2005), split by Tang et al. (2015), is a binary sentiment analysis dataset on sentence level (mean sequence length and SD: 25  11 ). Table 2 shows the dataset characteristics. In the transductive setup, as used in TextGCN, the test documents are visible and actually used for the preprocessing step. In the inductive setting, the test documents remain unseen until test time (i. e., they are not available for preprocessing). We report the scores of the graph-based models for both setups from the literature, where available. BoW-based and sequence-based models are inherently inductive. Ragesh et al. (2021) have evaluated a variant of TextGCN that is capable of inductive learning, which we include in our results, too. We have extracted accuracy scores from the literature according to our systematic selection from Section 2. Below, we provide a detailed description of the procedure for the models that we have run ourselves. We borrow the tokenization strategy 1 http://qwone.com/~jason/20Newsgroups/ 2 http://disi.unitn.it/moschitti/ corpora.htm 3 https://www.cs.cornell.edu/people/ pabo/movie-review-data/ from BERT (Devlin et al., 2019) along with its uncased vocabulary. The tokenizer relies primarily on WordPiece (Wu et al., 2016) for a high coverage while maintaining a small vocabulary. Training our BoW-Models. Our WideMLP has one hidden layer with 1,024 rectified linear units (one input-to-hidden and one hidden-to-output layer). We apply dropout after each hidden layer, notably also after the initial embedding layer. Only for GloVe+WideMLP, neither dropout nor ReLU is applied to the frozen pretrained embeddings but only on subsequent layers. The variant WideMLP-2 has two ReLU-activated hidden layers (three layers in total) with 1 , 024 hidden units each. While this might be overparameterized for single-label text classification tasks with few classes, we rely on recent findings that overparameterization leads to better generalization (Neyshabur et al., 2018; Nakkiran et al., 2020). In pre-experiments, we realized that MLPs are not very sensitive to hyperparameter choices. Therefore, we optimize cross-entropy with Adam (Kingma and Ba, 2015) and its default learning rate of 10  3 , a linearly decaying learning rate schedule and train for a high amount of steps (Nakkiran et al., 2020) (we use 100 epochs) with small batch sizes (we use 16) for sufficient stochasticity, along with a dropout ratio of 0 . 5 . Fine-tuning our BERT models. For BERT and DistilBERT, we fine-tune for 10 epochs with a linearly decaying learning rate of 5  10  5 and an effective batch size of 128 via gradient accumulation of 8 x 16 batches. We truncate all inputs to 512 tokens. To isolate the influence of word order on BERT's performance, we conduct two further ablations. First, we set all position embeddings to zero and disable their gradient ( BERT w/o pos ids ). By doing this, we force BERT to operate on a bag-of-words without any notion of word order or position. Second, we shuffle each sequence to augment the training data. We use this augmentation strategy to increase the number of training examples by a factor of two ( BERT w/ shuf. augm. ). We report accuracy as evaluation metric, which is equivalent to Micro-F1 in single-label classification (see Appendix C). We repeat all experiments five times with different random initialization of the parameters and report the mean and standard deviation of these five runs. Table 3 shows the accuracy scores for the text categorization models on the five datasets. All graph-based models in the transductive setting show similar accuracy scores (maximum difference is 2 points). As expected, the scores decrease in the inductive setting up to a point where they are matched or even outperformed by our WideMLP. In the inductive setting, the WideMLP models perform best among the BoW models, in particular, TFIDF+WideMLP and WideMLP on an unweighted BoW. The best-performing graph-based model is HyperGAT, yet DADGNN has a slight advantage on R8, R52, and MR. For the sequence-based models, BERT attains the highest scores, closely followed by DistilBERT. The strong performance of WideMLP rivals all graph-based techniques reported in the literature, in particular, the recently published graph-inducing methods. MLP only falls behind HyperGAT, which relies on topic models to set up the graph. Another observation is that 1 hidden layer (but wide) is sufficient for the tasks, as the scores for MLP variants with 2 hidden layers are lower. We further observe that both pure BoW and TF-IDF weighted BoW lead to better results than approaches that exploit pretrained word embeddings such as GloVe-MLP, fastText, and SWEM. With its immense pretraining, BERT yields the overall highest scores, closely followed by DistilBERT. DistilBERT outperforms HyperGAT by 7 points on the MR dataset while being on par on the others. BERT outperforms the strongest graph-based competitor, HyperGAT, by 8 points on MR, 1.5 points on ohsumed, 1 point on R52 and R8, and 0.5 points on 20ng. Our results further confirm that position embeddings are important for BERT with a notable decrease when those are omitted. Augmenting the data with shuffled sequences has led to neither a consistent decrease nor increase in performance. Parameter Count of the Models Table 4 lists the parameter counts of the models. Even though the MLP is fully-connected on top of a bag-of-words with the dimensionality of the vocabulary size, it has only half of the parameters as DistilBERT and a quarter of the parameters of BERT. Using TF-IDF does not change the number of model parameters. Due to the high vocabulary size, GloVe-based models have a high number of parameters, but the majority of those is frozen, i. e., does not get gradient updates during training. provide the total running times in Table 5 as observed while conducting the experiments on a single NVIDIA A100-SXM4-40GB card. All WideMLP variants are an order of magnitude faster than DistilBERT when considering the average runtime per epoch . DistilBERT is twice as fast as the original BERT. The transformers are only faster than BoW models on the MR dataset. This is because the sequences in the MR dataset are much shorter and less O ( L 2 ) attention weights have to be computed. Key Insights Our experiments show that our MLP models using BoW outperform the recent graph-based models TextGCN and HeteGCN in an inductive text classification setting. Furthermore, the MLP models are comparable to HyperGAT. Only transformer-based BERT and DistilBERT models outperform our MLP and set a new state-of-the-art. This result is important for two reasons: First, the strong performance of a pure BoW-MLP questions the added value of synthetic graphs in models like TextGCN to the text categorization task. Only HyperGAT, which uses the expensive Latent Dirichlet Allocation for computing the graph, slightly outperforms our BoW-WideMLP in two out of five datasets. Thus, we argue that using strong baseline models for text classification is important to assess the true scientific advancement (Dacrema et al., 2019). Second, in contrast to conventional wisdom (Iyyer et al., 2015), we find that pretrained embeddings, e. g., GloVe, can have a detrimental effect when compared to using an MLP with one wide hidden layer. Such an MLP circumvents the bottleneck of the small dimensionality of word embeddings and has a higher capacity. Furthermore, we experiment with more hidden layers (see WideMLP-2), but do not observe any improvement when the single hidden layer is sufficiently wide. A possible explanation is that already a single hidden layer is sufficient to approximate any compact function to an arbitrary degree of accuracy depending on the width of the hidden layer (Cybenko, 1989). ing. However, as our efficiency analysis shows, the MLPs require only a fraction of the parameters and are faster in their combined training and inference time except for the MR dataset. The attention mechanism of (standard) Transformers is quadratic in the sequence length, which leads to slower processing of long sequences. With larger batches, the speed of the MLP could be increased even further. Detailed Discussion of Results Graph-based models come with high training costs, as not only the graph has to be first computed, but also a GNN has to be trained. For standard GNN methods, the whole graph has to fit into the GPU memory and mini-batching is nontrivial, but possible with dedicated sampling techniques for GNNs (Fey et al., 2021). Furthermore, the original TextGCN is inherently transductive, i. e., it has to be retrained whenever new documents appear. Strictly transductive models are effectively useless in practice (Lu et al., 2019) except for applications, in which a partially labeled corpus needs to be fully annotated. However, recent extensions such as HeteGCN, HyperGAT, and DADGNN already relax this constraint and enable inductive learning. Nevertheless, word-document graphs require O ( N 2 ) space, where N is the number of documents plus the vocabulary size, which is a hurdle for large-scale applications. There are also tasks where the natural structure of the graph data provides more information than the mere text, e. g., citations networks or connections in social graphs. In such cases, the performance of graph neural networks is the state of the art (Kipf and Welling, 2017; Velickovic et al., 2018) and are superior to MLPs that use only the node features and not the graph structure (Shchur et al., 2018). GNNs also find application in various NLP tasks, other than classification (Wu et al., 2021). An interesting factor is the ability of the models to capture word order. BoW models disregard word order entirely and yield good results, but still fall behind order-aware Transformer models. In an extensive study, Conneau et al. (2018) have shown 4044 Table 5: Total runtime (training+inference). that memorizing the word content (which words appear at all) is most indicative of downstream task performance. Sinha et al. (2021) have experimented with pretraining BERT by disabling word order during pretraining and show that it makes surprisingly little difference for fine-tuning. In their study, word order is preserved during fine-tuning. In our experiments, we have conducted complementary experiments: we have used a BERT model that is pretrained with word order, but we have deactivated the position encoding during fine-tuning. Our results show that there is a notable drop in performance but the model does not fail completely. Other NLP tasks such as question answering (Ra-jpurkar et al., 2016) or natural language inference (Wang et al., 2019a) can also be regarded as text classification on a technical level. Here, the positional information of the sequence is more important than for pure topical text categorization. One can expect that BoW-based models perform worse than sequence-based models. Generalizability We expect that similar observations would be made on other text classification datasets because we have already covered a range of different characteristics: long and short texts, topical categorization (20ng, Reuters, and Ohsumed) and sentiment prediction (MR) in the domains of forum postings, news, movie reviews, and medical abstracts. Our results are in line with those from other fields, who have reported a resurgence of MLPs. For example, in business prediction, an MLP baseline outperforms various other Deep Learning models (Venugopal et al., 2021; Yedida et al., 2021). In computer vision, Tolstikhin et al. (2021) and Melas-Kyriazi (2021) proposed attention-free MLP models that are on par with the Vision Transformer (Dosovitskiy et al., 2021). In natural language processing, Liu et al. (2021a) show similar results, while acknowledging that a small attention module is necessary for some tasks. Threats to Validity We acknowledge that the experimental datasets are limited to English. While word order is important in the English language, it is notable that methods that discard word order still work well for text categorization. Another possible bias is the comparability of the results. However, we carefully checked all relevant parameters such as the train/test split, the number of classes in the datasets, if datasets have been pre-processed in such a way that, e. g., makes a task easier like reducing the number of classes, the training procedure, and the reported evaluation metrics. Regarding our efficency analysis, we made sure to report numbers for the parameter count and a measure for the speed other than FLOPs, as recommended by Dehghani et al. (2021). Since runtime is heavily dependant on training parameters such as batch size, we complement this with asymptotic complexity. has an immediate impact on practitioners who seek to employ robust text categorization models in research projects and in industrial operational environments. Furthermore, we advocate to use an MLP baseline in future text categorization research, for which we provide concrete guidelines in Appendix A. As future work, it would be interesting to analyze multi-label classification tasks and to compare with hierarchical text categorization methods (Peng et al., 2018; Xiao et al., 2019). Another interesting yet challenging setting would be few-shot classification (Brown et al., 2020). We argue that a wide multi-layer perceptron enhanced with today's best practices should be considered as a strong baseline for text classification tasks. In fact, the experiments show that our WideMLP is oftentimes on-par or even better than recently proposed models that synthesize a graph structure from the text. The source code is available online: https://github.com/lgalke/ text-clf-baselines The focus of this work is text classification. Potential risks that apply to text classification in general also apply to this work. Nonetheless, we present alternatives to commonly used pretrained language models, which suffer from various sources of bias due to the large and poorly manageable data used for pretraining (Bender et al., 2021). In contrast, the presented alternatives render full control over the training data and, thus, contribute to circumvent the biases otherwise introduced during pretraining.
abstain result result abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain method abstain method result result result abstain abstain result abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result method abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain result abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain method objective other abstain method method abstain