Publicly traded companies are required to submit periodic reports with eXtensive Business Reporting Language ( xbrl ) word-level tags. Manually tagging the reports is tedious and costly. We, therefore, introduce xbrl tagging as a new entity extraction task for the financial domain and release f i ner -139, a dataset of 1.1M sentences with gold xbrl tags. Unlike typical entity extraction datasets, f i ner 139 uses a much larger label set of 139 entity types. Most annotated tokens are numeric, with the correct tag per token depending mostly on context, rather than the token itself. We show that subword fragmentation of numeric expressions harms bert 's performance, allowing word-level bilstm s to perform better. To improve bert 's performance, we propose two simple and e  ective solutions that replace numeric expressions with pseudo-tokens reflecting original token shapes and numeric magnitudes. We also experiment with fin bert , an existing bert model for the financial domain, and release our own bert ( sec bert ), pre-trained on financial filings, which performs best. Through data and error analysis, we finally identify possible limitations to inspire future work on xbrl tagging. Natural language processing ( nlp ) for finance is an emerging research area (Hahn et al., 2019; Chen et al., 2020; El-Haj et al., 2020). Financial data are mostly reported in tables,but substantial information can also be found in textual form, e.g., in company filings, analyst reports, and economic news. Such information is useful in numerous financial intelligence tasks, like stock market prediction (Chen et al., 2019; Yang et al., 2019), financial sentiment analysis (Malo et al., 2014; Wang et al., Source code: https://github.com/nlpaueb/finer Correspondence: eleftheriosloukas@aueb.gr Figure 1: Sentences from f i ner -139, with xbrl tags on numeric and non-numeric tokens. xbrl tags are actually xml -based and most tagged tokens are numeric. 2013; Akhtar et al., 2017), economic event detection (Ein-Dor et al., 2019; Jacobs et al., 2018; Zhai and Zhang, 2019), and causality analysis (Tabari et al., 2018; Izumi and Sakaji, 2019). In this work, we study how financial reports can be automatically enriched with word-level tags from the eXtensive Business Reporting Language ( xbrl ), a tedious and costly task not considered so far. 1 To promote transparency among shareholders and potential investors, publicly traded companies are required to file periodic financial reports. These comprise multiple sections, including financial tables and text paragraphs, called text notes . In addition, legislation in the us , the uk , the eu and elsewhere requires the reports to be annotated with tags of xbrl , an xml -based language, to facilitate the processing of financial information. The annotation of tables can be easily achieved by using company-specific pre-tagged table templates, since the structure and contents of the tables in the reports of a particular company rarely change. On the other hand, the unstructured and dynamic nature of text notes (Figure 1) makes adding xbrl tags to them much more di  cult. Hence, we focus on automatically tagging text notes. Tackling this task could facilitate the annotation of new and old reports (which may not include xbrl tags), e.g., 1 See https://www.xbrl.org/the-standard/what/ an-introduction-to-xbrl/ for an introduction to xbrl . Towards this direction, we release f i ner -139, a new dataset of 1.1M sentences with gold xbrl tags, from annual and quarterly reports of publicly traded companies obtained from the us Securities and Exchange Commission ( sec ). Unlike other entity extraction tasks, like named entity recognition ( ner ) or contract element extraction (Table 1), which typically require identifying entities of a small set of common types (e.g., persons, organiza-tions), xbrl defines approx. 6k entity types. As a first step, we consider the 139 most frequent xbrl entity types, still a much larger label set than usual. Another important di  erence from typical entity extraction is that most tagged tokens (  91%) in the text notes we consider are numeric, with the correct tag per token depending mostly on context, not the token itself (Figure 1). The abundance of numeric tokens also leads to a very high ratio of out-of-vocabulary ( oov ) tokens, approx. 10.4% when using a custom word 2 vec (Mikolov et al., 2013a) model trained on our corpus. When using subwords, e.g., in models like bert (Devlin et al., 2019), there are no oov tokens, but numeric expressions get excessively fragmented, making it di  cult for the model to gather information from the fragments and correctly tag them all. In our experiments, this is evident by the slightly better performance of stacked bilstm s (Graves et al., 2013; Lample et al., 2016) operating on word embeddings compared to bert . The latter improves when using a crf (La  erty et al., 2001) layer, which helps avoid assigning nonsensical sequences of labels to the fragments (subwords) of numeric expressions. To further improve bert 's performance, we propose two simple and e  ective solutions that replace numeric expressions with pseudo-tokens reflecting the original token shapes and magnitudes. We also experiment with fin bert (Yang et al., 2020), an existing bert model for the financial domain, and release our own family of bert models, pretrained on 200k financial filings, achieving the best overall performance. Our key contributions are: 1. We introduce xbrl tagging, a new financial nlp task for a real-world need, and we release f i ner -139, the first xbrl tagging dataset. 2 2. We provide extensive experiment bilstm s and bert with generic or in-domain pretraining, which establish strong baseline results for future work on f i ner -139. 3. We show that replacing numeric tokens with pseudo-tokens reflecting token shapes and magnitudes significantly boosts the performance of bert -based models in this task. 4. We release a new family of bert models ( sec bert , sec bert num , sec bert shape ) pre-trained on 200k financial filings that obtains the best results on f i ner -139. 3,4,5 2 Related Work Entity extraction: xbrl tagging di  ers from ner and other previous entity extraction tasks (Table 1), like contract element extraction (Chalkidis et al., 2019). Crucially, in xbrl tagging there is a much larger set of entity types (6k in full xbrl , 139 in f i ner -139), most tagged tokens are numeric (  91%), and the correct tag highly depends on context. In most ner datasets, numeric expressions are classified in generic entity types like amount' or date' (Bikel et al., 1999); this can often be achieved with regular expressions that look for common formats of numeric expressions, and the latter are often among the easiest entity types in ner datasets. By contrast, although it is easy to figure out that the first three highlighted expressions of Figure 1 are amounts, assigning them the correct xbrl tags requires carefully considering their context. Contract element extraction (Chalkidis et al., 2019) also requires considering the context of dates, amounts etc. to distinguish, for example, start dates from end dates, total amounts from other mentioned amounts, but the number of entity types in f i ner -139 is an order of magnitude larger (Table 1) and the full tag set of xbrl is even larger (6k). 2 https: // huggingface.co / datasets / nlpaueb / finer-139 3 https: // huggingface.co / nlpaueb / sec-bert-base 4 https: // huggingface.co / nlpaueb / sec-bert-num 5 https: // huggingface.co / nlpaueb / sec-bert-shape 4420 Financial ner : Previous financial ner applications use at most 9 (generic) class labels. Salinas Alvarado et al. (2015) investigated ner in finance to recognize organizations, persons, locations, and miscellaneous entities on 8 manually annotated sec financial agreements using crf s. Francis et al. (2019) experimented with transfer learning by unfreezing di  erent layers of a bilstm with a crf layer, pre-trained on invoices, to extract 9 entity types with distinct morphological patterns (e.g., iban , company name, date, total amount). Also, Hampton et al. (2015, 2016) applied a Maximum Entropy classifier, crf s, and handcrafted rules to London Stock Exchange filings to detect 9 generic entity types (e.g., person, organization, location, money, date, percentages). Finally, Kumar et al. (2016) extended the work of Finkel et al. (2005) and built a financial entity recognizer of dates, numeric values, economic terms in sec and nonsec documents, using numerous handcrafted text features. By contrast, f i ner -139 uses a specialized set of 139 highly technical economic tags derived from the real-world need of xbrl tagging, and we employ no handcrafted features. Numerical reasoning: Neural numerical reasoning studies how to represent numbers to solve numeracy tasks, e.g., compare numbers, understand mathematical operations mentioned in a text etc. Zhang et al. (2020) released n um bert , a Transformer-based model that handles numerical reasoning tasks by representing numbers by their scientific notation and applying subword tokenization. On the other hand, g en bert (Geva et al., 2020) uses the decimal notation and digit-by-digit tokenization of numbers. Both models attempt to deal with the problem that word-level tokenization often turns numeric tokens to oov s (Thawani et al., 2021). This is important, because numerical reasoning requires modeling the exact value of each numeric token. In f i ner -139, the correct xbrl tags of numeric tokens depend much more on their contexts and token shapes than on their exact numeric values (Fig. 1). Hence, these methods are not directly relevant. g en bert 's digit-by-digit tokenization would also lead to excessive fragmentation, which we experimentally find to harm performance. Traditionally, business filings were simply rendered in plain text. Thus, analysts and researchers needed to manually identify, copy, and paste each Figure 2: Frequency distribution of the 139 xbrl tags used in this work over the entire f i ner -139 dataset. amount of interest (e.g., from filings to spread-sheets). With xbrl -tagged filings, identifying and extracting amounts of interest (e.g., to spreadsheets or databases) can be automated. More generally, xbrl facilitates the machine processing of financial documents. Hence, xbrl -tagged financial reports are required in several countries, as already noted (Section 1). However, manually tagging reports with xbrl tags is tedious and resource-intensive. Therefore, we release f i ner -139 to fos-ter research towards automating xbrl tagging. f i ner -139 was compiled from approx. 10k annual and quarterly English reports (filings) of publicly traded companies downloaded from sec 's edgar system. 6 The downloaded reports span a 5-year period, from 2016 to 2020. They are annotated with xbrl tags by professional auditors and describe the performance and projections of the companies. We used regular expressions to extract the text notes from the Financial Statements Item of each filing, which is the primary source of xbrl tags in annual and quarterly reports. xbrl taxonomies have many di  erent attributes, making xbrl tagging challenging even for humans (Baldwin et al., 2006; Hoitash and Hoitash, 2018). Furthermore, each jurisdiction has its own xbrl taxonomy. Since we work with us documents, our labels come from us gaap . 7 Since this is the first e  ort towards automatic xbrl tagging, we chose to work with the most essential and informative attribute, the tag names , which populate our label set. Also, since xbrl tags change periodically, we selected the 139 (out of 6,008) most frequent xbrl tags with at least 1,000 appearances in f i ner -139. The distribution of these tags seems to follow a power law (Figure 2), hence most of the 6k xbrl tags that we did not consider are very rare. We used the iob 2 annotation scheme to distinguish tokens at the beginning, inside, or outside of tagged expressions, which leads to 279 possible token labels. We split the text notes into 1.8M sentences, the majority of which (  90%) contained no tags. 8 The sentences are also html -stripped, normalized, and lower-cased. To avoid conflating trivial and more di  cult cases, we apply heuristic rules to discard sentences that can be easily flagged as almost certainly requiring no tagging; in a real-life setting, the heuristics, possibly further improved, would discard sentences that do not need to be processed by the tagger. The heuristic rules were created by inspecting the training subset and include regular expressions that look for amounts and other expressions that are typically annotated. Approx. 40% of the 1.8M sentences were removed, discarding only 1% of tagged ones. We split chronologically the remaining sentences into training, development, and test sets with an 80 / 10 / 10 ratio (Table 2). spa C y (Honnibal et al., 2020) is an open-source nlp library. 9 It includes an industrial ner that uses word-level Bloom embeddings (Serr and Karat-zoglou, 2017) and residual Convolutional Neural Networks ( cnn s) (He et al., 2016). We trained spa C y 's ner from scratch on f i ner -139. bilstm : This baseline uses a stacked bidirectional Long-Short Term Memory ( lstm ) network (Graves et al., 2013; Lample et al., 2016) with residual connections. Each token t i of a sentence S is mapped to an embedding and passed through the bilstm stack to extract the corresponding contextualized embedding. A shared multinomial logistic regression ( lr ) layer operates on top of each contextualized embedding to predict the correct label. We use the word 2 vec embeddings (Mikolov et al., 2013a,b) of Loukas et al. (2021). bert : This is similar to bilstm , but now we fine-tune bert base (Devlin et al., 2019) to extract contextualized embeddings of subwords. Again, a multinomial lr layer operates on top of the contextualized embeddings to predict the correct label of the corresponding subword. crf s: In this case, we replace the lr layer of the previous two models with a Conditional Random Field ( crf ) layer (La  erty et al., 2001), which has been shown to be beneficial in several token labeling tasks (Huang et al., 2015; Lample et al., 2016; Chalkidis et al., 2020b). 10 5 Baseline Results We report micro-F 1 (  -F 1 ) and macro-F 1 (m-F 1 ) at the entity level, i.e., if a gold tag annotates a multi-word span, a model gets credit only if it tags the exact same span. This allows comparing more easily methods that label words vs. subwords. Table 3 shows that spa C y performs poorly, possibly due to the di  erences from typical token labeling tasks, i.e., the large amount of entity types, the abundance of numeric tokens, and the fact that in f i ner -139 the tagging decisions depend mostly on context. Interestingly enough, bilstm (with word embeddings) performs slightly better than bert . However, when a crf layer is added, bert achieves the best results, while the performance of bilstm (with word embeddings) deteriorates significantly, contradicting previous studies. We hypothesize that the inconsistent e  ect of crf s is due to tokenization di  erences. When using bert 's subword tokenizer, there are more decisions that need to be all correct for a tagged span to be correct (one decision per subword) than when using word tokenization (one decision per word). Thus, it becomes more di  cult for subword models to avoid nonsensical sequences of token labels, 10 We use a linear-chain crf layer with log-likelihood optimization and Viterbi decoding. e.g., labeling two consecutive subwords as beginning and inside of di  erent entity types, especially given the large set of 279 labels (Table 1). The crf layer on top of subword models helps reduce the nonsensical sequences of labels. On the other side, when using words as tokens, there are fewer opportunities for nonsensical label sequences, because there are fewer tokens. For instance, the average number of subwords and words per gold span is 2.53 and 1.04, respectively. Hence, it is easier for the bilstm to avoid predicting nonsensical sequences of labels and the crf layer on top of the bilstm (with word embeddings) has less room to contribute and mainly introduces noise (e.g., it often assigns low probabilities to acceptable, but less frequent label sequences). With the crf layer, the model tries to maximize both the confidence of the bilstm for the predicted label of each word and the probability that the predicted sequence of labels is frequent. When the bilstm on its own rarely predicts nonsensical sequences of labels, adding the crf layer rewards commonly seen sequences of labels, even if they are not the correct labels, without reducing the already rare nonsensical sequences of labels. To further support our hypothesis, we repeated the bilstm experiments, but with subword (instead of word) embeddings, trained on the same vocabulary with bert . Without the crf , the subword bilstm performs much worse than the word bil stm (6 p.p drop in  -F 1 ), because of the many more decisions and opportunities to predict nonsensical label sequences. The crf layer substantially improves the performance of the subword bilstm (4.9 p.p. increase in  -F 1 ), as expected, though the word bilstm (without crf ) is still better, because of the fewer opportunities for nonsensical predictions. A drawback of crf s is that they significantly slow down the models both during training and inference, especially when using large label sets (Goldman and Goldberger, 2020), as in our case. Hence, although bert with crf was the best model in Table 3, we wished to improve bert 's performance further without employing crf s. In f i ner -139, the majority (91.2%) of the gold tagged spans are numeric expressions, which cannot all be included in bert 's finite vocabulary; e.g., the token  9,323.0 ' is split into five subword units, [ 9 ',  ##, ',  ##323 ',  ##. ',  ##0 '] , while the token 12.78 ' is split into [ 12 ',  ##. ',  ##78 '] . The excessive fragmentation of numeric expressions, when using subword tokenization, harms the performance of the subword-based models (Table 3), because it increases the probability of producing nonsensical sequences of labels, as already discussed. We, therefore, propose two simple and e  ective solutions to avoid the over-fragmentation of numbers. bert + [ num ] : We detect numbers using regular expressions and replace each one with a single [ num ] pseudo-token, which cannot be split. The pseudo-token is added to the bert vocabulary, and its representation is learned during fine-tuning. This allows handling all numeric expressions in a uniform manner, disallowing their fragmentation. bert + [ shape ] : We replace numbers with pseudo-tokens that cannot be split and represent the number's shape. For instance,  53.2 ' becomes  [XX . X] ', and  40,200.5 ' becomes  [XX,XXX . X] '. We use 214 special tokens that cover all the number shapes of the training set. Again, the representations of the pseudo-tokens are fine-tuned, and numeric expressions (of known shapes) are no longer fragmented. The shape pseudo-tokens also capture information about each number's magnitude; the intuition is that numeric tokens of similar magnitudes may require similar xbrl tags. Figure 3 illustrates the use of [ num ] and [ shape ] pseudo-tokens. Driven by the recent findings that pre-training language models on specialized domains is beneficial for downstream tasks (Alsentzer et al., 2019; Belt-4423 agy et al., 2019; Yang et al., 2020; Chalkidis et al., 2020b), we explore this direction in our task which is derived from the financial domain. fin bert : We fine-tune fin bert (Yang et al., 2020), which is pre-trained on a financial corpus from sec documents, earnings call transcripts, and analyst reports. 11 The 30k subwords vocabulary of fin bert is built from scratch from its pre-training corpus. Again, we utilize fin bert with and without our numeric pseudo-tokens, whose representations are learned during fine-tuning. sec bert : We also release our own family of bert models. Following the original setup of Devlin et al. (2019), we pre-trained bert from scratch on edgar corpus , a collection of financial documents released by Loukas et al. (2021). The resulting model, called sec bert , has a newly created vocabulary of 30k subwords. To further examine the impact of the proposed [ num ] and [ shape ] special tokens, we also pre-trained two additional bert variants, sec bert num and sec bert shape , on the same corpus, having replaced all numbers by [ num ] or [ shape ] pseudo-tokens, respectively. In this case, the representations of the pseudo-tokens are learned during pre-training and they are updated during fine-tuning. Table 4 reports micro-averaged precision, recall, and F 1 on development and test data. As with Table 3, a lr layer is used on top of each embedding to predict the correct label, unless specified otherwise. 11 We use the finbert finvocab uncased version from https: // github.com / yya518 / FinBERT. Focusing on the second zone, we observe that the [ num ] pseudo-token improves bert 's results, as expected, since it does not allow numeric expressions to be fragmented. The results of bert + [ num ] are now comparable to those of bert + crf . Performance improves further when utilizing the shape pseudo-tokens ( bert + [ shape ] ), yielding 79.4  -F 1 and showing that information about each number's magnitude is valuable in xbrl tagging. Interestingly, fin bert (3rd zone) performs worse than bert despite its pre-training on financial data. Similarly to bert , this can be attributed to the fragmentation of numbers (2.5 subwords per gold tag span). Again, the proposed pseudo-tokens ( [ num ] , [ shape ] ) alleviate this problem and allow fin bert to leverage its in-domain pre-training in order to finally surpass the corresponding bert variants, achieving an 80.1  -F 1 test score. Our new model, sec bert (last zone), which is pre-trained on sec reports, performs better than the existing bert and fin bert models, when no numeric pseudo-tokens are used. However, sec bert is still worse than bert with numeric pseudo-tokens (75.7 vs. 78.3 and 79.4 test  -F 1 ), su  ering from number fragmentation (2.4 subwords per gold tag span). sec bert (without pseudo-tokens) also performs worse than the bilstm with word embeddings (75.7 vs. 77.3  -F 1 , cf. Table 3). However, when the proposed pseudo-tokens are used, sec bert num and sec bert shape achieve the best overall performance, boosting the test  -F 1 to 80.4 and 82.1, respectively. This indicates that learning to handle numeric expressions during model pretraining is a better strategy than trying to acquire this knowledge only during fine-tuning. An alternative way to bypass word fragmentation is to use subword pooling for each word. cs et al. (2021) found that for ner tasks, it is better to use the first subword only, i.e., predict the label of an entire word from the contextualized embedding of its first subword only; they compared to several other methods, such as using only the last subword of each word, or combining the contextualized embeddings of all subwords with a self-attention mechanism. Given this finding, we conducted an ablation study and compare (i) our best model ( sec bert ) with first subword pooling (denoted sec bert first ) to (ii) sec bert with our special tokens ( sec bert num , sec bert shape ), which avoid segmenting numeric tokens. Table 5 shows that, in xbrl tagging, using the proposed special tokens is comparable ( sec bert num ) or better ( sec bert shape ) than performing first pooling ( sec bert first ). It might be worth trying other pooling strategies as well, like last -pooling or subword self-attention pooling. It's worth noting, however, that the latter will increase the training and inference times. To further investigate the e  ectiveness of our pseudo-tokens, we incorporated them in the bil stm operating on subword embeddings (3rd model of Table 3). Again, we replace each number by a single [ num ] pseudo-token or one of 214 [ shape ] pseudo-tokens, for the two approaches, respectively. These replacements also happen when pretraining word 2 vec subword embeddings; hence, an embedding is obtained for each pseudo-token. Table 6 shows that bilstm num outperforms the bilstm subword model. bilstm shape further improves performance and is the best bilstm subword model overall, surpassing the subword bilstm with crf , which was the best subword bilstm model in Table 3. These results further support our hypothesis that the [ num ] and [ shape ] pseudo-tokens help subword models successfully generalize over numeric expressions, with [ shape ] being the best of the two approaches, while also avoiding the over-fragmentation of numbers. Since xbrl tagging is derived from a real-world need, it is crucial to analyze the model's performance in a business use case. After consulting with experts of the financial domain, we concluded that one practical use case would be to use an xbrl tagger as a recommendation engine that would propose the k most probable xbrl tags for a specific token selected by the user. The idea is that an expert (e.g., accountant, auditor) knows beforehand the token(s) that should be annotated and the tagger would assist by helping identify the appropriate tags more quickly. Instead of having to select from several hundreds of xbrl tags, the expert would only have to inspect a short list of k proposed tags. We evaluate our best model, sec bert shape , in this use case using Hits@ k . We use the model to return the k most probable xbrl tags for each token that needs to be annotated, now assuming that 4425 the tokens to be annotated are known. If the correct tag is among the top k , we increase the number of hits by one. Finally, we divide by the number of tokens to be annotated. Figure 4 shows the results for di  erent values of k . The curve is steep for k = 1 to 5 and saturates as k approaches 10, where Hits@ k is nearly perfect (99.4%). In practice, this means that a user would have to inspect 10 recommended xbrl tags instead of hundreds for each token to be annotated; and in most cases, the correct tag would be among the top 5 recommended ones. We also performed an exploratory data and error analysis to unveil the peculiarities of f i ner -139, extract new insights about it, and discover the limitations of our best model. Specifically, we manually inspected the errors of sec bert shape in under-performing classes (where F 1 < 50%) and identified three main sources of errors. Specialized terminology: In this type of errors, the model is able to understand the general financial semantics, but does not fully comprehend highly technical details. For example, Operating Lease Expense amounts are sometimes missclassified as Lease And Rental Expense , i.e., the model manages to predict that these amounts are about expenses in general, but fails to identify the specific details that distinguish operating lease expenses from lease and rental expenses. Similarly, Payments to Acquire Businesses (Net of Cash Acquired) amounts are mostly misclassified as Payments to Acquire Businesses (Gross) . In this case, the model understands the notion of business acquisition, but fails to di  erentiate between net and gross payments. Financial dates: Another interesting error type is the misclassification of financial dates. For example, tokens of the class Debt Instrument Maturity Date are mostly missclassified as not belonging to any entity at all ( O' tag). Given the previous type of errors, one would expect the model to miss-classify these tokens as a di  erent type of financial date, but this is not the case here. We suspect that errors of this type may be due to annotation inconsistencies by the financial experts. Annotation inconsistencies: Even though the gold xbrl tags of f i ner -139 come from professional auditors, as required by the Securities & Exchange Commission ( sec ) legislation, there are still some discrepancies. We provide an illustrative Figure 5: A manually inspected sentence from f i ner 139 showing some inconsistencies in the gold xbrl tags of the auditors. example in Figure 5. We believe that such inconsistencies are inevitable to occur and they are a part of the real-world nature of the problem. We hope that this analysis inspires future work on xbrl tagging. For example, the specialized terminology and financial date errors may be alleviated by adopting hierarchical classifiers (Chalkidis et al., 2020a; Manginas et al., 2020), which would first detect entities in coarse classes (e.g., expenses, dates) and would then try to classify the identified entities into finer classes (e.g., lease vs. rent expenses, instrument maturity dates vs. other types of dates). It would also be interesting to train classifiers towards detecting wrong (or missing) gold annotations, in order to help in quality assurance checks of xbrl -tagged documents. We introduced a new real-word nlp task from the financial domain, xbrl tagging, required by regulatory commissions worldwide. We released f i ner 139, a dataset of 1.1M sentences with xbrl tags. Unlike typical entity extraction tasks, f i ner -139 uses a much larger label set (139 tags), most tokens to be tagged are numeric, and the correct tag depends mostly on context rather than the tagged token. We experimented with several neural classifiers, showing that a bilstm outperforms bert due to the excessive numeric token fragmentation of the latter. We proposed two simple and e  ective solutions that use special tokens to generalize over the shapes and magnitudes of numeric expressions. We also experimented with fin bert , an existing 4426 bert model for the financial domain, which also benefits from our special tokens. Finally, we pretrained and released our own domain-specific bert model, sec bert , both with and without the special tokens, which achieves the best overall results with the special tokens, without costly crf layers. In future work, one could hire experts to reannotate a subset of the dataset to measure human performance against the gold tags. Future work could also consider less frequent xbrl tags (few-and zero-shot learning) and exploit the hierarchical dependencies of xbrl tags, possibly with hierarchical classifiers, building upon our error analysis.
abstain abstain objective abstain abstain result objective method method abstain abstain other method abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain objective abstain abstain result objective method result abstain method