Detecting disclosures of individuals' employment status on social media can provide valuable information to match job seekers with suitable vacancies, offer social protection, or measure labor market flows. However, identifying such personal disclosures is a challenging task due to their rarity in a sea of social media content and the variety of linguistic forms used to describe them. Here, we examine three Active Learning (AL) strategies in real-world settings of extreme class imbalance, and identify five types of disclosures about individuals' employment status (e.g. job loss) in three languages using BERT-based classification models. Our findings show that, even under extreme imbalance settings, a small number of AL iterations is sufficient to obtain large and significant gains in precision, recall, and diversity of results compared to a supervised baseline with the same number of labels. We also find that no AL strategy consistently outperforms the rest. Qualitative analysis suggests that AL helps focus the attention mechanism of BERT on core terms and adjust the boundaries of semantic expansion, highlighting the importance of interpretable models to provide greater control and visibility into this dynamic learning process. Up-to-date information on individuals' employment status is of tremendous value for a wide range of economic decisions, from firms filling job vacancies to governments designing social protection systems. At the aggregate level, estimates of labor market conditions are traditionally based on nationally representative surveys that are costly to produce, especially in lowand middle-income countries (Devarajan, 2013; Jerven, 2013). As social media becomes more ubiquitous all over the world, more individuals can now share their employment status with peers and unlock the social capital of their networks. This, in turn, can provide a new lens to examine the labor market and devise Lost Job Was hired Job offer Is unemployed Looking for a job Needing a job cuz I dnt like nt makin money. policy, especially in countries where traditional measures are lagging or unreliable. A key challenge in using social media to identify personal disclosures of employment status is that such statements are extremely rare in an abundance of social media content  roughly one in every 10,000 posts  which renders random sampling ineffective and prohibitively costly for the development of a large labeled dataset. On the other hand, simple keyword-based approaches run the risk of providing seemingly high-accuracy classifiers while substantially missing linguistic variety used to describe events such as losing a job, looking for a job, or starting a new position (see Figure 1 for example). In the absence of a high-quality, comprehensive, and diverse ground-truth about personal employment disclosures, it is difficult to develop classification models that accurately capture the flows in and out of the labor market in any country, let alone robustly estimating it across multiple countries. Furthermore, state-of-the-art deep neural models provide little visibility into or control over the linguistic patterns captured by the model, which hampers the ability of researchers and practitioners to determine whether the model has truly learned new linguistic forms and sufficiently converged. Active Learning (AL) is designed for settings where there is an abundance of unlabeled examples and limited labeling resources (Cohn et al., 1994). It aims to focus the learning process on the most informative samples and maximize model perfor-6564 mance for a given labeling budget. In recent years, AL proved successful in several settings, including policy-relevant tasks involving social media data (Pohl et al., 2018; Palakodety et al., 2020). The success of pre-trained language models such as BERT (Devlin et al., 2019) in a variety of language understanding tasks has sparked interest in using AL with these models for imbalanced text classification. Yet, most research in this field has focused on artificially-generated rarity in data or imbalance that is not as extreme as the present setting (Ein-Dor et al., 2020; Schrder et al., 2021). Therefore, there is no evidence of the efficiency of AL using BERT-based models for sequence classification in real-world settings with extreme imbalance. It is unclear whether some AL strategies will perform significantly better than others in these settings, how quickly the different strategies will reach convergence (if at all), and how the different strategies will explore the linguistic space. In this work, we leverage BERT-based models (Devlin et al., 2019) in three different AL paradigms to identify tweets that disclose an individual's employment status or change thereof. We train classifiers in English, Spanish, and Portuguese to determine whether the author of a tweet recently lost her job, was recently hired, is currently unemployed, posting to find a job, or posting a job offer. We use two standard AL strategies, Uncertainty Sampling (Lewis and Gale, 1994) and Adaptive Retrieval (Mussmann et al., 2020), and propose a novel strategy we name Exploit-Explore Retrieval that uses k-skip-n-grams (n-grams with k skipped tokens) to explore the space and provide improved interpretability. We evaluate the models both quantitatively and qualitatively across languages and AL strategies, and compare them to a supervised learning baseline with the same number of labels. Therefore, our contributions are:  An evaluation of three AL strategies for BERT-based binary classification under extreme class imbalance using real-world data. A novel AL strategy for sequence classification that performs on par with other strategies, but provides additional interpretability and control over the learning process. A qualitative analysis of the linguistic patterns captured by BERT across AL strategies. A large labeled dataset of tweets about unemployment and fine-tuned models in three languages to stimulate research in this area 1 . Social media users disclose information that is valuable for public policy in a variety of areas ranging from health (Achrekar et al., 2011; Mahata et al., 2018; Klein et al., 2018) to emergency response to natural disasters (Bruns and Liang, 2012; Kry-vasheyeu et al., 2016) through migration flows (Fio-rio et al., 2017; Chi et al., 2020; Palotti et al., 2020). A key challenge in identifying self-disclosures on social media is the rare and varied nature of such content with a limited labeling budget. Prior work that studied self-disclosures on Twitter had either used pattern matching, which is prone to large classification errors (Antenucci et al., 2014; Proserpio et al., 2016), or focused on curated datasets (Li et al., 2014; Preotiuc-Pietro et al., 2015; Sarker et al., 2018; Ghosh Chowdhury et al., 2019), which provide no guarantees about recall or coverage of the positive class. These issues are more severe in real-world settings of extreme imbalance, where random sampling is unlikely to retrieve any positives, let alone diverse. These challenges motivate the use of AL, as described next. AL has been used successfully in various settings to maximize classification performance for a given labeling budget (see Settles (1995) for a survey). With the emergence of pre-trained language models such as BERT (Devlin et al., 2019) and their success across a number of different language tasks, recent work has studied the combination of AL and BERT, either by using BERT to enhance traditional AL methods (Yuan et al., 2020) or by applying established AL methods to improve BERT's classification performance (Zhang and Zhang, 2019; Shelmanov et al., 2019; Liu et al., 2020; Griehaber et al., 2020; Prabhu et al., 2021; Schrder et al., 2021). In the specific case of binary classification with moderate class imbalance, Ein-Dor et al. (2020) show that AL with BERT significantly outperforms random sampling but that no single AL strategy stands out in terms of BERT-based classification performance, both for balanced and imbalanced 1 Labeled datasets and models can be found at https://github.com/manueltonneau/ twitter-unemployment 6565 settings. Yet, the authors only consider a relatively moderate class imbalance of 10-15% positives, and does not cover extreme imbalance, which is common in many text classification tasks. Our current research examines a considerably more extreme imbalance of about 0.01% positives, where traditional AL approaches can be ineffective (Attenberg and Provost, 2010). Under this extreme imbalance, Mussmann et al. (2020) show the potential of AL for BERT to outperform random sampling for pairwise classification. To the best of our knowledge, this work is the first to compare the performance of AL methods for BERT-based sequence classification in real-world extreme imbalance settings. Our dataset was collected from the Twitter API. It contains the timelines of the users with at least one tweet in the Twitter Decahose and with an inferred profile location in the United States, Brazil, and Mexico. In addition to the United States, we chose to focus on Brazil and Mexico as both of them are middle-income countries where Twitter's penetration rate is relatively high. For each country, we drew a random sample of 200 million tweets covering the period between January 2007 and December 2020 and excluding retweets. We then split it evenly in two mutually exclusive random samples R e and R s . In the following sections, we use R e to evaluate each model's performance in a real-world setting and R s to sample new tweets to label. Our labeling process sought to identify four nonexclusive, binary states that workers may experience during their career: losing a job (Lost Job), being unemployed (Is Unemployed), searching for a job (Job Search), and finding a job (Is Hired). We only considered first-person disclosures as positives. For the classes Lost Job and Is Hired, we only considered such events that happened in the past month as positives as we want to determine the user's current employment status. To complement the focus on workers, we also labeled tweets containing job offers ("Job Offer"). We used Amazon Mechanical Turk (MTurk) to label tweets according to these 5 classes (see Figure 1 and Section A.2 for details). As previously stated, the extreme imbalance of our classification task of one positive example for every 10,000 tweets renders random sampling ineffective and prohibitively costly. In order to build high-performing classifiers at a reasonable cost, we selected a set of 4 to 7 seed keywords that are highly specific of the positives and frequent enough for each class and country. To do so, we defined a list of candidate seeds, drawing from Antenucci et al. (2014) for the US and asking native speakers in the case of Mexico and Brazil, and individually evaluated their specificity and frequency (see Section A.1 for additional details). We then randomly sampled 150 tweets containing each seed from R s , allowing us to produce a stratified sample L 0 of 4,524 English tweets, 2703 Portuguese tweets, and 3729 Spanish tweets respectively (Alg. 1). We then labeled each tweet using Amazon Mechanical Turk (MTurk) allowing us to construct a language-specific stratified sample that is common to the 5 classes (see Section A.3 for descriptive statistics of the stratified sample). We trained five binary classifiers to predict each of the five aforementioned labeled classes. Preliminary analysis found that BERT-based models considerably and consistently outperformed keyword-based models, static embedding models, and the combination of these models. We benchmarked several BERT-based models and found that the following models gave the best performance on our task: Conversational BERT for English tweets (Burtsev et al., 2018), BERTimbau for Brazilian Portuguese tweets (Souza et al., 2020) and BETO for Mexican Spanish tweets (Caete et al., 2020) (see Section A.4 for details on model selection). We fine-tuned each BERT-based model on a 70:30 train-test split of the labeled tweets for 20 epochs (Alg. 1). Following Dodge et al. (2020), we repeated this process for 15 different random seeds and retained the best performing model in terms of area under the ROC curve (AUROC) on the test set at or after the first epoch (see Section A.5 for details). While the standard classification performance measure in an imbalanced setting is the F1 score with a fixed classification threshold (e.g. 0.5), it is not applicable in our case for two reasons. First, we care about the performance on a large random set of tweets and the only labeled set we could compute the F1 metric from is the stratified test set 6566 which is not representative of the extremely imbalanced random sample R e . Second, the fact that neural networks are poorly calibrated (Guo et al., 2017) makes the choice of a predefined classification threshold somewhat arbitrary and most likely sub-optimal. We developed an alternative threshold-setting evaluation strategy. First, we computed the predicted score of each tweet in R e (Alg. 1), which is a random sample. Then, for each class, we labeled 200 tweets in R e along the score distribution (see section A.7.1 for more details). We measured the performance of each classifier on R e by computing:  the Average Precision as common in information retrieval. the number of predicted positives , defined as the average rank in the confidence score distribution when the share of positives reaches 0.5. the diversity , defined as the average pairwise distance between true positives. Details about the evaluation metrics can be found in Section A.7. Initialization : for each seed s , sample 150 tweets containing s from R s ; have them labeled for the five classes; the resulting labeled set is the stratified sample L 0 = S 0 ; discard already sampled tweets from R s ( R s = R s  L 0 ) At each iteration i and for each class:  Finetuning : train-test split of S i in 70/30; finetune 15 BERT models on the train set using different seeds; select the best model M  i with the highest AUROC on the test set. Inference on R e and R s using M  i  Active Learning : sample most informative tweets from R s (100 per class); have them labeled for the five classes; the resulting labeled set is L i +1 ; define S i +1 = (cid:83) i +1 j =0 L j and R s = R s  L i +1  Evaluation: sample tweets along the score distribution in R e ; have them labeled; compute the average precision, number of predicted positives and diversity metrics Algorithm 1: Experimental procedure 3.5 Active Learning strategies Next, we used pool-based AL (Settles, 1995) in batch mode, with each class-specific fine-tuned model as the classification model, in order to query new informative tweets in R s . We compared three different AL strategies aiming to balance the goal of improving the precision of a classifier while expanding the number and the diversity of detected positives instances:  Uncertainty Sampling consists in sampling instances that a model is most uncertain about. In a binary classification problem, the standard approach is to select examples with a predicted score close to 0.5 (Settles, 2009). In practice, this rule of thumb might not always lead to identify uncertain samples when imbalance is high (Mussmann et al., 2020), especially with neural network models known to be poorly calibrated (Guo et al., 2017). To overcome this issue, we contrast a naive approach which consists in querying the 100 instances whose uncalibrated scores are the closest to 0.5, to an approach that uses calibrated scores (see Section A.9 for details). Adaptive Retrieval aims to maximize the precision of a model by querying instances for which the model is most confident of their positivity (Mussmann et al., 2020). This approach is related to certainty sampling (Attenberg et al., 2010). Here, we select the 100 tweets whose predicted score is the highest for each class. Our novel strategy, Exploit-Explore Retrieval (see Section A.8 for details), aims to maximize precision (exploitation') while improving recall by feeding new and diverse instances at each iteration (exploration'):  Exploitation : Randomly query 50 new tweets from the top 10 4 tweets with the highest predicted score in R s . Exploration : Identify the 10 k-skip-n-grams with the highest frequency of occurrences in the top 10 4 tweets, relative to their frequency in R s . Then, randomly sample 50 new tweets containing each k-skip-n-gram (see Section A.8 for formal definition of k-skip-n-grams and a discussion on the choice of threshold). Additionally, we compared these AL strategies to a supervised Stratified Sampling baseline, that consists of the same initial motifs defined in Section 3.2 and the same number of labels as available to all other AL strategies. Overall, for each strategy, each iteration and each class, we labeled 100 new tweets in R s . We then combined the 500 new labels across classes with the existing ones to finetune and evaluate a new BERT-based model for 6567 each class as described in Section 3.3, which we then used to select tweets for labeling for the next iteration. We considered that an AL strategy had converged when there was no significant variation of average precision, number of predicted positives and diversity for at least two iterations (see Section A.7.6 for details). At iteration 0, we fine-tuned a BERT-based classifier on a 70:30 train-test split of the initialization sample L 0 for each class and country. All the AUROC values on the test set are reported in Table 7. We obtain very high AUROCs ranging from 0.944 to 0.993 across classes and countries. Job Offer has the highest AUROCs with values ranging from 0.985 for English to 0.991 for Portuguese and 0.993 for Spanish. Upon closer examination of positives for this class, we find that the linguistic structure of tweets mentioning job offers is highly repetitive, a large share of these tweets containing sentences such as We're #hiring! Click to apply: or naming job listing platforms (e.g: #Ca-reerArc). By contrast, the most difficult class to predict is Lost Job, with an AUROC on the test set equal to 0.959 for English and 0.944 for Spanish. This class also has the highest imbalance, with approximately 6% of positives in the stratified sample for these two languages. Taken together, these results show that a fine-tuned BERT model can achieve very high classification performance on a stratified sample of tweets across classes and languages. However, these num-bers cannot be extrapolated to directly infer the models' performance on random tweets, which we discuss in the next section. Next, we compared the performance of our exploit-explore retrieval strategy on English, Spanish and Portuguese tweets. We used exploit-explore retrieval as it provides similar results to other strategies (Section 4.3), while allowing greater visibility into selected motifs during the development process (Section 4.4). We ran 8 AL iterations for each language and report the results in Fig. 2, Fig. 5 and Table 10. First, we observe substantial improvements in average precision (AP) across countries and classes with just one or two iterations. These improvements are especially salient in cases where precision at iteration 0 is very low. For instance, for the English Is Unemployed class and the Spanish Is Hired class, average precision goes respectively from 0.14 and 0.07 to 0.83 and 0.8 from iteration 0 to iteration 1 (Fig. 2 and Fig. 5). A notable exception to this trend is the class Job Offer, especially for English and Portuguese. These performance differences can in part be explained by the varying quality of the initial seed list across classes. This is confirmed by the stratified sampling baseline performance discussed in 4.3. In the case of Job Offer, an additional explanation discussed earlier in Section 4.1 is the repetitive structure of job offers in tweets which makes this class easier to detect compared to others. Also, the class Lost Job has the worst performance in terms of AP across countries. One reason is that the data imbalance for this class is even higher than for other classes, as mentioned in Section 4.1. Another explanation for the low precision is the ambiguity inherent to the recency constraint, namely that an individual must have lost her job at most one month prior to posting the tweet. Apart from the Job Offer class in English and Portuguese, AL consistently allows to quickly expand from iteration 0 levels with the number of predicted positives multiplied by a factor of up to 10 4 (Fig. 2). Combined with high AP values, this result means that the classifiers manage to capture substantially more positives compared to iteration 0. This high expansion is combined with increasing semantic diversity among true positive instances. The class Job Offer stands out with little expansion and diversity changes in the English and Portuguese cases. For Spanish, expansion and diversity changes are higher. One explanation is that the structure of Mexican job offers is less repetitive, with individual companies frequently posting job offers, as opposed to job aggregators in the case of the US and Brazil. Overall, apart from a few edge cases, we find that AL used with pre-trained language models is successful at significantly improving precision while expanding the number and the diversity of predicted positive instances in a small number of iterations across languages. Indeed, precision gains reach up to 90 percentage points from iteration 0 to the last iteration across languages and classes and the number of predicted positives is multiplied 6568 Figure 2: Average precision, number of predicted positives and diversity of true positives (in row) for each class (in column) for English (green), Portuguese (orange), and Spanish (purple). by a factor of up to 10 4 . Furthermore, on average, the model converges in only 5.6 iterations across classes for English and Portuguese, and in 4.4 iterations for Spanish (see Table 10 for details). In this section, we evaluated on English tweets the stratified sampling baseline and the four AL strategies described in Section 3.5, namely exploit-explore retrieval, adaptive retrieval and uncertainty sampling with and without calibration. We ran five iterations for each strategy and reported the results on Figure 3 in this section as well as Table 11 and Figure 6 in Section A.10. We find that AL brings an order of magnitude more positives and does so while preserving or improving both the precision and the diversity of results. Apart from the Job Offer class discussed in Section 4.2, AL consistently outperforms the stratified sampling baseline. This is especially true for the classes Is Unemployed and Lost Job where the baseline performance stagnates at a low level, suggesting a poor seed choice, but also holds for classes Is Hired and Job Search with stronger baseline performance. We also find that no AL strategy consistently dominates the rest in terms of precision, number and diversity of positives. The gains in performance are similar across AL strategies, and are particularly high for the classes Lost Job and Is Unemployed, which start with a low precision. The number of predicted positives and the diversity measures also follow similar trends across classes and iterations. We also observe occasional drops in average precision of more than 25% from one iteration to the next. Uncalibrated uncertainty sampling seems particularly susceptible to these drops, with at least one occurrence for each class. Upon examination of the tweets sampled for labeling by this strategy, the vast majority of tweets are negatives and when a few positives emerge, their number is not large enough to allow the model to generalize well. This variability slows down the convergence process of uncertainty sampling when it is not uncalibrated (table 11). In contrast, calibrated uncertainty sam-6569 Figure 3: Average precision, number of predicted positives and diversity of true positives (in row) for each class (in column) across AL strategies. pling is less susceptible to these swings, emphasizing the importance of calibration for more stable convergence in settings of extreme imbalance. Taken together, our quantitative results show that the positive impact of AL on classification performance in an extremely imbalanced setting holds across AL strategies. Aside from a few occasional performance drops, we find significant gains in precision, expansion and diversity across strategies. Yet, we find that no AL strategy consistently dominates the others across a range of prediction tasks for which the number and the linguistic complexity of positive instances vary widely. Next, we investigate the results qualitatively to gain deeper understanding of the learning process. We qualitatively examined the tweets selected for labeling by each strategy to understand better what BERT-based models capture and reflect on the quantitative results. We focused on English tweets only and took a subsample of tweets at each iteration to better understand each strategy's performance. We excluded the Job Offer class from this analysis since the performance, in this case, is exceptionally high, even at iteration 0. Our analysis finds that many tweets queried by the various AL strategies capture a general tone that is present in tweets about unemployment, but that is not specific to one's employment status. For example, these include tweets of the form of I'm excited to ... in two days for the recently hired class, I've been in a shitty mood for ... for unemployment or I lost my ... for job loss. This type of false positives seems to wane down as the AL iterations progress, which suggests that a key to the success of AL is first to fine-tune the attention mechanism to focus on the core terms and not the accompanying text that is not specific to employment status. In the stratified sampling case, the focus on this unemployment tone remains uncorrected, explaining the poor performance for classes Lost Job and Is Unemployed and the performance drops for Is Hired and Job Search. A second theme in tweets queried by AL in-6570 volves the refinement of the initial motifs. Uncertainty sampling (calibrated and uncalibrated), adaptive retrieval, and the exploitation part of our exploit-explore retrieval method seem to query tweets that either directly contain a seed motif or a close variant thereof. For example, tweets for the class Lost Job may contain the seed motifs laid off, lost my job, and just got fired. As mentioned in Section 4.2 to explain occasional drops in performance, many tweets labeled as negatives contain over-generalization of the semantic concept such as expanding to other types of losses (e.g. lost my phone), other types of actions (e.g. got pissed off), or simply miss the dependence on first-person pronouns (e.g. @user got fired). Many of the positively labeled tweets contain more subtle linguistic variants that do not change the core concept such as I really need a job, I really need to get a job, I need to find a job, or I need a freaken job. Adaptive retrieval chooses these subtle variants more heavily than other strategies with some iterations mostly populated with I need a job variants. Overall, these patterns are consistent with a view of the learning process, specifically the classification layer of the BERT model, as seeking to find the appropriate boundaries of the target concept. Finally, the exploration part of the exploit-explore retrieval makes the search for new forms of expression about unemployment more explicit and interpretable. For example, the patterns explored in the first few iterations of explore-exploit retrieval include I ... lost ... today, quit .. my .. job, I ... start my ... today, and I'm ... in ... need. A detailed presentation of the explored k-skip-n-grams for US tweets can be found in Table 9 of Section A.8. While this strategy suffers from issues that also affect other AL strategies, we find that the explore part of exploit-explore retrieval is more capable of finding new terms that were not part of the seed list (e.g., quit, career) and provides the researcher with greater insight into and control over the AL process. This work developed and evaluated BERT-based models in three languages and used three different AL strategies to identify tweets related to an individual's employment status. Our results show that AL achieves large and significant improvements in precision, expansion, and diversity over stratified sampling with only a few iterations and across languages. In most cases, AL brings an order of magnitude more positives while preserving or improving both the precision and diversity of results. Despite using fundamentally different AL strategies, we observe that no strategy consistently outperforms the rest. Within the extreme imbalance setting, this is in line with  and complements  the findings of Ein-Dor et al. (2020). Additionally, our qualitative analysis and exploration of the exploit-explore retrieval give further insights into the performance improvements provided by AL, finding that substantial amounts of queried tweets hone the model's focus on employment rather than surrounding context and expand the variety of motifs identified as positive. This puts exploit-explore retrieval as a valuable tool for researchers to obtain greater visibility into the AL process in extreme imbalance cases without compromising on performance. While the present work demonstrates the potential of AL for BERT-based models under extreme imbalance, an important direction for future work would be to further optimize the AL process. One could for instance study the impact on performance of the stratified sample size or the AL batch size. To overcome the poor seed quality for some classes, other seed generation approaches could be tested, such as mining online unemployment forums using topic modeling techniques to discover different ways to talk about unemployment. In terms of model training and inference, the use of multitask learning for further performance improvement could be studied due to the fact that classes of unemployment are not mutually exclusive. We hope that our experimental results as well as the resources we make available will help bridge these gaps in the literature. We acknowledge that there is some risk, like any other technology that makes inferences at the individual level, that the technology presented here will be used for harm. However, due to the public nature of the content and the fact that the potential harm already exists using basic keyword search, we believe that the marginal risk added by our classifier is minimal. We thank participants of the Israeli Seminar on Computational Linguistics at Ben-Gurion University of the Negev as well as the anonymous reviewers for their valuable comments. We also thank Aleister Montfort, Varnitha Kurli Reddy and Boris Sobol for their excellent research assistance. This work was supported by the SDG Partnership Fund.
abstain abstain method result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain method objective objective objective objective other other other other other other other other other abstain other objective abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain result abstain abstain result abstain abstain abstain abstain abstain result result result abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result abstain result abstain objective abstain abstain abstain abstain abstain result method method other other other