The enrichment of tabular datasets using external sources has gained significant attention in recent years. Existing solutions, however, either ignore external unstructured data completely or devise dataset-specific solutions. In this study, we proposed Few-Shot Transformer based Enrichment (FeSTE), a generic and robust framework for the enrichment of tabular datasets using unstructured data. By training over multiple datasets, our approach is able to develop generic models that can be applied to additional datasets with minimal training (i.e., few-shot). Our approach is based on an adaptation of BERT, for which we present a novel fine-tuning approach that reformulates the tuples of the datasets as sentences. Our evaluation, conducted on 17 datasets, shows that FeSTE is able to generate high quality features and significantly outperform existing fine-tuning solutions. Tabular data is the most diverse format of data representation, spanning domains from nutrition to banking. It does, however, suffer from a lack of contextual information that could make its analysis more effective. Data scientists seek to overcome this limitation by using feature engineering (FE), which involves applying transformations on existing features to create additional representations of the data. When the available data is not sufficiently diverse (or when additional improvement is sought), one may attempt to use external information sources to enrich the data. We refer to this process as external enrichment of datasets (EED). The use of external sources for feature engineering is both computationally-heavy and time consuming. The process first involves matching entities in the data to those in the external source, a process known as Entity Linking (Shen et al., 2014). Once entities in the external source have been matched, candidate features need to be generated, evaluated, and finally integrated into the tabular dataset. While multiple studies in recent years (Paulheim and Fmkranz, 2012; Ristoski et al., 2015; Friedman and Markovitch, 2018; Mountantonakis and Tzitzikas, 2017; Galhotra et al., 2019; Harari and Katz, 2022) have sought to automate the EED process, a large majority focuses solely on structured external sources, e.g., DBpedia tables, and do not attempt to use the large amounts of available unstructured data (i.e., free text). In this study, we present Few-Shot Transformer based Enrichment (FeSTE) a generic and robust framework for the enrichment of tabular datasets using unstructured data. Our approach utilizes transformer-based, pre-trained Language Models (LM) (Devlin et al., 2018) to identify and prioritize promising candidate features in the external data source. FeSTE then applies a novel process of analyzing the relationships between the unstructured features and the dataset's target class values, and automatically generating new tabular features. To overcome the difficulty imposed by datasets of limited size, we train FeSTE on multiple datasets in order to create a generic model that can later be applied to additional datasets. Additionally, we propose a novel fine-tuning (FT) process that enables pre-trained LM to quickly adapt to new datasets (i.e., perform few-shot learning). The result of this process is a more robust model that is also more effective on small datasets. While previous studiesTAPAS (Herzig et al., 2020), TaBERT (Yin et al., 2020), TURL (Deng et al., 2020), and TPN (Wang et al., 2021)have attempted to use Transformers for analyzing tabular data, FeSTE focuses on analyzing the connection between external texts and the dataset's entities. We are therefore able to leverage the Transformer architecture to generate additional features and fine-tune the generation process in a novel way. diverse characteristics (number of samples, feature composition, etc.). For our evaluation, we use BERT as the Transformer architecture and Wikipedia as the external source, with its page abstracts as our unstructured texts. Our results show that FeSTE outperforms existing BERT fine-tuning strategies and that FeSTE is highly effective, achieving an average improvement of 9.2% when combined with the datasets' original features. Finally, we show FeSTE performs well even when it is applied on its own (without any original fea-tures), achieving an average AUC of 0.664. To summarize, our contributions in this study are as follows:  Our work is the first to propose a generic and fully-automated approach for tabular data enrichment using unstructured external sources. We propose a novel few-shot fine-tuning approach for transformer-based pre-trained LM, which performs well even for training sets consisting of as little as tens of samples. We make our code publicly available. The large majority of work in the field of automated features generation from external information sources mainly focuses on leveraging structured data. For example, (Paulheim and Fmkranz, 2012) uses structured data from knowledge bases (KB) such as DBpedia to generate new features, which are then used to augment tabular datasets. RapidMiner (Ristoski et al., 2015) processes KB of structured tabular and graphical data by modeling the relations among their entities. Friedman et al. (Friedman and Markovitch, 2018) focus on features generation for text classification problems. They leverage structured data from two KBs: FreeBase (Bollacker et al., 2008) and YAGO2 (Hoffart et al., 2013). The authors first identify each entity in the text, and then recursively explore the KB to extract new features. The LodsyndesisML framework (Mountantonakis and Tzitzikas, 2017) leverages KB's (e.g DBpedia) to create thousands of new features for classification tasks using nine operators. Each operator creates different types of features, which are then used to enrich the original data. Galhotra et el. (Galhotra et al., 2019) use structured web data to generate new features for classification and regression tasks. Their approach generates thousands of candidate features, then selects the final set using information theory-based measures such as Information Gain and Pearson correlation. To the best of our knowledge, the only study to utilize both structured and unstructured sources is the recently proposed FGSES framework (Harari and Katz, 2022). FGSES extracts features from both structured and unstructured DBpedia content, generates thousands of candidate features, and then uses a meta learning-based approach to rank them and return a small final set. While this approach is the most similar to the one proposed in this study, there are significant differences: FeSTE focuses on the analysis of the texts, performs fine-tuning rather than relying on a general model, and takes into account the context of analyzed datasets. Moreover, our approach generates a small set of features and is, therefore, more computationally efficient. Wikipedia is widely used as an external source of information due to its availability, richness, and diversity (Lehmann et al., 2015). An important addition to Wikipedia from an entity linking standpoint is DBpedia, a project that extracts Wikipedia data and makes it accessible in a more structured form. DBpedia is used as an external data source for feature engineering by multiple studies (Paulheim and Fmkranz, 2012; Ristoski et al., 2015; Galhotra et al., 2019; Mountantonakis and Tzitzikas, 2017) because of its accessible format. To utilize DBpedia for feature engineering in tabular data, one should first link the entities in the analyzed dataset to unique DBpedia entities. DBpedia Spotlight (Mendes et al., 2011) is a tool for automatically identifying and linking textual entities to ones on DBpedia. Unfortunately, DBpedia Spotlight tends to capture entities whose name consists only of one or two words, while ignoring entities composed of longer sequences. In recent years, Transformers (Vaswani et al., 2017) and other deep learning-based approaches are being applied in the field of semantic relatedness, in order to link free texts to DBpedia. Blink (Wu et al., 2020) is a BERT-based (Devlin et al., 2018) approach which receives a mention and its surrounding text, and links the mention to its corre-1578 sponding DBpedia entity. It should be noted, however, that the names of DBpedia entities tend to be shorter than in free text, which hampers Blink's performance. Recently, (Harari and Katz, 2022) developed an entity linking algorithm whose aim is to link entities in tabular datasets with Wikipedia pages. We analyze the performance of this approach in Section 5.3. One of the most influential developments in the field of NLP in recent years is the emergence of Transformer-based LM (Vaswani et al., 2017). BERT (Devlin et al., 2018) and its various extensions, GPT (Radford et al., 2018) and XLnet (Yang et al., 2019) achieve state-of-the-art (SOTA) performance on a variety of tasks, including text classification, question answering, next word prediction, and more. Unfortunately, training these models requires expensive hardware and very large amounts of data. For this reasons, the large majority of studies and applications use pre-trained versions of these models. However, fine tuning (FT) these models, i.e., additional limited training on data from the task at hand, has been shown to improve performance (Gururangan et al., 2020). Studies such as (Sun et al., 2019) and (Gururan-gan et al., 2020) propose three FT strategies: (1) Task-specific , in which one trains the pre-trained LM on similar ML task (e.g text classification); (2) Domain-specific , where the pre-trained LM is trained on a similar domain (e.g biology), and; (3) Target task-specific , where the final training step is performed on the targeted dataset directly. The aforementioned studies report a significant improvement in BERT's performance, especially where multi-phase FT was performed. Another fine-tuning strategy called Multi-Task Deep Neural Network (MT-DNN) training was proposed by (Liu et al., 2019): for each task in each training step, the approach modifies the output layer but keeps the lower layers unchanged. This approach can also be applied in cases where the training and target datasets have different characteristics, e.g., different number of classes. One significant drawback of MT-DNN is the need to replace and train the final layer (e.g., the softmax layer) whenever it is applied to a new problem. This approach has two potential shortcomings: first, given that FT mainly affect BERT's final layers (Gururan-gan et al., 2020), some loss of earlier knowledge may occur. Secondly, MT-DNN needs to maintain separate final layers for each task during training (three different heads in the original study). In cases where MT-DNN is training on a large number of datasets, this could pose problems in terms of memory consumption. A different FT approach that does not require task-specific layers was proposed by (Wei et al., 2021), who used an "instruction FT" phase. The authors added instructions (i.e., statements) to the text, and required the model to determine whether the statements are correct. While effective, analysis shows that the approach is only applicable to very large datasets, and the addition of over 8B parameters to the already large architecture of 137B. In contrast to all the aforementioned studies, FeSTE uses a single architecture for its training process, regardless of the number of datasets used. Moreover, we propose a novel dataset reformulation process that enables us to apply the same architecture on all datasets, regardless of their number of classes. This approach enables a much more efficient FT process, as shown in Section 5.3. For our task of feature generation from the free text of external sources, we assume a target tabular dataset D t with a classification tasks. Additionally, we assume a set of pre-analyzed tabular datasets with different classification tasks (i.e. different number of classes) D = { D 1 ...D n } . For each dataset, let there be target class values tc i and original features F i . Of F i , let there be at least one feature representing entities e i = { e i, 1 ...e i,m } . For the purpose of generating new features, we assume an external data source EX which consists of entities e ex and text related to these entities. We denote this set of texts as T . For the purpose of linking e i and e ex , we assume an entity linking function  . We generate a set of new features f newt from T using a Language Model LM . Overview. Our proposed approach is presented in Figure 1 and Algorithm 1. FeSTE consists of three main phases: a) entity linking ; b) fine-tuning , and; c) features generation . In the entity linking phase, FeSTE automatically matches entity names in the tabular dataset to their corresponding entries in the external data source. In the fine-tuning phase, we fine-tune a pre-trained LM for the task of feature 1579 generation. This phase consists of two stages: a preliminary stage where we fine-tune the model offline on multiple datasets, and an online stage where we fine-tune the model on the training samples of the analyzed dataset. Finally, in the features generation phase, we add the newly-generated features to the original features of the tabular dataset. The goal of this phase is to link entities from our analyzed dataset D t to entries/entities in the external data source EX (Figure 1 step #1). The identification of relevant entities is a necessary first step, since the entities selected in this phase will be processed in the following phases. In this study we use Wikipedia as our external source of information, and Google Search as our linking and disambiguation tool. Obviously, other external sources of information (e.g., Reuters news or Yago (Hoffart et al., 2013)) will require a different linking strategy, but our approach can easily be adapted to support them. Our chosen entity linking process is straightforward: for each dataset entity in e i in D i we query Google Search, focusing on Wikipedia and taking into account the domain of the entity: < lookup > is a < domain > site:en.wikipedia.org where < lookup > is the entity e i,j mention and < domain > is the entities domain (entities column name). for example: USA is a country site:en.wikipedia.org . Each of our queries returns a list of Wikipedia pages which are most likely to represent the entity. FeSTE then extracts the Wikipedia page referenced in the first entry. This step also serves as a form of automatic disambiguation, because we pair e i,j with its most popular interpretation. At the end of this phase, each dataset D i entity e i,j has a linked Wikipedia entity e exi,j . FeSTE then extracts the abstracts of those entities using DBpedia. The goal of this phase is to adapt current state-of-the-art NLP architectures, (e.g., GPT, BERT, and their extensions) to the task of selecting the most relevant features from each of our linked external source entities e ex . As explained in Section 2.3, two common FT approaches are task-specific fine-tuning , which is performed on the target dataset, and preliminary fine-tuning , which is applied on other datasets. While the former is more common, recent studies (Sun et al., 2019; Gururangan et al., 2020) have shown that applying boththe latter and then the formeryields better results. The main difficulty in applying preliminary FT to tabular datasets stems from their diversity: tabular datasets differ greatly in their domains, number of classes, feature composition, etc. These differences make the training of a generic features engineering tool very difficult. To overcome this challenge, we propose a novel FT approach (Figure 1 step #2), which consists of two stages: first, we perform preliminary FT with dataset task reformulation . Then, we perform Target Dataset Fine-Tuning using only the target dataset's training set, i.e., task-specific FT . Preliminary FT with dataset task reformulation. The main challenge in learning from multiple tabular datasets, aside from their highly diverse content and characteristics, is their different number of classes. Such a setup makes using a single output layer with a fixed number of entries impossible. We overcome this challenge as follows: For each dataset D i let there be a set of free texts T i , each associated with an entity e i,j in D i . For each T i,j , we create a Cartesian product T i,j XT C i , where T C i consists of all the target classes of the dataset D i . Namely, we pair the text T i,j with all possible target class values. We can now treat the problem as one of Sentence-pairs classification . In this setting, we are presented with a set consisting of three elements { T ri , tc ri , l ri }, where T ri is the text (first sentence), tc ri is a possible target class value (second sentence) and l r i the label. l r i set to True if { T ri , tc ri }  { T i , tc i }. This setting, which is presented in full in Algorithm 2, creates a unified representation for all tabular datasets regardless of their original number of classes . Simply put, we reformulated the original task of each dataset into a NLP downstream task whose goal is to classify whether a given text T ri is related to a given class value tc ri . Once we have reformulated our problem, we can use it to perform a preliminary-FT of BERT . The input we provide consists of two sentences, a classification token and a separation token: [ CLS ] < T ri,j > [ SEP ] < tc ri,j > [ SEP ] 1580 University Original Features Class Alaska Pacific University x1, where T ri,j is the free text, tc ri,j is the assigned target class value, and [ CLS ] and [ SEP ] are BERT's special tokens. An example from the dataset AAUP, whose task is to predict whether a university is ranked as high or low, is presented below: [CLS] Alaska Pacific University (APU) is a private university in Anchorage, Alaska ... [SEP] Low [SEP] This phase of our FT process is similar to BERT's standard auxiliary training task, where the architecture is tasked with determining whether the class assigned to a sentence is correct (i.e., is it the one that appeared in the dataset?). For our fine-tuning, we use the same loss function that is used by the original BERT architecture's auxiliary task. Our data formulation enables us to fine-tune BERT simultaneously over a large set of datasets, thus creating a generic model that can then be effectively applied to additional datasets. It should be noted that a similar process of including the class value as part of the input was previously used in the domain of zero-shot Text Classification (Yin et al., 2019), to address the possibility of new classes appearing in mid-training. Target dataset fine-tuning. The goal of the preliminary FT was to adapt the pre-trained LM for the general task of feature generation for tabular datasets. Now we perform additional FT, designed to optimize the LM for the currently analyzed dataset. To this end, we now repeat the process described above for the target dataset . The process repeats all the steps of the preliminary FT, including the reformulation into a classification task. The deep architecture used for the two fine-tuning phases is presented in Figure 2. We partition the training set of the target dataset D t,train into two equal parts. One half is used for the target dataset FT, while the second is used for the features generation process , which we describe next. The goal of this phase is to produce the generated features that will augment the target dataset. The features generation process is as follows: for each sample (i.e., dataset tuple), we provide the pre-trained LM with an input consisting of: a) all the free text associated with the tuple's entity T rt,j , and; b) the possible target class values we generated tc r t,j . Simply put, we task the LM with predicting the likelihood of the text belonging to each of the target dataset's classes. The output of this process is a set of values, equal in length to the number of classes in the target dataset. Each of these values is added as a new feature to the target dataset. An example of this process is presented in Figure 1, step #3. The dataset presented in the example has only two class valueshigh and lowso FeSTE creates only two additional features that are added to the original features set. It should be noted that because of the varying number of target class values in our analyzed datasets, we use the Sigmoid function and evaluate each class individ-1581 ually (which is why our values for a given entity don't add up to 1, as shown in Figure 1). Once the new features F newt have been generated, we apply the Softmax function row-wise to receive a distribution over each target class value. The process described above is first applied to the target dataset's training set (i.e., the half that is retained for this purpose). We then train our classifier and apply it to the test set. Before each tuple in the test set is classified, we use the LM to generate the same set of features as the training set. In addition to the efficacy of our proposed approach, on which we elaborate in the following section, another advantage of FeSTE is the small number of features it generates. Unlike previously-proposed approaches such as (Harari and Katz, 2022), which generate thousands of features, the small number of features generated by FeSTE does not result in a large computational overhead. We compare FeSTE to the two leading fine-tuning methods: target dataset FT and MT-DNN FT : Target dataset FT. For this baseline, we fine-tune a BERT-based architecture (Figure 2, left side) on the target dataset and the texts without reformulation nor preliminary FT (Algorithem 1, lines 1,9-11). This approach is the commonly used FT strategy. MT-DNN FT. For this baseline, we first execute MT-DNN (Liu et al., 2019) as a preliminary FT step for the BERT-based architecture (Figure 2, left side). Then, we fine-tune BERT again using Target Dataset FT (Algorithm 1, lines 1,6,9-11). No reformulation is performed . It is important to note that all baselines, as well as FeSTE, are evaluated using the same experimental settings. The only difference between the approaches is their fine-tuning methods . For full details on our baselines, see Section 2. Datasets and evaluated classifiers. We evaluate our approach on 17 classification datasets with a large variance in their characteristics. The datasets were obtained from public repositories such as Kag-gle, UCI (Dua and Graff, 2017), OpenML (Van-schoren et al., 2013), and relevant studies (Ristoski et al., 2016). The datasets and their characteristics are presented in the Appendix. When applying the classifiers on each dataset (after its features have already been augmented), we used four-fold cross-validation, where we train on three folds and 1582 evaluate the fourth. We use the following five classifiers to evaluate the performance of FeSTE and the baselines: Ran-domForest, MLP, SVC, KNeighbors, and Gradient-Boosting. We used the implementations available in Scikit-learn, with the default hyper-parameter settings. The only preprocessing we perform is feature normalization. Since results are consistent for all algorithms, we present the average results . Individual results are presented in the Appendix. Architectures and parameter tuning. All evaluated models (FeSTE and baselines) use a pre-trained BERT architecture with 12 transformer blocks, 12 attention heads, and 110 million parameters (Hugging Face Tensorflow implementation). Additionally, the loss functions used by all fine-tuning approaches were either binary cross-entropy or multi-class cross-entropy, depending on the number of target classes. Finally, only the embedding [CLS] vector was passed to the output layer. When evaluating the performance of our approach on dataset D t = D i , we trained the BERT-based architecture on the remaining datasets, i.e., d i  D where i  = t . Since we evaluate FeSTE on 17 datasets, our architecture was fine-tuned on 16 datasets and tested on the 17th. This form of training was also performed for MT-DNN. FeSTE's preliminary and target-dataset fine-tuning settings were as follows: 20 training epochs with early stopping, mini-batches of 8 samples, a warm-up period of one epoch, no dropout, and the Adam optimizer. We used a learning rate of 1e-5 and 2e-5 for preliminary and target-dataset FTs, respectively. We also used a linear learning rate decay. For all experiments we used an Intel Xeon Gold 6140 2.3GHz Processor and 192GB RAM. We conducted two sets of experiments. The goal of the first is to evaluate the efficacy of our novel FT approach compared to the two leading baselines: target-dataset FT, and MT-DNN. The second set of experiments is designed to determine whether FeSTE is generic by evaluating its performance when using a different entity linking approach. Evaluating the efficacy of our FT method. In this experiment we focus on the efficacy of the features generated from the external data source (i.e., DBpedia unstructured text). We, therefore, train our classifiers only on the generated features Table 1: The AUC results obtained by our full proposed approach (Reformulated), and by versions of our approach that utilize the baseline FT methods for the fine-tuning phase. and ignore the original features of the dataset. This evaluation enables us to more accurately quantify the performance of each FT approach. The setup of this evaluation is as follows: the FeSTE algorithm is used in all experiments, but the FT phases of our approach is either the Reformulation method presented in Section 4.2 (Algorithm 1, lines 2-8) , or one of the two baselines. The results of this experiment are presented in Table 1. While it is clear that FeSTE performs well with all FT approaches, our proposed reformulation approach outperforms the baselines, achieving the highest results in 10 out of 17 datasets. In terms of AUC, Reformulated FT improves upon the baselines by 4.7%-6.8%. Using the paired t-test, we were able to determine that Reformulated FT outperforms both baselines with p < 0 . 001 . While Reformulated FT outperforms the baselines across all dataset sizes, it is noteworthy our approach achieves a larger relative improvement for smaller datasets. Improving the performance of such datasets is more difficult because of the limited amount of data available for the FT of the model. For example, the "Zoo" and "Country Codes" datasets contain only 35 and 75 records in their training set, respectively. Nonetheless, Reformulated FT outperforms the other baselines by 37% and 8.9% in terms of AUCwell above the overall average. These results demonstrate the effectiveness of our novel tuning approach, which leverages 1583 Table 2: An evaluation of FeSTE when it uses our Google-based entity linking approach, and when it implements the entity linking approach proposed by the recent FGSES framework. Evaluating the efficacy of our FT method with the original features. We now evaluate all approaches on the joint set of original and generated features. The only preprocessing we apply is feature normalization (no feature selection or engi-neering). We consider this setup the most realistic. The results of this experiment are presented in Table 3. Again, FeSTE performs well with all FT approaches and our reformulation approach outperforms the baselines, achieving the highest results in 9 out of 17 datasets. In terms of AUC, Reformulated FT improves upon the baselines by 1.4%, 2.3%, and 9.2%. Using the paired t-test, we were able to determine that Reformulated FT outperforms the three baselines with p < 0 . 001 . Evaluating FeSTE using additional entity linking approaches. In the previous experiment we demonstrated the efficacy of the features generated by FeSTE. Our goal now is to determine whether our approach is sufficiently generic to be applied with additional forms of entity linking. We, therefore, evaluate FeSTE's performance when our Google-based entity linking approach is replaced by the recently proposed FGSES approaches presented in (Harari and Katz, 2022). Table 2. We present the results for the two FeSTE versionsGoogle and FGSES-basedwhere the generated features are added to the original features set. To provide a meaningful point of reference, we also include the results obtained by using only the original features set for each dataset. It is clear that both versions of FeSTE outperform the original set of features. Our approach achieved better performances in 10 out of 17 datasets, with the original features achieving top performance in only 6 datasets. On average, FeSTE outperforms the results obtained by the original features by 9.2% and 5.2% for the Google-based and FGSES-based entity linking, respectively. Using the paired-t statistical tests, we were once again shown that FeSTE su-perior performance is statistically significant, with p < 0 . 001 , compare to the original set of features. Cases where the original features outperformed the augmented features set. The results in Section 5.3 clearly show that FeSTE significantly outperforms the baselines in a large majority of the evaluated datasets. In this section, however, we focus on datasets where our approach did not perform well compared to the original set of features. As shown in Table 2, there are six datasets in which the original features set outperformed FeSTE. We analyzed these datasets in an attempt to determine the causes of our approach's lower performance. Our conclusion is that FeSTE is in greater danger of underperforming in cases of spe-cialized datasets, i.e., datasets that are dedicated to highly specific topics that are not of general interest. In such use-cases, information extracted from a general data source like DBPedia might not be adequate. An example of such a use case is the WDI dataset, whose goal is to determine the income groups of various countries. Our analysis shows that the abstracts of the linked entities simply do not elaborate on the topic of income. Finally, we compare the performance achieved using only FeSTE's generated features (Table 1) to the performance of the original features (Table 2). Note that our generated features outperform the original features in 10 out of 17 datasetsan impressive accomplishment given that the original features are often highly informative. On average for all datasets, features generated by our approach outperform the original features by 2%. Moreover, in some datasets our approach significantly outperforms the original features by as much as 192%. Analyzing FeSTE's Generalization Capabilities. In all our previous experiments, FeSTE was finetuned on 16 datasets. We now analyze our approach's ability to generalize as a function of the number of its fine-tuning datasets. Figure 3 presents FeSTE's relative improvement compared to preliminary FT. The results show that even four FT datasets yields an improvement (1.8%) compared to this baseline, with the gap rapidly expanding as new datasets are added. This analysis highlights FeSTE's generic nature and its ability to leverage knowledge from multiple sources. In this analysis we compare FeSTE both to target dataset FT and to MT-DNN (see Section 2). Target dataset FT is clearly the most efficient of the three approaches, as it constitutes a part of the other approaches. While FeSTE and MT-DNN were implemented using identical architectures (with one minor difference, described below), their comparison requires us to consider two aspects of their respective implementations: (1) While FeSTE employs the same architecture for all datasets, MT-DNN must train a new output layer for each new task, as well as for datasets with Figure 3: FeSTE's relative performance to preliminary FT, as a function of the number of datasets on which our approach performs its fine-tuning. the same task but with a different number of classes. In our experiments, for example, we trained seven output layers for MT-DNN. In addition to the need to constantly re-train the model, MT-DNN incurs significant storage costs because of the need to maintain multiple architectures. (2) FeSTE incurs an additional computational cost due to its reformulation phase. The cost of reformulation consists of two parts: the first is the reformulation process itself, and the other is the additional FT as a results of the larger number of samples. The computational cost of both tasks is O ( | C || UniqueEntities | ) . Please note, however, that in tabular dataset both number of classes and the number of unique entities is relatively small. To summarize, MT-DNN will likely be more efficient for a small number of tasks/datasets, each consisting of a large number of training samples. FeSTE, on the other hand, will be more effective on a diverse set of datasets and tasks, possibly containing a relatively smaller number of samples. We present FeSTE, a framework for generating new features for tabular datasets from unstructured sources. Our approach uses a novel two-step fine-tuning process that enables it to effectively apply transformer based LM for the extraction of useful features even when the target dataset is limited in size. Our FT approach significantly outperforms the existing SOTA.
abstain abstain objective objective objective result abstain abstain abstain abstain abstain abstain abstain abstain other method method abstain method objective abstain abstain objective abstain method result result objective objective abstain other other other other other other other other other other other abstain other objective method other other other other other other other other other other method other other other other other other other other other other other other other other other other other objective other method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method abstain method method method abstain method abstain method abstain abstain method abstain abstain abstain abstain method method abstain abstain abstain method method method method method abstain abstain method method method abstain method method method abstain abstain abstain method method abstain abstain method method abstain method abstain method method abstain abstain abstain abstain method method abstain method method method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method method method method abstain abstain abstain abstain abstain method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain result result result result result abstain method method abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain objective objective result