TabReD: A Benchmark of Tabular Machine Learning in-the-Wild
Benchmarks that closely reflect downstream application scenarios are
essential for the streamlined adoption of new research in tabular machine
learning (ML). In this work, we examine existing tabular benchmarks and find
two common characteristics of industry-grade tabular data that are
underrepresented in the datasets available to the academic community. First,
tabular data often changes over time in real-world deployment scenarios. This
impacts model performance and requires time-based train and test splits for
correct model evaluation. Yet, existing academic tabular datasets often lack
timestamp metadata to enable such evaluation. Second, a considerable portion of
datasets in production settings stem from extensive data acquisition and
feature engineering pipelines. For each specific dataset, this can have a
different impact on the absolute and relative number of predictive,
uninformative, and correlated features, which in turn can affect model
selection. To fill the aforementioned gaps in academic benchmarks, we introduce
TabReD -- a collection of eight industry-grade tabular datasets covering a wide
range of domains from finance to food delivery services. We assess a large
number of tabular ML models in the feature-rich, temporally-evolving data
setting facilitated by TabReD. We demonstrate that evaluation on time-based
data splits leads to different methods ranking, compared to evaluation on
random splits more common in academic benchmarks. Furthermore, on the TabReD
datasets, MLP-like architectures and GBDT show the best results, while more
sophisticated DL models are yet to prove their effectiveness.
668543ec49af4905c09ba830
Ivan Rubachev
admin_assigned
2024-07-04T11:59:27.502Z
630bd0e9910e17bbfeadf22c
/avatars/ff041e7a8cbca8e7130c3102a5d31c0c.svg
Ivan Rubachev
user
puhsu
668543ec49af4905c09ba831
Nikolay Kartashev
668543ec49af4905c09ba832
Yury Gorishniy
admin_assigned
2024-07-04T12:01:02.404Z
6655ae06ec19a307d34f8764
/avatars/fbab83d3f7667e1c96954824976678f4.svg
Yury Gorishniy
user
Yura52
668543ec49af4905c09ba833
Artem Babenko
TabReD: A Benchmark of Tabular Machine Learning in-the-Wild
Benchmarks that closely reflect downstream application scenarios are
essential for the streamlined adoption of new research in tabular machine
learning (ML). In this work, we examine existing tabular benchmarks and find
two common characteristics of industry-grade tabular data that are
underrepresented in the datasets available to the academic community. First,
tabular data often changes over time in real-world deployment scenarios. This
impacts model performance and requires time-based train and test splits for
correct model evaluation. Yet, existing academic tabular datasets often lack
timestamp metadata to enable such evaluation. Second, a considerable portion of
datasets in production settings stem from extensive data acquisition and
feature engineering pipelines. For each specific dataset, this can have a
different impact on the absolute and relative number of predictive,
uninformative, and correlated features, which in turn can affect model
selection. To fill the aforementioned gaps in academic benchmarks, we introduce
TabReD -- a collection of eight industry-grade tabular datasets covering a wide
range of domains from finance to food delivery services. We assess a large
number of tabular ML models in the feature-rich, temporally-evolving data
setting facilitated by TabReD. We demonstrate that evaluation on time-based
data splits leads to different methods ranking, compared to evaluation on
random splits more common in academic benchmarks. Furthermore, on the TabReD
datasets, MLP-like architectures and GBDT show the best results, while more
sophisticated DL models are yet to prove their effectiveness.
TabReD is a new collection of industry-grade tabular datasets that address temporal dynamics and feature engineering, revealing that simpler models outperform more complex deep learning architectures in these settings.
TabReD MLP-like architectures GBDT deep learning architectures time-based train and test splits temporally-evolving data industry-grade tabular data feature engineering predictive uninformative correlated features