title: TabReD: A Benchmark of Tabular Machine Learning in-the-Wild
summary: Benchmarks that closely reflect downstream application scenarios are
  essential for the streamlined adoption of new research in tabular machine
  learning (ML). In this work, we examine existing tabular benchmarks and find
  two common characteristics of industry-grade tabular data that are
  underrepresented in the datasets available to the academic community. First,
  tabular data often changes over time in real-world deployment scenarios. This
  impacts model performance and requires time-based train and test splits for
  correct model evaluation. Yet, existing academic tabular datasets often lack
  timestamp metadata to enable such evaluation. Second, a considerable portion of
  datasets in production settings stem from extensive data acquisition and
  feature engineering pipelines. For each specific dataset, this can have a
  different impact on the absolute and relative number of predictive,
  uninformative, and correlated features, which in turn can affect model
  selection. To fill the aforementioned gaps in academic benchmarks, we introduce
  TabReD -- a collection of eight industry-grade tabular datasets covering a wide
  range of domains from finance to food delivery services. We assess a large
  number of tabular ML models in the feature-rich, temporally-evolving data
  setting facilitated by TabReD. We demonstrate that evaluation on time-based
  data splits leads to different methods ranking, compared to evaluation on
  random splits more common in academic benchmarks. Furthermore, on the TabReD
  datasets, MLP-like architectures and GBDT show the best results, while more
  sophisticated DL models are yet to prove their effectiveness.
paper_authors._id: 668543ec49af4905c09ba830
paper_authors.name: Ivan Rubachev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T11:59:27.502Z
paper_authors._id: 630bd0e9910e17bbfeadf22c
paper_authors.avatarUrl: /avatars/ff041e7a8cbca8e7130c3102a5d31c0c.svg
paper_authors.fullname: Ivan Rubachev
paper_authors.type: user
paper_authors.user: puhsu
paper_authors._id: 668543ec49af4905c09ba831
paper_authors.name: Nikolay Kartashev
paper_authors._id: 668543ec49af4905c09ba832
paper_authors.name: Yury Gorishniy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-04T12:01:02.404Z
paper_authors._id: 6655ae06ec19a307d34f8764
paper_authors.avatarUrl: /avatars/fbab83d3f7667e1c96954824976678f4.svg
paper_authors.fullname: Yury Gorishniy
paper_authors.type: user
paper_authors.user: Yura52
paper_authors._id: 668543ec49af4905c09ba833
paper_authors.name: Artem Babenko
paper_title: TabReD: A Benchmark of Tabular Machine Learning in-the-Wild
paper_summary: Benchmarks that closely reflect downstream application scenarios are
  essential for the streamlined adoption of new research in tabular machine
  learning (ML). In this work, we examine existing tabular benchmarks and find
  two common characteristics of industry-grade tabular data that are
  underrepresented in the datasets available to the academic community. First,
  tabular data often changes over time in real-world deployment scenarios. This
  impacts model performance and requires time-based train and test splits for
  correct model evaluation. Yet, existing academic tabular datasets often lack
  timestamp metadata to enable such evaluation. Second, a considerable portion of
  datasets in production settings stem from extensive data acquisition and
  feature engineering pipelines. For each specific dataset, this can have a
  different impact on the absolute and relative number of predictive,
  uninformative, and correlated features, which in turn can affect model
  selection. To fill the aforementioned gaps in academic benchmarks, we introduce
  TabReD -- a collection of eight industry-grade tabular datasets covering a wide
  range of domains from finance to food delivery services. We assess a large
  number of tabular ML models in the feature-rich, temporally-evolving data
  setting facilitated by TabReD. We demonstrate that evaluation on time-based
  data splits leads to different methods ranking, compared to evaluation on
  random splits more common in academic benchmarks. Furthermore, on the TabReD
  datasets, MLP-like architectures and GBDT show the best results, while more
  sophisticated DL models are yet to prove their effectiveness.
paper_ai_keywords: tabular benchmarks industry-grade tabular data time-based train and test splits timestamp metadata data acquisition feature engineering pipelines predictive features uninformative features correlated features model selection TabReD feature-width temporally-evolving data MLP-like architectures GBDT DL models