  The current boom of learned query optimizers (LQO) can be explained not only
by the general continuous improvement of deep learning (DL) methods but also by
the straightforward formulation of a query optimization problem (QOP) as a
machine learning (ML) one. The idea is often to replace dynamic programming
approaches, widespread for solving QOP, with more powerful methods such as
reinforcement learning. However, such a rapid "game change" in the field of QOP
could not pass without consequences - other parts of the ML pipeline, except
for predictive model development, have large improvement potential. For
instance, different LQOs introduce their own restrictions on training data
generation from queries, use an arbitrary train/validation approach, and
evaluate on a voluntary split of benchmark queries.
  In this paper, we attempt to standardize the ML pipeline for evaluating LQOs
by introducing a new end-to-end benchmarking framework. Additionally, we guide
the reader through each data science stage in the ML pipeline and provide novel
insights from the machine learning perspective, considering the specifics of
QOP. Finally, we perform a rigorous evaluation of existing LQOs, showing that
PostgreSQL outperforms these LQOs in almost all experiments depending on the
train/test splits.

Databases