LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as
  Evolutionary Optimizers
Automated feature engineering plays a critical role in improving predictive
model performance for tabular learning tasks. Traditional automated feature
engineering methods are limited by their reliance on pre-defined
transformations within fixed, manually designed search spaces, often neglecting
domain knowledge. Recent advances using Large Language Models (LLMs) have
enabled the integration of domain knowledge into the feature engineering
process. However, existing LLM-based approaches use direct prompting or rely
solely on validation scores for feature selection, failing to leverage insights
from prior feature discovery experiments or establish meaningful reasoning
between feature generation and data-driven performance. To address these
challenges, we propose LLM-FE, a novel framework that combines evolutionary
search with the domain knowledge and reasoning capabilities of LLMs to
automatically discover effective features for tabular learning tasks. LLM-FE
formulates feature engineering as a program search problem, where LLMs propose
new feature transformation programs iteratively, and data-driven feedback
guides the search process. Our results demonstrate that LLM-FE consistently
outperforms state-of-the-art baselines, significantly enhancing the performance
of tabular prediction models across diverse classification and regression
benchmarks.
67dc4650129a710a8042aa99
Nikhil Abhyankar
extracted_pending
2025-03-20T16:46:09.424Z
6490a82b231a197da764ace8
/avatars/5d8b7cafa2658a01e706037a3994ebb9.svg
Nikhil Abhyankar
user
nikhilsa
67dc4650129a710a8042aa9a
Parshin Shojaee
extracted_confirmed
2025-03-20T17:22:12.321Z
6520621836008ecc88699622
/avatars/b08c00af00f1736a4f4938443e575b0e.svg
Parshin Shojaee
user
parshinsh
67dc4650129a710a8042aa9b
Chandan K. Reddy
LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as
  Evolutionary Optimizers
Automated feature engineering plays a critical role in improving predictive
model performance for tabular learning tasks. Traditional automated feature
engineering methods are limited by their reliance on pre-defined
transformations within fixed, manually designed search spaces, often neglecting
domain knowledge. Recent advances using Large Language Models (LLMs) have
enabled the integration of domain knowledge into the feature engineering
process. However, existing LLM-based approaches use direct prompting or rely
solely on validation scores for feature selection, failing to leverage insights
from prior feature discovery experiments or establish meaningful reasoning
between feature generation and data-driven performance. To address these
challenges, we propose LLM-FE, a novel framework that combines evolutionary
search with the domain knowledge and reasoning capabilities of LLMs to
automatically discover effective features for tabular learning tasks. LLM-FE
formulates feature engineering as a program search problem, where LLMs propose
new feature transformation programs iteratively, and data-driven feedback
guides the search process. Our results demonstrate that LLM-FE consistently
outperforms state-of-the-art baselines, significantly enhancing the performance
of tabular prediction models across diverse classification and regression
benchmarks.
LLM-FE integrates domain knowledge and evolutionary search to automate feature engineering, enhancing tabular prediction model performance.
LLM-FE evolutionary search large language models feature engineering feature transformation programs data-driven feedback