Multi-Domain Explainability of Preferences
Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and
reward models, are central to aligning and evaluating large language models
(LLMs). Yet, the underlying concepts that drive these preferences remain poorly
understood. In this work, we propose a fully automated method for generating
local and global concept-based explanations of preferences across multiple
domains. Our method utilizes an LLM to identify concepts that distinguish
between chosen and rejected responses, and to represent them with concept-based
vectors. To model the relationships between concepts and preferences, we
propose a white-box Hierarchical Multi-Domain Regression model that captures
both domain-general and domain-specific effects. To evaluate our method, we
curate a dataset spanning eight challenging and diverse domains and explain
twelve mechanisms. Our method achieves strong preference prediction
performance, outperforming baselines while also being explainable.
Additionally, we assess explanations in two application-driven settings. First,
guiding LLM outputs with concepts from LaaJ explanations yields responses that
those judges consistently prefer. Second, prompting LaaJs with concepts
explaining humans improves their preference predictions. Together, our work
establishes a new paradigm for explainability in the era of LLMs.
683947458987f50a5ec45a01
Nitay Calderon
683947458987f50a5ec45a02
Liat Ein-Dor
683947458987f50a5ec45a03
Roi Reichart
Multi-Domain Explainability of Preferences
Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and
reward models, are central to aligning and evaluating large language models
(LLMs). Yet, the underlying concepts that drive these preferences remain poorly
understood. In this work, we propose a fully automated method for generating
local and global concept-based explanations of preferences across multiple
domains. Our method utilizes an LLM to identify concepts that distinguish
between chosen and rejected responses, and to represent them with concept-based
vectors. To model the relationships between concepts and preferences, we
propose a white-box Hierarchical Multi-Domain Regression model that captures
both domain-general and domain-specific effects. To evaluate our method, we
curate a dataset spanning eight challenging and diverse domains and explain
twelve mechanisms. Our method achieves strong preference prediction
performance, outperforming baselines while also being explainable.
Additionally, we assess explanations in two application-driven settings. First,
guiding LLM outputs with concepts from LaaJ explanations yields responses that
those judges consistently prefer. Second, prompting LaaJs with concepts
explaining humans improves their preference predictions. Together, our work
establishes a new paradigm for explainability in the era of LLMs.
A new automated method using concept-based vectors and a Hierarchical Multi-Domain Regression model improves preference explanations and predictions for large language models.
LLM-as-a-Judge reward models large language models concept-based explanations domain-general effects domain-specific effects Hierarchical Multi-Domain Regression model preference prediction explainability