title: Multi-Domain Explainability of Preferences
summary: Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and
  reward models, are central to aligning and evaluating large language models
  (LLMs). Yet, the underlying concepts that drive these preferences remain poorly
  understood. In this work, we propose a fully automated method for generating
  local and global concept-based explanations of preferences across multiple
  domains. Our method utilizes an LLM to identify concepts that distinguish
  between chosen and rejected responses, and to represent them with concept-based
  vectors. To model the relationships between concepts and preferences, we
  propose a white-box Hierarchical Multi-Domain Regression model that captures
  both domain-general and domain-specific effects. To evaluate our method, we
  curate a dataset spanning eight challenging and diverse domains and explain
  twelve mechanisms. Our method achieves strong preference prediction
  performance, outperforming baselines while also being explainable.
  Additionally, we assess explanations in two application-driven settings. First,
  guiding LLM outputs with concepts from LaaJ explanations yields responses that
  those judges consistently prefer. Second, prompting LaaJs with concepts
  explaining humans improves their preference predictions. Together, our work
  establishes a new paradigm for explainability in the era of LLMs.
paper_authors._id: 683947458987f50a5ec45a01
paper_authors.name: Nitay Calderon
paper_authors._id: 683947458987f50a5ec45a02
paper_authors.name: Liat Ein-Dor
paper_authors._id: 683947458987f50a5ec45a03
paper_authors.name: Roi Reichart
paper_title: Multi-Domain Explainability of Preferences
paper_summary: Preference mechanisms, such as human preference, LLM-as-a-Judge (LaaJ), and
  reward models, are central to aligning and evaluating large language models
  (LLMs). Yet, the underlying concepts that drive these preferences remain poorly
  understood. In this work, we propose a fully automated method for generating
  local and global concept-based explanations of preferences across multiple
  domains. Our method utilizes an LLM to identify concepts that distinguish
  between chosen and rejected responses, and to represent them with concept-based
  vectors. To model the relationships between concepts and preferences, we
  propose a white-box Hierarchical Multi-Domain Regression model that captures
  both domain-general and domain-specific effects. To evaluate our method, we
  curate a dataset spanning eight challenging and diverse domains and explain
  twelve mechanisms. Our method achieves strong preference prediction
  performance, outperforming baselines while also being explainable.
  Additionally, we assess explanations in two application-driven settings. First,
  guiding LLM outputs with concepts from LaaJ explanations yields responses that
  those judges consistently prefer. Second, prompting LaaJs with concepts
  explaining humans improves their preference predictions. Together, our work
  establishes a new paradigm for explainability in the era of LLMs.
paper_ai_summary: A new automated method using concept-based vectors and a Hierarchical Multi-Domain Regression model improves preference explanations and predictions for large language models.
paper_ai_keywords: LLM-as-a-Judge reward models large language models concept-based explanations domain-general effects domain-specific effects Hierarchical Multi-Domain Regression model preference prediction explainability