Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical
  Perspective
Large Language Models (LLMs) often generate responses with inherent biases,
undermining their reliability in real-world applications. Existing evaluation
methods often overlook biases in long-form responses and the intrinsic
variability of LLM outputs. To address these challenges, we propose
FiSCo(Fine-grained Semantic Computation), a novel statistical framework to
evaluate group-level fairness in LLMs by detecting subtle semantic differences
in long-form responses across demographic groups. Unlike prior work focusing on
sentiment or token-level comparisons, FiSCo goes beyond surface-level analysis
by operating at the claim level, leveraging entailment checks to assess the
consistency of meaning across responses. We decompose model outputs into
semantically distinct claims and apply statistical hypothesis testing to
compare inter- and intra-group similarities, enabling robust detection of
subtle biases. We formalize a new group counterfactual fairness definition and
validate FiSCo on both synthetic and human-annotated datasets spanning gender,
race, and age. Experiments show that FiSco more reliably identifies nuanced
biases while reducing the impact of stochastic LLM variability, outperforming
various evaluation metrics.
685b50b6d2ee4fac76521db9
Weijie Xu
claimed_verified
2025-06-25T08:08:56.415Z
63e3f57754f51ea342ce26be
/avatars/df9d52c376bed6868d341bb006bec212.svg
Weijie Xu
user
xwjzds
685b50b6d2ee4fac76521dba
Yiwen Wang
685b50b6d2ee4fac76521dbb
Chi Xue
685b50b6d2ee4fac76521dbc
Xiangkun Hu
685b50b6d2ee4fac76521dbd
Xi Fang
685b50b6d2ee4fac76521dbe
Guimin Dong
685b50b6d2ee4fac76521dbf
Chandan K. Reddy
Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical
  Perspective
Large Language Models (LLMs) often generate responses with inherent biases,
undermining their reliability in real-world applications. Existing evaluation
methods often overlook biases in long-form responses and the intrinsic
variability of LLM outputs. To address these challenges, we propose
FiSCo(Fine-grained Semantic Computation), a novel statistical framework to
evaluate group-level fairness in LLMs by detecting subtle semantic differences
in long-form responses across demographic groups. Unlike prior work focusing on
sentiment or token-level comparisons, FiSCo goes beyond surface-level analysis
by operating at the claim level, leveraging entailment checks to assess the
consistency of meaning across responses. We decompose model outputs into
semantically distinct claims and apply statistical hypothesis testing to
compare inter- and intra-group similarities, enabling robust detection of
subtle biases. We formalize a new group counterfactual fairness definition and
validate FiSCo on both synthetic and human-annotated datasets spanning gender,
race, and age. Experiments show that FiSco more reliably identifies nuanced
biases while reducing the impact of stochastic LLM variability, outperforming
various evaluation metrics.
FiSCo evaluates LLM fairness by detecting semantic differences in long-form responses across demographic groups, using entailment checks and statistical hypothesis testing to identify subtle biases.
Large Language Models FiSCo Fine-grained Semantic Computation group-level fairness semantic differences entailment checks statistical hypothesis testing group counterfactual fairness