Paper Review, Deep Learning, Rlhf, Llm, NLP.
humans into a single reward model without considering these differences leads to a fundamentally misspecified problem. Current techniques often model differences among evaluators as noise instead of essential disagreement sources. Majority preferences might prevail, potentially disadvantaging