ElicitationGPT: Text Elicitation Mechanisms via Language Models
<|reference_start|>ElicitationGPT: Text Elicitation Mechanisms via Language Models: Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information and the training of machine learning models. This paper develops mechanisms for scoring elicited text against ground truth text using domain-knowledge-free queries to a large language model (specifically ChatGPT) and empirically evaluates their alignment with human preferences. The empirical evaluation is conducted on peer reviews from a peer-grading dataset and in comparison to manual instructor scores for the peer reviews.<|reference_end|>
@article{wu2024elicitationgpt:,
    title={ElicitationGPT: Text Elicitation Mechanisms via Language Models},
    author={Yifan Wu, Jason Hartline},
    journal={arXiv preprint arXiv:2406.09363},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09363},
    primaryClass={cs.AI cs.GT cs.LG}
}