  Large language models (LLMs) have shown impressive capabilities across
various natural language tasks. However, evaluating their alignment with human
preferences remains a challenge. To this end, we propose a comprehensive human
evaluation framework to assess LLMs' proficiency in following instructions on
diverse real-world tasks. We construct a hierarchical task tree encompassing 7
major areas covering over 200 categories and over 800 tasks, which covers
diverse capabilities such as question answering, reasoning, multiturn dialogue,
and text generation, to evaluate LLMs in a comprehensive and in-depth manner.
We also design detailed evaluation standards and processes to facilitate
consistent, unbiased judgments from human evaluators. A test set of over 3,000
instances is released, spanning different difficulty levels and knowledge
domains. Our work provides a standardized methodology to evaluate human
alignment in LLMs for both English and Chinese. We also analyze the feasibility
of automating parts of evaluation with a strong LLM (GPT-4). Our framework
supports a thorough assessment of LLMs as they are integrated into real-world
applications. We have made publicly available the task tree, TencentLLMEval
dataset, and evaluation methodology which have been demonstrated as effective
in assessing the performance of Tencent Hunyuan LLMs. By doing so, we aim to
facilitate the benchmarking of advances in the development of safe and
human-aligned LLMs.

Computation and Language;Artificial Intelligence