Modified introductions

utils.py

@@ -28,26 +28,27 @@ COLUMN_NAMES = MODEL_INFO
 LEADERBOARD_INTRODUCTION = """# Chumor Leaderboard
 
 ## Introduction
-We introduce Chumor, an enhanced benchmark designed to evaluate language understanding models across broader and more challenging tasks. Building on the Massive Multitask Language Understanding (MMLU) dataset, MMLU-Pro integrates more challenging, reasoning-focused questions and increases the answer choices per question from four to ten, significantly raising the difficulty and reducing the chance of success through random guessing. MMLU-Pro comprises over 12,000 rigorously curated questions from academic exams and textbooks, spanning 14 diverse domains including Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology, and Others.  
+We construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets. Chumor is sourced from Ruo Zhi Ba (弱智吧), a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes.
 
-Note: For inclusion in our leaderboard, submissions must provide substantial evidence demonstrating that their system is a genuine language model. We maintain strict verification standards to ensure the integrity and comparability of the results.
 
 ## What's new about MMLU-Pro
 
 Compared to the original MMLU, there are three major differences:
 
-- The original MMLU dataset only contains 4 options, MMLU-Pro increases it to 10 options. The increase in options will make the evaluation more realistic and challenging. The random guessing will lead to a much lower score.
-- The original MMLU dataset contains mostly knowledge-driven questions without requiring much reasoning. Therefore, PPL results are normally better than CoT. In our dataset, we increase the problem difficulty and integrate more reasoning-focused problems. In MMLU-Pro, CoT can be 20% higher than PPL. 
-- By increasing the distractor numbers, we significantly reduce the probability of correct guess by chance to boost the benchmark’s robustness. Specifically, with 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro.
+Unlike existing datasets that focus on tasks such as humor detection, punchline identification, or humor generation, Chumor addresses the challenge of humor explanation. This involves not just identifying humor but understanding the reasoning behind it, a task that requires both linguistic and cultural knowledge. Specifically, Chumor tasks the LLMs with determining whether an explanation fully explains the joke. We source the explanations from GPT-4o and ERNIE-4-turbo, and have the entire dataset manually annotated by five native Chinese speakers.
 
-For detailed information about the dataset, visit our page on Hugging Face:  https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro. 
+For detailed information about the dataset, visit our page on Hugging Face:  https://huggingface.co/datasets/dnaihao/Chumor. 
 
-If you are interested in replicating these results or wish to evaluate your models using our dataset, access our evaluation scripts available on GitHub: https://github.com/TIGER-AI-Lab/MMLU-Pro.
+If you are interested in replicating these results or wish to evaluate your models using our dataset, access our evaluation scripts available on GitHub: https://github.com/dnaihao/Chumor-dataset.
 
-If you would like to learn more details about our dataset, please check out our paper: https://arxiv.org/abs/2406.01574.
+If you would like to learn more details about our dataset, please check out our paper: https://arxiv.org/abs/2406.12754.
 
 Below you can find the accuracies of different models tested on this dataset.
 
+### Acknowledgements
+
+We construct the leaderboard based on the templated by https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro.
+
 """
 
 TABLE_INTRODUCTION = """
@@ -74,23 +75,17 @@ CITATION_BUTTON_TEXT = r"""
 
 SUBMIT_INTRODUCTION = """# Submit on MMLU-Pro Leaderboard Introduction
 
-## ⚠ Please note that you need to submit the JSON file with the following format:
-
-```json
-[
-    {
-        "question_id": 123,
-        "question": "abc",
-        "options": ["abc", "xyz", ...], 
-        "answer": "ABC",
-        "answer_index": 1,
-        "category": "abc,
-        "pred": "B",
-        "model_outputs": ""
-    }, ...
-]
+## ⚠ Please note that you need to submit the CSV file with the following format:
+
+```csv
+labels
+good
+good
+bad
+...
 ```
-You can generate an output file in the above format using the evaluation script provided in our GitHub repository. For your convenience, the script and detailed instructions are available at GitHub: https://github.com/TIGER-AI-Lab/MMLU-Pro. After generating the file, please send us an email at [email protected], attaching the output file.
+
+You can generate an output file in the above format using the evaluation script provided in our GitHub repository. For your convenience, the script and detailed instructions are available at GitHub: https://github.com/dnaihao/Chumor-dataset. After generating the file, please send us an email at [email protected], attaching the output file.
 """