  The Chinese character riddle is a unique form of cultural entertainment
specific to the Chinese language. It typically comprises two parts: the riddle
description and the solution. The solution to the riddle is a single character,
while the riddle description primarily describes the glyph of the solution,
occasionally supplemented with its explanation and pronunciation. Solving
Chinese character riddles is a challenging task that demands understanding of
character glyph, general knowledge, and a grasp of figurative language. In this
paper, we construct a \textbf{C}hinese \textbf{C}haracter riddle dataset named
CC-Riddle, which covers the majority of common simplified Chinese characters.
The construction process is a combination of web crawling, language model
generation and manual filtering. In generation stage, we input the Chinese
phonetic alphabet, glyph and meaning of the solution character into the
generation model, which then produces multiple riddle descriptions. The
generated riddles are then manually filtered and the final CC-Riddle dataset is
composed of both human-written riddles and these filtered, generated riddles.
In order to assess the performance of language models on the task of solving
character riddles, we use retrieval-based, generative and multiple-choice QA
strategies to test three language models: BERT, ChatGPT and ChatGLM. The test
results reveal that current language models still struggle to solve Chinese
character riddles. CC-Riddle is publicly available at
\url{https://github.com/pku0xff/CC-Riddle}.

Computation and Language