Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts
<|reference_start|>Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts: This study presents a multi-modal multi-granularity tokenizer specifically designed for analyzing ancient Chinese scripts, focusing on the Chu bamboo slip (CBS) script used during the Spring and Autumn and Warring States period (771-256 BCE) in Ancient China. Considering the complex hierarchical structure of ancient Chinese scripts, where a single character may be a combination of multiple sub-characters, our tokenizer first adopts character detection to locate character boundaries, and then conducts character recognition at both the character and sub-character levels. Moreover, to support the academic community, we have also assembled the first large-scale dataset of CBSs with over 100K annotated character image scans. On the part-of-speech tagging task built on our dataset, using our tokenizer gives a 5.5% relative improvement in F1-score compared to mainstream sub-word tokenizers. Our work not only aids in further investigations of the specific script but also has the potential to advance research on other forms of ancient Chinese scripts.<|reference_end|>
@article{chen2024multi-modal,
    title={Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts},
    author={Yingfa Chen, Chenlong Hu, Cong Feng, Chenyang Song, Shi Yu, Xu Han,
  Zhiyuan Liu, Maosong Sun},
    journal={arXiv preprint arXiv:2409.01011},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.01011},
    primaryClass={cs.CL cs.CV}
}