title: ChARM: Character-based Act-adaptive Reward Modeling for Advanced
    Role-Playing Language Agents
summary: Role-Playing Language Agents (RPLAs) aim to simulate characters for realistic
  and engaging human-computer interactions. However, traditional reward models
  often struggle with scalability and adapting to subjective conversational
  preferences. We propose ChARM, a Character-based Act-adaptive Reward Model,
  addressing these challenges through two innovations: (1) an act-adaptive margin
  that significantly enhances learning efficiency and generalizability, and (2) a
  self-evolution mechanism leveraging large-scale unlabeled data to improve
  training coverage. Additionally, we introduce RoleplayPref, the first
  large-scale preference dataset specifically for RPLAs, featuring 1,108
  characters, 13 subcategories, and 16,888 bilingual dialogues, alongside
  RoleplayEval, a dedicated evaluation benchmark. Experimental results show a 13%
  improvement over the conventional Bradley-Terry model in preference rankings.
  Furthermore, applying ChARM-generated rewards to preference learning techniques
  (e.g., direct preference optimization) achieves state-of-the-art results on
  CharacterEval and RoleplayEval. Code and dataset are available at
  https://github.com/calubkk/ChARM.
paper_authors._id: 683dd20fb1080514bd945374
paper_authors.name: Feiteng Fang
paper_authors._id: 683dd20fb1080514bd945375
paper_authors.name: Ting-En Lin
paper_authors._id: 683dd20fb1080514bd945376
paper_authors.name: Yuchuan Wu
paper_authors._id: 683dd20fb1080514bd945377
paper_authors.name: Xiong Liu
paper_authors._id: 683dd20fb1080514bd945378
paper_authors.name: Xiang Huang
paper_authors._id: 683dd20fb1080514bd945379
paper_authors.name: Dingwei Chen
paper_authors._id: 683dd20fb1080514bd94537a
paper_authors.name: Jing Ye
paper_authors._id: 683dd20fb1080514bd94537b
paper_authors.name: Haonan Zhang
paper_authors._id: 683dd20fb1080514bd94537c
paper_authors.name: Liang Zhu
paper_authors._id: 683dd20fb1080514bd94537d
paper_authors.name: Hamid Alinejad-Rokny
paper_authors._id: 683dd20fb1080514bd94537e
paper_authors.name: Min Yang
paper_authors._id: 683dd20fb1080514bd94537f
paper_authors.name: Fei Huang
paper_authors._id: 683dd20fb1080514bd945380
paper_authors.name: Yongbin Li
paper_title: ChARM: Character-based Act-adaptive Reward Modeling for Advanced
    Role-Playing Language Agents
paper_summary: Role-Playing Language Agents (RPLAs) aim to simulate characters for realistic
  and engaging human-computer interactions. However, traditional reward models
  often struggle with scalability and adapting to subjective conversational
  preferences. We propose ChARM, a Character-based Act-adaptive Reward Model,
  addressing these challenges through two innovations: (1) an act-adaptive margin
  that significantly enhances learning efficiency and generalizability, and (2) a
  self-evolution mechanism leveraging large-scale unlabeled data to improve
  training coverage. Additionally, we introduce RoleplayPref, the first
  large-scale preference dataset specifically for RPLAs, featuring 1,108
  characters, 13 subcategories, and 16,888 bilingual dialogues, alongside
  RoleplayEval, a dedicated evaluation benchmark. Experimental results show a 13%
  improvement over the conventional Bradley-Terry model in preference rankings.
  Furthermore, applying ChARM-generated rewards to preference learning techniques
  (e.g., direct preference optimization) achieves state-of-the-art results on
  CharacterEval and RoleplayEval. Code and dataset are available at
  https://github.com/calubkk/ChARM.
paper_ai_summary: ChARM, a character-focused adaptive reward model, improves preference learning for role-playing language agents by using an act-adaptive margin and self-evolution with unlabeled data, achieving superior results on dedicated benchmarks.
paper_ai_keywords: Role-Playing Language Agents ChARM act-adaptive margin self-evolution mechanism RoleplayPref RoleplayEval preference learning direct preference optimization CharacterEval