[NeurIPS 2024] Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing
This is a discriminator for persona-driven role-playing, distilled from GPT-4. It evaluates the NLI relation between the character's response and a persona statement of the character.
The discriminator is a component of a global role-playing faithfulness optimization framework (as shown below), to use it please visit github.com/KomeijiForce/Active_Passive_Constraint_Koishiday_2024
from classifier import Classifier, get_relevance_discriminator, get_nli_discriminator
from score import score_apc, score_APC
relevance_discriminator = get_relevance_discriminator(character=None, statement_query_relevance_dataset=None, relevance_finetune_epoch=None, use_pretrained_discriminator=True)
nli_discriminator = get_nli_discriminator(character=None, statement_to_response_nli_v2_dataset=None, nli_finetune_epoch=None, use_pretrained_discriminator=True)
character = "Komeiji Koishi"
statements = ["Komeiji Koishi lives with her sister, Komeiji Satori.", "Komeiji Koishi lives in Chireiden."]
query = "Where do you live, Koishi?"
responses = ["I live in Chireiden with my sister, Satori!", "I live in Chireiden!", "I live in Hakurei Shrine!"]
print([score_APC(character, statements, query, response, relevance_discriminator, nli_discriminator).item() for response in responses])
# [1.6079180240631104, 0.9955980777740479, 0.03315635025501251]
- Downloads last month
- 114
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.