license: apache-2.0
language:
- en
tags:
- metric
- scoring
- implicit_langauge
- implicitness
- linguistic
This repo contains a trained metric, ImpScore, which is trained on 112,580 sentence pairs using contrastive learning. It calculates an implicitness score ranging from [0, 2] for an English input sentence. A higher score indicates greater implicitness. Additionally, it can calculate the pragmatic distance between two input sentences, with the distance value ranging from [0, ∞). A higher distance means the two sentences differ more in their intended meaning.
The training code for this metric is available on Github: https://github.com/audreycs/ImpScore
Download ImpScore:
Since the model is fully customized, you need to download the model file first to use it.
Method 1: Dynamic loading
from huggingface_hub import hf_hub_download
repo_id = "audreyeleven/ImpScore"
# Download the model python file
model_path = hf_hub_download(repo_id=repo_id, filename="impscore.py")
# Load the model dynamically
import importlib.util
spec = importlib.util.spec_from_file_location("ModelClass", model_path)
model_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(model_module)
model = model_module.ImpModel.from_pretrained(repo_id)
device = "cuda" # or "cpu"
model.device = device
model.to(device)
model.eval()
Method 2: Local loading
You can also download the model python file into your local and import it.
from impscore import ImpModel
model = ImpModel.from_pretrained(repo_id)
device = "cuda" # or "cpu"
model.device = device
model.to(device)
model.eval()
Use This Metric
Calculating implicitness score for single sentence
The metric has the .infer(sentence) function which takes a single sentence as the input, and returns
- its implicitness score
- implicit embedding
- pragmatic embedding
# test inference
imp_score, imp_embedding, prag_embedding = model.infer("I have to leave now. Talk to you later.")
print(imp_score, imp_embedding, prag_embedding)
imp_score, imp_embedding, prag_embedding = model.infer("I can't believe we've talked for so long.")
print(imp_score, imp_embedding, prag_embedding)
The outputs:
tensor(0.6709, device='cuda:0', grad_fn=<RsubBackward1>) tensor([ 0.0458, -0.0149, -0.0182, -0.0905, 0.0541, -0.0133, ...])
tensor(1.0984, device='cuda:0', grad_fn=<RsubBackward1>) tensor([-0.0086, -0.1357, -0.0067, -0.0513, -0.0225, 0.0664, ...])
This means the second sentence "I can't believe we've talked for so long." is more implicit.
Calculating implicitness score and pragmatic distance for sentence pairs
The .infer_pairs(sent_batch1, sent_batch2) function takes pairs of sentences as input, and calculates
- their individual implicitness score
- their pragmatic distance
sent_batch1 is the list of the first sentence in each pair, and sent_batch2 is the list of the second sentence in each pair.
sentence_batch = [
["I have to leave now. Talk to you later.", "I can't believe we've talked for so long."],
["You must find a new place and move out by the end of this month.", "Maybe exploring other housing options could benefit us both?"]
]
s1 = sentence_batch[:][0]
s2 = sentence_batch[:][1]
imp_score1, imp_score2, prag_distance = model.infer_pairs(s1, s2)
print(imp_score1, imp_score2, prag_distance)
The output is:
tensor([0.6709, 0.9273]) tensor([1.0984, 1.3642]) tensor([0.6660, 0.7115])
Which means the implicit score for
- "I have to leave now. Talk to you later." is 0.6709
- "I can't believe we've talked for so long." is 1.0984
- and the pragmatic distance between "I have to leave now. Talk to you later." and "I can't believe we've talked for so long." is 0.6660