IMPARA-GED

This is the repository of the IMPARA-GED, a quality estimator for the grammatical error correction evaluation.

---

Abstract

From IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (Findings of ACL2025)

We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations.

---

Usage

STEP 1:

pip install gec-metric

STEP 2:

from gec_metrics import get_metric, get_meta_eval
from pprint import pprint
c = get_metric('impara')
metric = c(c.Config(
    model_qe='naist-nlp/IMPARA-GED',
    threshold=-999.0,  # Ignore similarity scores
    pooling='mean'  # Our model was trained with mean pooling
))
print(metric.config)
c = get_meta_eval('seeda')
meta = c(c.Config('base'))

print('=== System-level meta-evaluation ===')
system_results = meta.corr_system(metric, aggregation='trueskill')
print('SEEDA-S')
pprint(system_results.ts_sent)
print('SEEDA-E')
pprint(system_results.ts_edit)

print('=== Sentence-level meta-evaluation ===')
sentence_results = meta.corr_sentence(metric)
print('SEEDA-S')
pprint(sentence_results.sent)
print('SEEDA-E')
pprint(sentence_results.edit)

See details: gec-metric.

---

Citation

Bibkey (For anthology.bib):

sakai-etal-2025-impara

Bibtex:

@inproceedings{sakai-etal-2025-impara,
    title = "{IMPARA}-{GED}: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator",
    author = "Sakai, Yusuke  and
      Goto, Takumi  and
      Watanabe, Taro",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.1315/",
    doi = "10.18653/v1/2025.findings-acl.1315",
    pages = "25647--25654",
    ISBN = "979-8-89176-256-5",
    abstract = "We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations."
}

Contact

Yusuke Sakai (@yusuke1997)