Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing
<|reference_start|>Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing: We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).<|reference_end|>
@article{shi2024structured,
    title={Structured Tree Alignment for Evaluation of (Speech) Constituency
  Parsing},
    author={Freda Shi, Kevin Gimpel, Karen Livescu},
    journal={arXiv preprint arXiv:2402.13433},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.13433},
    primaryClass={cs.CL cs.DS}
}