N-Version Assessment and Enhancement of Generative AI
<|reference_start|>N-Version Assessment and Enhancement of Generative AI: Generative AI (GAI) holds great potential to improve software engineering productivity, but its untrustworthy outputs, particularly in code synthesis, pose significant challenges. The need for extensive verification and validation (V&V) of GAI-generated artifacts may undermine the potential productivity gains. This paper proposes a way of mitigating these risks by exploiting GAI's ability to generate multiple versions of code and tests to facilitate comparative analysis across versions. Rather than relying on the quality of a single test or code module, this "differential GAI" (D-GAI) approach promotes more reliable quality evaluation through version diversity. We introduce the Large-Scale Software Observatorium (LASSO), a platform that supports D-GAI by executing and analyzing large sets of code versions and tests. We discuss how LASSO enables rigorous evaluation of GAI-generated artifacts and propose its application in both software development and GAI research.<|reference_end|>
@article{kessel2024n-version,
    title={N-Version Assessment and Enhancement of Generative AI},
    author={Marcus Kessel, Colin Atkinson},
    journal={IEEE Software September 2024},
    year={2024},
    doi={10.1109/MS.2024.3469388},
    archivePrefix={arXiv},
    eprint={2409.14071},
    primaryClass={cs.SE cs.AI}
}