Towards Using Multiple Iterated, Reproduced, and Replicated Experiments with Robots (MIRRER) for Evaluation and Benchmarking
<|reference_start|>Towards Using Multiple Iterated, Reproduced, and Replicated Experiments with Robots (MIRRER) for Evaluation and Benchmarking: The robotics research field lacks formalized definitions and frameworks for evaluating advanced capabilities including generalizability (the ability for robots to perform tasks under varied contexts) and reproducibility (the performance of a reproduced robot capability in different labs under the same experimental conditions). This paper presents an initial conceptual framework, MIRRER, that unites the concepts of performance evaluation, benchmarking, and reproduced/replicated experimentation in order to facilitate comparable robotics research. Several open issues with the application of the framework are also presented.<|reference_end|>
@article{norton2024towards,
    title={Towards Using Multiple Iterated, Reproduced, and Replicated Experiments
  with Robots (MIRRER) for Evaluation and Benchmarking},
    author={Adam Norton and Brian Flynn (New England Robotics Validation and
  Experimentation (NERVE) Center, University of Massachusetts Lowell)},
    journal={arXiv preprint arXiv:2408.04736},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04736},
    primaryClass={cs.RO}
}