MLModelScope: A Distributed Platform for ML Model Evaluation and Benchmarking at Scale
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major *pain point* for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope*s capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.
This paper studies the question of model evaluation and reproducibility in machine learning research, specifically deep learning research, and designs and tests an extensive system for evaluating and comparing models. Users specify their evaluation parameters through a text file, and this is used by the runtime, which can be interfaced through a UI or the command line, in order to carry out the evaluation. Several experiments shed interesting light on various aspects of model, framework, and hardware performance. Hypothetically, if I were to design a new model and wish to evaluate its performance relative to existing SotA models, I would potentially use this system to run all of the models, including my own. That would mean that I need to *upload* or otherwise integrate my model into this system, and it was unclear from my reading of the paper how easy such a process would be. Similarly, I would wish to maintain similar training and evaluation conditions for my model, e.g., the same pre and post-processing, and that would involve *extracting* those steps from the system for use during training. I would also like to understand whether or not this is feasible and easy given the system*s design. In section 3.1, the authors write *The hardware details are not present in the manifest, but are user-provided options when performing the evaluation.* An example of how this operates would be useful in the paper. As far as experiments go, I*m not sure what the main takeaway is from section 4.1. To me, the takeaway that pre-processing is important and existing models are sensitive to pre-processing is not a new finding. The results from Table 1 could certainly be obtained without the use of the proposed system, and though there would be some scaffolding involved, I don*t think that the coding would not be particularly difficult. Is the takeaway that the proposed system makes it easier or faster to evaluate the effects of different types of pre-processing? Wouldn*t this be most interesting at training time? I find the experiments in sections 4.2 and 4.3 interesting. In section 4.2, I*m not sure if figure 9 includes enough information or description to conclude that *GPU instances in general are more cost-efficient than CPU instances for batched inference*, and some more detail here would be useful. Generally, I believe that the work is well-motivated and timely, the authors seem to have done a good job in citing related work (though admittedly I don*t know much about this area), and the results are supportive of the claims of the system*s usefulness.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
MLModelScope: A Distributed Platform for ML Model Evaluation and Benchmarking at Scale
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major *pain point* for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope*s capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies the question of model evaluation and reproducibility in machine learning research, specifically deep learning research, and designs and tests an extensive system for evaluating and comparing models. Users specify their evaluation parameters through a text file, and this is used by the runtime, which can be interfaced through a UI or the command line, in order to carry out the evaluation. Several experiments shed interesting light on various aspects of model, framework, and hardware performance. Hypothetically, if I were to design a new model and wish to evaluate its performance relative to existing SotA models, I would potentially use this system to run all of the models, including my own. That would mean that I need to *upload* or otherwise integrate my model into this system, and it was unclear from my reading of the paper how easy such a process would be. Similarly, I would wish to maintain similar training and evaluation conditions for my model, e.g., the same pre and post-processing, and that would involve *extracting* those steps from the system for use during training. I would also like to understand whether or not this is feasible and easy given the system*s design. In section 3.1, the authors write *The hardware details are not present in the manifest, but are user-provided options when performing the evaluation.* An example of how this operates would be useful in the paper. As far as experiments go, I*m not sure what the main takeaway is from section 4.1. To me, the takeaway that pre-processing is important and existing models are sensitive to pre-processing is not a new finding. The results from Table 1 could certainly be obtained without the use of the proposed system, and though there would be some scaffolding involved, I don*t think that the coding would not be particularly difficult. Is the takeaway that the proposed system makes it easier or faster to evaluate the effects of different types of pre-processing? Wouldn*t this be most interesting at training time? I find the experiments in sections 4.2 and 4.3 interesting. In section 4.2, I*m not sure if figure 9 includes enough information or description to conclude that *GPU instances in general are more cost-efficient than CPU instances for batched inference*, and some more detail here would be useful. Generally, I believe that the work is well-motivated and timely, the authors seem to have done a good job in citing related work (though admittedly I don*t know much about this area), and the results are supportive of the claims of the system*s usefulness.
assistant
This paper studies the question of model evaluation and reproducibility in machine learning research, specifically deep learning research, and designs and tests an extensive system for evaluating and comparing models. Users specify their evaluation parameters through a text file, and this is used by the runtime, which can be interfaced through a UI or the command line, in order to carry out the evaluation. Several experiments shed interesting light on various aspects of model, framework, and hardware performance. Hypothetically, if I were to design a new model and wish to evaluate its performance relative to existing SotA models, I would potentially use this system to run all of the models, including my own. That would mean that I need to *upload* or otherwise integrate my model into this system, and it was unclear from my reading of the paper how easy such a process would be. Similarly, I would wish to maintain similar training and evaluation conditions for my model, e.g., the same pre and post-processing, and that would involve *extracting* those steps from the system for use during training. I would also like to understand whether or not this is feasible and easy given the system*s design. In section 3.1, the authors write *The hardware details are not present in the manifest, but are user-provided options when performing the evaluation. * An example of how this operates would be useful in the paper. As far as experiments go, I*m not sure what the main takeaway is from section 4.1. To me, the takeaway that pre-processing is important and existing models are sensitive to pre-processing is not a new finding. The results from Table 1 could certainly be obtained without the use of the proposed system, and though there would be some scaffolding involved, I don*t think that the coding would not be particularly difficult. Is the takeaway that the proposed system makes it easier or faster to evaluate the effects of different types of pre-processing? Wouldn*t this be most interesting at training time? I find the experiments in sections 4.2 and 4.3 interesting. In section 4.2, I*m not sure if figure 9 includes enough information or description to conclude that *GPU instances in general are more cost-efficient than CPU instances for batched inference*, and some more detail here would be useful. Generally, I believe that the work is well-motivated and timely, the authors seem to have done a good job in citing related work (though admittedly I don*t know much about this area), and the results are supportive of the claims of the system*s usefulness.