Answers: <p>You should do a comparison of commercial virtualization products on the same hardware resources, OR benchmarking commercial virtualization products individually, there are ways to do it, you can even script it.</p>
  
  <p>For example, VMmark is a free tool used to measure the performance and scalability of virtualization platforms VMmark</p>
  
  <p>Since your question is very generic, you should clarify real performance, overall performance. It might mean "real performance in virtualization".</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-11-06T11:32:46.537Z
Answers: 2018-11-06T12:16:38.817Z
Answers: 2018-11-06T12:16:38.817Z
Answers: hakkican
Answers: <p>IMHO an absolute benchmark can be rather useless/misleading, I'd always consider the specific context of the application intended to be executed on the platforms being benchmarked and its performance requirements and use only benchmarks relevant to that application context.</p>
  
  <p>A few examples:</p>
  
  <ul>
  <li><p>if the application doesn't make a lot of use of the CPU and/or its memory, comparing the number of cores and/or the amount of RAM or benchmarking CPU/RAM use would not be very relevant</p></li>
  <li><p>if the application accesses data on <a href="https://aws.amazon.com/s3" rel="nofollow noreferrer">AWS S3</a> using an AWS platform may provide better performance than using an on-premises bare metal or VMware platform even if the AWS platform's absolute I/O performance is below that of the on-premise platform</p></li>
  </ul>
  
  <p>Side note: by always starting with the application context in mind potentially more performant solutions other than bare metal/VM/Infrastructure as a Service (IaaS) can be selected, for example <a href="https://devops.stackexchange.com/tags/platform-as-a-service/info">Platform as a Service (PaaS)</a> or <a href="https://devops.stackexchange.com/tags/serverless/info">serverless/Function as a Service (FaaS)</a> solutions. It all depends on the application.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-10-06T18:50:09.110Z
Answers: And what if you have a mixture of quick mixed peaks on all ends,  cpu, i/o, and average high load? I consider bootstrapping in several small environments to experimentally find out how to scale at best, but as said .. The app is not finally there, so question after a benchmark to compare systems meaningfully. Sure it should deliver not just one number but an array for each scale dimension.
Answers: Ta Mu
Answers: CC BY-SA 4.0
Answers: 2018-10-06T19:14:51.873Z
Answers: I guess you'll have to give weights to the benchmarks of interest, probably based on estimates of how much each matters for the overall app's performance and make a compromise. Yep, running some actual experiments could help on the decision, especially to rule out options if they're obvious outliers.
Answers: Dan Cornilescu
Answers: CC BY-SA 4.0
Answers: 2018-10-06T16:34:49.313Z
Answers: 2018-10-06T16:34:49.313Z
Answers: Dan Cornilescu
Body: <p>Which performance index is most suitable to find out real performance of your VM limited by sharing and quotas?</p>
  
  <p>For example, if you have a physical server with 4 cores and 16 GB RAM, its performance limit might be higher than if you have same on AWS or in some VMWare product.</p>
  
  <p>How to compare to find out the gap? Is there an overall index or better to use some reference applications' performance as benchmark? (tough, though, if you still going to develop that application)</p>
  
Tags: <cloud><metrics><performance><distributed-systems>
Title: Identifying a benchmark to compare overall performance of VM virtualization vs hardware
UserDisplayName: Ta Mu
ThreadText: # Identifying a benchmark to compare overall performance of VM virtualization vs hardware
  
  (Asked by: username_0 on 2018-10-05)
  
  Which performance index is most suitable to find out real performance of your VM limited by sharing and quotas?
  
  For example, if you have a physical server with 4 cores and 16 GB RAM, its performance limit might be higher than if you have same on AWS or in some VMWare product.
  
  How to compare to find out the gap? Is there an overall index or better to use some reference applications' performance as benchmark? (tough, though, if you still going to develop that application)
  
  **Tags:** cloud, metrics, performance, distributed-systems
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-11-06. Score: 2
  
  You should do a comparison of commercial virtualization products on the same hardware resources, OR benchmarking commercial virtualization products individually, there are ways to do it, you can even script it.
  
  For example, VMmark is a free tool used to measure the performance and scalability of virtualization platforms VMmark
  
  Since your question is very generic, you should clarify real performance, overall performance. It might mean "real performance in virtualization".
  
  
  ---
  
  ### Answer by username_2 on 2018-10-06. Score: 1
  
  IMHO an absolute benchmark can be rather useless/misleading, I'd always consider the specific context of the application intended to be executed on the platforms being benchmarked and its performance requirements and use only benchmarks relevant to that application context.
  
  A few examples:
  
  * if the application doesn't make a lot of use of the CPU and/or its memory, comparing the number of cores and/or the amount of RAM or benchmarking CPU/RAM use would not be very relevant
  * if the application accesses data on [AWS S3](https://aws.amazon.com/s3) using an AWS platform may provide better performance than using an on-premises bare metal or VMware platform even if the AWS platform's absolute I/O performance is below that of the on-premise platform
  
  Side note: by always starting with the application context in mind potentially more performant solutions other than bare metal/VM/Infrastructure as a Service (IaaS) can be selected, for example [Platform as a Service (PaaS)](https://devops.stackexchange.com/tags/platform-as-a-service/info) or [serverless/Function as a Service (FaaS)](https://devops.stackexchange.com/tags/serverless/info) solutions. It all depends on the application.
  
  > **Comments:**
  > **username_0**: And what if you have a mixture of quick mixed peaks on all ends,  cpu, i/o, and average high load? I consider bootstrapping in several small environments to experimentally find out how to scale at best, but as said .. The app is not finally there, so question after a benchmark to compare systems meaningfully. Sure it should deliver not just one number but an array for each scale dimension.  
  > **username_2**: I guess you'll have to give weights to the benchmarks of interest, probably based on estimates of how much each matters for the overall app's performance and make a compromise. Yep, running some actual experiments could help on the decision, especially to rule out options if they're obvious outliers.  
  
  ---
  
  