I&#x27;ve done benchmarks across mobile devices as my bread &amp; butter job for several years. All of the benchmarks in this review mean almost nothing. Even the testing methodology is fundamentally flawed. I&#x27;ll explain.<p>Javascript benchmarks are heavily dependent on the browser&#x27;s javascript engine(big surprise). With Apple&#x27;s restriction on the ability to use a custom Javascript engine on iOS - there really is no way to compare say for example: Chrome on iOS vs Chrome on Android. They are running very different Javascript engines. The hardware is not the only variable that is changing.<p>Running Octane&#x2F;Kraken on Safari on the iPad vs Chrome on a Surface Pro - again why even bother running this test?<p>Also, ever wonder why the companies making the browser have their own browser benchmark suites? Google with Octane and Mozilla with Kraken, and Microsoft with their Testdrive? Guess who is the winner in each of the benchmark suites.<p>It is not that these companies consciously cheat by building the browser and the test to make it look good. The teams building the benchmarks and the teams building the browser are not in collusion. It just happens that when you use a single benchmark as the only metric to optimize for, you will eventually build a product that is optimized for that metric. No cheating required, it just looks that way.<p>Geekbench - I&#x27;ve seen refered to by my coworkers as &quot;Jokebench&quot;. Not that the benchmark itself is bad in some way - it is very useful in comparing one Apple device generation to another - or one mac to another mac. But cross-platform, cross-ISA, cross-OS? Joke. The authors of Geekbench know this fact, and yet they market GeekBench as a &quot;cross-platform&quot; processor benchmark. And your average tech product reviewer - who has never heard of the terms &#x27;LINPACK&#x27; or &#x27;SPECint&#x27; will have you believe it is a great benchmark - and loves basing their product recommendations on these scores rather than the subjective review of the experience of devices that they&#x27;re paid to review.<p>Additionally, all mobile devices throttle quite heavily under sustained load. 2.26GHz processor? Yeah 10 minutes in with 100% CPU load, I will bet real money that frequency drops below 1GHz. This is not a jab towards the A9 or Apple specifically. Qualcomm&#x27;s chips do the exact same thing. Intel&#x27;s Core does the same thing. Infact, they&#x27;ve put considerable marketing money behind this feature - calling it &quot;TurboBoost&quot;.<p>Anyone in the industry knows you absolutely have to measure performance in the thermal steady state - fancy way of saying when the temperatures across the device stay the same throughout the test. Try this right now. Download Geekbench or Antutu on your phone. Run it once. Note the score. Run the same test immediately again. What? The score is lower the second time? Did your hardware just get worse all of a sudden? No. The device is warmed up. So it has to throttle its frequency and power envelope to keep itself from burning you. Now the problem is that the difference in scores between the cold state and the warm state is quite dramatic. Which one do you use when you want to compare it to the iPad? Was the measurement you just did a &#x27;cold&#x27; one or a &#x27;hot&#x27; one?<p>Our team has entire rigs built out specifically for keeping devices cool during benchmarks. It&#x27;s the only reliable way to ensure thermal throttling doesn&#x27;t fog up your benchmark results. The reviews you&#x27;ll see on the internet never use such rigs. So the results are mostly non-sense.<p>ARM vs x86 ISA? Entirely pointless discussion. It is like saying Germany has a higher GDP than the UK because they speak German rather than English.<p>So what do you use for comparison? You need to base the benchmarks on real-world tests. Who cares about kraken or octane? Fire up a 1000 page loads for some common top sites on Safari, and calculate via image capture and analysis, the time the device takes to render the page. The time it takes to render the screen is one of the key psycho-visual cues to device performance. Which is why iOS has been steadily decreasing their animation intervals every iOS version - to make it feel faster even though, it might not actually be.<p>Another key metric is how long things take to copy&#x2F;move&#x2F;download&#x2F;encode - the wait interval. How fast does the device install apps? Typically, users get a sense of device performance when they try to download an app. The user wants to access the app he&#x2F;she wants to access. All of the things your device must do - download the app binary, authenticate your user with the app store, install the downloaded bundle and then launch - all these things are crucial indicators for device performance.<p>This article talks almost exclusively about benchmark performance. Not a single sentence about how fast the iPad Pro actually feels in comparison to the iPad Air or the Surface Pro. Very little words are devoted to talking about the capabilities of the platform. The Surface Pro runs Windows desktop and the Macbook runs Deskop OS X. Radically different capabilities on these platforms. On a Macbook, I can SSH into a production server, reboot it, while I write code in XCode. How does it compare with what I can do on an iPad or a Surface? What is the point of this article even?