 This has happened more often than I care to think about, and this time of year is particularly bad. I mean, obviously we want to cover all the new phones and CPUs and GPUs. And I mean, we've got this incredible video coming where we bought 20 used mining GPUs so we could find out once and for all if they're safe to buy. But we've also got a corporate mandate to maintain healthy work-life balance for our team. So who's gonna actually do all of this testing? Meet the subject of today's video, Markbench, or well, rather, Markbench. Mark here is an automated benchmarking tool that our labs team has been cooking up for the last six months. It's still early days, but even now, Mark is able to improve our test efficiency by some percent. And in time, we intend to make it freely available for personal use to our community. So let's take a deeper dive together, and maybe you guys can tell us what you think is good and also give us your feedback about what you'd like to see us work on as we continue Mark's development and continue to tell you about our sponsor. Simple MDM. They provide ridiculously simple Apple device management for IT. Enrolling your company's Apple devices and keeping them up to date doesn't have to be frustrating. Try it for free for 30 days on unlimited devices at simplemdm.com slash Linus. Let's begin with some napkin math to explain why we decided that it was finally time to bite the bullet and build Markbench. We'll use that upcoming mining GPU video as our example. Each of those cards will be subjected to 12 different benchmarks to ensure that it is free of strange performance anomalies. Let's say, optimistically, that the 12 benchmarks take three minutes each. That's 36 minutes. Factor in that each test runs five times. Now we're up to three hours plus half an hour of thermal stress testing that lands you at about three and a half hours per card multiplied by 24 cards, 20 from eBay and four lightly used control cards. And that is 84 hours of testing. And that doesn't even account for reinstalling drivers, swapping cards or taking bio breaks. So it's pretty clear that even if we could just pound back Red Bulls and power through it, that is not the kind of thing that we'd wanna do regularly. And do it regularly is basically in our job description. Oh yeah. Not like that. I mean to say that it is tech-tober and our corporate overlords seem to get their jollies out of scheduling product releases to cause as much inconvenience as possible. I'm pretty sure the target is actually each other, but retailers, media and consumers definitely end up getting caught in the crossfire to varying degrees. So if we want to keep up, the answer is automation. And while Mark doesn't look like much at the moment, everything has to start somewhere. So for now, Markbench is a Golang GUI with a Python framework that collects all the sensor and frame data that is output from our system. During each test, he collects this data using Presentmon and Libre Hardware Monitor. Presentmon is a tool for collecting frame data and is actually the basis of NVIDIA's FrameView software. And as for Libre Hardware Monitor, it's an open source fork of open hardware monitor, which gives us access to all of the sensors in our system. You know, fan RPMs, CPU, GPU temperature, power consumption, stuff like that. After a test is finished, our Python framework outputs the data in the form of CSVs, then converts those into protobufs, a smaller binary format. The data gets uploaded to a local ingest server before being sent to our cloud hosted Postgres database. In layman's terms, the labs team builds what's called a harness for every game we wanna test. Then using scripts, Mark adjusts the settings of the game, launches the game, loads up a benchmark, and then records all of the relevant data while the benchmark is running and stores it in a database. Rinse and repeat until all the benchmarks are done and we can swap off the card and put on the next one. Now, obviously we could get a similar level of automation using commercial software like 3DMark that conveniently already exists, but scripting automation into real games has a few major benefits for you, the consumer. First up, while a single bigger is better number is convenient, it really doesn't tell the full story. Take Intel's Arc A750 for example, it might actually perform well on average compared to say Nvidia's RTX 3060. But if your main game is CSGO, you are not gonna be happy with that purchase, which leads perfectly to reason number two, a collection of individual game benchmarks allows you to focus on what matters most to you. Geekbench for example, contains cryptography tests that heavily influenced the final score, but yet have very little bearing on how most people will actually use the products being tested. It's so bad that it's often dismissed as kinda irrelevant in media circles, even though it does also contain tests that are perfectly valid. Finally, Markbench is a great way to keep manufacturers honest. Everyone from Samsung to Volkswagen has been caught cheating on standardized synthetic tests to make their products look better than they are. So by giving ourselves the option to run any number of different real games, all of which will automatically be updated with new patches that would be hard to optimize for, we're making it extremely impractical to try to game the system and artificially elevate test scores. I mean, unless they just wanna optimize their product for real games in which case, well, that's not really cheating and we all win then, right? Once we've got our juicy data, we use Grafana to transform it into nice pretty graphs for your viewing pleasure, or well, at least that's the plan. We still have a lot of work to do as some of you have helpfully pointed out on automating our data visualization, because depending on what we're trying to convey, it can be really challenging to quickly and effectively present this much data. It's still a big time saver already though. Let's compare it to our current process. First, we choose from our suite of benchmarks, like say these ones, which is gonna come down to what we're trying to learn about the product. Does it perform well in lighter titles? What about the latest AAA games? What about older DirectX 9 games? That sort of thing. Then we get everything installed and patched and adjust the in-game settings to our liking. Oh, and don't forget to reboot the game if you happen to adjust that setting. Then we fire up frame view, set it for the length of the benchmark, go into the game, run the benchmark, wait for the game to load, then press the record button right at the exact right time as we load in. And then we play the waiting game. It's a very manual and tedious process that requires just enough of our attention that it's pretty hard to get any other real work done at the same time. But because Markbench has all of that built in, once it's up and running, you're free to do whatever you want until each card has completed the entire test suite. Wanna test 20 games? Easy. Wanna repeat every test five times so that we can throw out the early cold runs and then average the last three results? No problem. The other big difference maker is that Markbench all but eliminates human error. And trust me, once you've been at it for two, four, eight hours, it is really easy to forget a small step, like opening up your background data logging software or to accidentally leave DLSS enabled or something like that. And if you don't notice until you've already moved on to another card, then those kinds of mistakes can cost a lot of time given that in order to do things properly, you need to not only swap the cards, but also remove and reinstall your GPU drivers in order to redo the run. And you might think this kind of thing affects me and not you guys, but here's the thing. Whenever we post a review, we invariably see questions like, why didn't you test this? Or how come nobody ever talks about that? And we feel the same way. We want to know these answers, but in many cases, our hands are tied. Companies like AMD and Intel send out review samples for their products with only seven to 10 days until the embargo lifts or for Intel. That means our testing needs to be done extremely quickly so that we can analyze the data, write a script, film the video, edit the video, and finally upload and release. All of that takes a lot of time that we don't really have, meaning that we can narrow our scope, which sucks, ask our employees to give up their precious time off, which really sucks, or miss the embargo, which as a business where views and clicks drive income, it's frankly unsustainable. Let's look at some numbers to demonstrate that. Apple is a great example since they don't send us stuff ahead of release at all. Ha ha ha ha ha! This means that unless we can get an early hookup from somewhere, we can't perform any meaningful tests on their products if we want to get our videos out in a timely manner. And to give you some idea why that is so important, look at this. We rushed out a video on the M1 MacBook Pro on our Short Circuit sister channel right near the release day. It was super shallow because we had no time to prepare anything, but it got triple the usual views that we see on that channel. Then when we covered that same product on our main channel, this one, a few weeks later, in way more depth, we ended up with, whoops, below average viewership for our trouble. And again, this isn't just our problem. It's a problem for consumers because favoring friendly media is one of the best ways for companies to control the narrative around their products. That initial boost, by being one of the first to cover a new device, often creates a positive feedback loop that continues to drive increased viewership over the entire sales cycle. So if you do a search for, say, M1 MacBook Pro review, you are much more likely to end up with an Apple approved media outlet. And the most insidious part of this is that the companies that play this game well are smart enough to keep the rules of engagement so vague and nebulous that they create this environment where every media outlet, even ones they've never spoken to, will carefully control their criticism to avoid stepping over some invisible line. And this kind of horse is why we pushed back so hard when Nvidia threatened to stop sending pre-launch GPUs to Tim and Steve from Hardware Unboxed. Of course, as Nvidia pointed out, they are well within their rights to send GPUs or not send GPUs to whomever they please. And besides, they're more than welcome to cover their GPUs later, except that for the reasons I just outlined, this was a clear attempt to suppress Hardware Unboxed's influence and their growth by killing their launch day viewership. To Nvidia's credit, unlike Apple, they actually cared about the outrage from the gaming community, who to their credit recognized this for what it was. And to my knowledge, Hardware Unboxed is reinstated in their reviewer program. But there are many other companies who, like Apple, maintain much more strict control over who is allowed to review their products, which is why it's time to break that cycle. And Markbench is the key. By automating this testing, we're gonna be able to piss off whoever we want and still deliver near launch day data to our viewers. And over time, we plan to publish not only videos, but also written articles, which you can expect to find on the labs website along with the mother of all testing databases. Obviously, none of that is ready yet. But in the meantime, we're gonna have much more in-depth testing in our regular videos, and we're hoping to publish some extra data or content on our forums or on floatplane.com. We're not 100% sure what this is gonna look like yet, but you should sign up for both. Our forum is free, and the link below will be a thread where you can submit your suggestions for Markbench features. And as for Floatplane, it's got great extras and exclusives right now, like Dennis's epic martial arts training sessions leading up to our fight. The only thing I need to do now is use it to get all of that testing finished and compiled for the 20 mining GPU video. And oh, I guess I also need to find a way to segue to our sponsor. Squarespace, if you're building your brand online in 2022, you should absolutely have a website. And if you need a tool to help build that brand, look no further than Squarespace. Squarespace is the all-in-one platform to help expand your brand online. Make a beautiful website, engage with your audience, and sell anything and everything from products to content. We love Squarespace so much, we use it here at LMG. Its custom templates make it easy to stand out with a beautiful website that fits your needs. You can maximize your visibility thanks to a suite of integrated SEO features and their analytic insights help you optimize for performance so you can see what's going well and what needs a little work. So get started today and head to squarespace.com forward slash LTT to get 10% off your first purchase. If you guys enjoyed this video, why don't you check out our labs video about our headphone testing device that believe it or not, we are still waiting to get delivered. That was a rental unit. We paid for it months ago. Cool.