 When I signed off at the end of the video about our amazing fast new all SSD storage server, I thought it was as simple as, okay, let's load the final OS on this thing, chuck it in the server, we're ready to start editing off of it. But it wasn't. So our story begins with some short video clips that I sent over to Wendell from LevelOneText complaining about Windows storage spaces on our new 24 drive NVMe server machine here. Because what was happening was while I was copying files to what should be one of the fastest storage servers on the freaking planet, I was getting great performance sometimes, and then rock bottom performance others. We're talking like 10, 20, 30 megabytes a second. So Wendell dug into the system logs and discovered that there was some kind of a problem at the driver or PCI Express level where it was actually resetting individual drives, like they were effectively timing out for seconds at a time while the data was in flight, and then the poor array would be sitting there trying to figure out what to do while a drive is effectively MIA. Then the drive reset would finish, which is essentially like if you were pulling a drive out for like two seconds and then popping it back in, and then the transfer would roll along at multiple hundreds of megabytes a second, or we even saw at times numbers as high as 20 plus gigabytes a second in Crystal Disk Mark. Then it would hitch again, rinse and repeat. Obviously, I can't deploy it like that. So I thought it was my knowledge of Windows storage spaces or lack thereof, and that I had configured it wrong. But then the mystery deepened. So this dropping out behavior actually happened with a simple Windows software raid with just four devices in it. I mean, that's a relatively pedestrian 16 gigabytes a second. By the way, this guy is our sponsor for this video, Pulseway. With Pulseway, you can remotely monitor, manage, and control all your Windows, Mac, and Linux machines from one app. Create your free account today at the link below. So we tried all the usual things. We tried updating the drivers. It was using the Microsoft drivers. We put the latest Intel drivers for these NVMe devices onto the system. That didn't work. We tried tweaking the power management to prevent the PCI Express lanes from switching to lower speeds when we were accessing all the drives. And that could be a desirable behavior because there's so many drives in here that you're going to run into other system bottlenecks before you could possibly hope to use all the bandwidth of even a PCI Gen 3 link. So Gen 2 could be a pretty good bet. But when it's happening automatically, this speed switching takes time. And that could be part of what's causing the problems. But neither of those things or both of them were able to solve the problem. And we only got a small improvement in the behavior. So Wendell suggested, gee, why don't we go over to Linux as he tends to do? But then get this. We got the same dropouts on Linux. That seemed to suggest a hardware issue of some sort. So, guys, this is why I ultimately made this video about it, because this is pretty dry technical stuff for a lot of people. But I thought it was fascinating. NVMe is already so fast that a lot of stuff, particularly software, is not engineered for it, which is turning out to be a bit of an industry wide problem. And when you take 24 of these drives that are capable of multiple gigabytes a second on paper, that is now 24 times the problem. Think about it this way. Even with eight channels of memory, which is pretty impressive, the theoretical maximum memory bandwidth of our system here is around 200 gigabytes a second. And real world, you're looking at more like 100 to 150 gigabytes a second. Now let's talk about this storage array here. This is capable on paper of about 100 gigabytes a second in reads. So we would need, assuming perfect efficiency, which obviously never happens in the real world, nearly half of our memory bandwidth just to handle shifting data around when we're reading or writing to our storage array. That's ridiculous. And even the Linux kernel is going to be on the struggle bus when you're talking about that much data, as Wendell so succinctly put it. Because here's the way it's supposed to work. The operating system kernel asks for some chunk of data, let's say a loot of your wife to enjoy on your lunch break. All right, the disk says, yep, no problem, but NAND flash is pretty slow. So I'm going to need a sec to load that into my buffer. I'll let you know when it's ready. The disk gets everything ready, loaded into the buffer. And then it sends what's called an interrupt to the CPU to say, hey, all right, it's chill. You can swing by and grab that data now. But here's the problem we're running into. If the CPU core that the interrupt was intended for is too busy doing something else, or it gets put to sleep, or it gets reassigned to some other task in the middle of this process, which can be quite common on multi-core CPUs, that interrupt never arrives. Your processor never goes and gets the data. And the whole train comes to a screeching halt. And that is why we had no issues last video slamming the individual drives with data. But then as soon as we put a file system, you know, as soon as we started running a ZFS raid and our CPU was doing parity calculations while we were reading and writing to the array, making the CPU actually do any work, we were getting crippling errors all over the place. So AWS just rolled out NVMe. And there are a ton of threads about issues under heavy load suggesting that this appears to be an industry wide problem. And the dumbest part of this is that I don't actually even need my server to be this fast. I'm only hitting it with a 40 gigabit connection here. That's only four gigabytes a second maximum. So Wendell actually even thought of turning down the PCI Express links to Gen 2 and just leaving them there. Gigabyte, meanwhile, the makers of this server was like, sorry, wait, you want a speed limiter on this thing? But then Wendell ended up finding a software way to do it. But then it turned out there was a kernel bug, something, something, something. Ultimately, it didn't pan out and it didn't work. Anyway, that's okay because Linux already has kind of a solution to this. Now, very, very high speed devices like RAM based caching devices operate in a completely different mode called polling, where the kernel essentially assumes that the device is so fast that the data is going to be ready right away. And it would add a lot of overhead to do this on slower drives because there'd be a lot of pointless, hey, are you done yet? Hey, are you done yet? So a single NVMe doesn't need to be polled. But 24, oh, there's an argument to be made for operating in that mode. So here's the mitigation that Wendell implemented. When possible, the kernel is going to wait for the interrupt because that's the most efficient thing. But if it waits for too long, the queuing algorithm will just have the CPU pull the drive rapidly and say, hey, do you have that? Do you have that? Okay, great. I'm going to take that now. All that tweaking and learning means that our final config ended up being quite different from the initial intention. So we're using the latest version of Proxmox, a Linux distro that's designed for virtualization with the ZFS support out of the box. And while we had actually initially intended to use ZFS, we were hitting 100% utilization on a 24 core, 48 thread CPU and doing best case scenario, assuming that the bug didn't surface 10 gigs a second reads, four gigs a second writes, which would have actually been fine. Remember, we've only got a 40 gigabit network connection, except that the access latency was not really suitable for a multi-user video editing environment. It was over 150 microseconds. And the craziest part of that is that we actually hit those numbers even with some pretty esoteric tweaks like disabling arc compression. I mean, most seasoned ZFS users would freak out about doing that. But the problem is that arc compression makes three copies of the data and memory while you are writing. And remember how much leftover memory bandwidth we have? So, yeah, tripling the load there ain't going to fly. So new plan. Linux multidisc ain't perfect. It's Linux's own built in software raid. And the main disadvantage is that in the event of an unexpected shutdown, it'll be really slow for the 30 minutes or so that it takes to resync four terabytes of data. But that should be fine. I mean, that's what the $17,000 battery backup in this room is supposed to be for. So we settled on four striped software raid fives. And the next experiment was to play around with the chunk size. So that's how the blocks of data are broken up on the raid as well as the block size, which is on the file system level. So the default raid chunk size is 512K and the file system is 64. But when we were running benchmarks based on an editor usage pattern, we actually found that the 512K chunks were a little bit higher latency than we'd like to see, which is really, really important when you're scrubbing through files on a timeline. So we actually ended up using 128K for both, which happens to line up with the buffer size on these devices. Perfect. Now, the conventional wisdom for accessing very large files over a network share would actually be to use a very large chunk size, like even one megabyte. But while that would be great for ingesting like big batches of new footage, when we're skipping around rather than reading them sequentially with many users doing that all at the same time, it actually makes sense that this would work well and experimentally so far it seems pretty good. I just realized I forgot to drive upstairs. I'm going to go grab that. With multi-disk, we ended up with a maximum throughput of around 16 gigabytes per second reads and 8 gigabytes per second writes, which obviously is way less than the maximum this hardware can theoretically do, but there's a lot of overhead to contend with and besides. That doesn't mean that there's no benefit to having all of this performance in reserve. So the latency advantage is something that we've already talked about. We've actually seen that high latency storage can cause instability in the video editing software that's accessing it. But another benefit, counter-intuitively, is that because the storage is so fast, no one chiplet on our CPU can keep up with it. A disadvantage of a chiplet design is that it's got huge horsepower, but it's hard to harness all of it for one single task, like a file copy from one user over the network. With that said, it's great as a multi-user experience because each discrete user, like, let's say, a camera operator who's dumping red footage and a video editor who also has to work at the same time, end up having their access spread over multiple chiplets that are individually kind of limited. So we had, remember guys, 150 gigabytes per second of memory bandwidth. One chiplet can't get at all of it. So when we have one user copying a file over the network, that user can only get to one or two cores. So there's no way that user can monopolize all the resources on the system because of the way the whole thing is architected. All of this, in theory so far, we haven't actually thrown our editors at it. So let's go see if it's booted up and get them to try it. Taron was about to eat, but now he has something very important to do. What? You laughed, Dennis, but you need to help too. No. Okay, we're off to a good start. Is this new Wannock? Yes, this is new Wannock. Hi Alex. Hi, how's it going? Hi Alex. I have a new server to log you into. I wouldn't even do that at this point. No. What would you do? No, no, this is not real. I'm acting out, come on. And I'm not acknowledging it. Ed, you too. Wait, are we supposed to work off of this? I just want to know if it works. So you're supposed to work, but like, not important work. I'm going to mirror old Wannock over to this one one more time. Okay. So anything you do here will be overwritten. So we're not supposed to use it. Look, do you want me to just? There's things to do. So do them, but then we're going to just wipe it out. Okay, when are you going to wipe it? What part of test is not clear? Just open up a project. How's it going? I don't know, seems fine. It's you know, let's see if we can pump it up to full res. Well, that's less of a network bottleneck thing and more of a, you know, the rest of the system, but okay. Just playing it though, which is kind of surprising. Well, Linus, you having wanting to do increasingly ambitious projects. I appreciate that we now have more space for them. Us running out of space has been a large, large, large problem. Good work. It's not broken. So this does this feel any different than it was before? It might be a little snippier, snappier, better. You don't have to lie to me. But I don't know. I don't really see much difference. This is at one eighth res though. What if you crank it a bit? Um, okay. Thank you. But is it better? Why are you asking me? I'm asking you. That's the whole point of this exercise. It's a blind test. You can't do anything? I'm not participating right now. Oh, okay. Sorry. Fine. From what I can tell, it's actually a lot snappier than what I remember. The editors say it's good enough and we're not getting any data corruption and the performance is fine. But every one of these line items is an NVMe device timing out. And we actually did some troubleshooting that I haven't talked about yet. So one of the first things that we did was we swapped out the 24 core CPU that I originally configured the server with for a 64 core one because we found that with the 24 core, the CPU during heavy reads and writes was getting hit with 50 or more buffer flushing tasks that were each pulling 20% usage of a single core. Just choking the poor thing. And 64 cores did help significantly. But I also didn't want to allocate a four or $5,000 CPU to the server. So we dialed back to 32 and that ended up being a big improvement as well. So bottom line, the 32 core, so adding just another eight cores and then tweaking the timing between going from interrupt based to polling based access to the drives gave us good enough performance that we've seen three gigabytes a second when we're hitting it with three different clients at a time in the real world without any significant jumps in access latency or dips in transfer speeds. So we're rolling with it. But there's something to be said for like a dual socket approach to this with more spare PCI Express lanes and even more CPU cores or, oh, I don't know, AMD working with their OEMs to make sure that, you know, when you actually hit their PCI Express lanes, it doesn't cause a bunch of traffic jams elsewhere in the CPU. A massive shout out to Wendell from Level One Text, by the way. That guy is anything but level one and I would strongly recommend going and subscribing to him if you love this kind of deep dive server stuff. Linode provides virtual servers that make it easy and affordable to host your own app, site, service or whatever in the cloud. Other entry level hostings work when you start up, but you'll eventually want to get something powerful, customizable and easy to use for cloud computing. They've got a DIY option if you want a full custom setup or you can easily set up your own server with their one click apps. You can deploy Minecraft, CSGO servers, WordPress and much more. And you can even spin up your own VPN and have plenty of space to host a website, app or game server. They've got affordable pricing with no hidden fees that try to sneak onto your monthly bill and they've got 100% human 24 7 365 customer service via phone or support tickets. Get $20 in free credit on your new account with code Linus20 or by clicking the link in the video description. So thanks for watching guys. If you're looking for another servery video to check out, maybe have a look at our petabyte project update. And actually, we're going to have another petabyte project coming soon. So make sure you're subscribed so you don't miss it. And remember how much memory, oh, no, I need to scroll down. Okay, no problem. But give me a second. Um, fuck off. And remember how much leftover memory bandwidth we have. So yeah, fuck off. Why then your CPU goes off? Why isn't this working? We're using the latest version of Proxmox, a Linux distribute. Fuck off. I need this to work. What the fuck? Okay.