 I'm not feeling great today, but we have a dire situation here at Linus Tech Tips. Our Wannock server, our main production editing storage server is completely full. Fortunately, I have been planning an upgrade to it for the last little while that is going to be freaking awesome. This is new Wannock. It's gonna have more storage space. Whoa! Over 100 terabytes of NVMe storage. It's gonna be faster. It is going to have lower power consumption. It is going to have more processing cores. It is going to be epic. And perhaps the most epic thing about it is going to be how many PCI Express lanes it has. You see, PCI Express gets used for all kinds of things. Graphics cards, video capture devices, peripheral expansion. But the thing is, it's been fast enough for all those consumer uses for many years now. So what's driving the creation of faster and faster PCI Express is actually the server world where it gets used for storage. That's right. That is a direct PCI Express by 4 connection at the back of all 24 of these drives. And the truly unbelievable thing about this server is that unlike our existing Wannock server, which takes a handful of those PCI Express connections and splits it across 48 drives, this one has a dedicated connection for every single one. We are going to be putting up some crazy numbers today, ladies and gentlemen. And it's going to be brought to you by Seasonic. No marketing today. Just happy holidays from Seasonic. Go check them out at the link in the video description. Really? They paid for a spot for that? Nice guys. While I'm cracking these open, let's talk about the drives that I'm using for the server and why. Reason number one is that Wannock is full, so I needed a capacity upgrade. We're going from 1.2 terabyte drives to 4 terabyte drives, because that's what a couple of years of NVMe development does for you. Reason number two is we didn't want to go back to SATA because NVMe was such a great company. So we're going to go back to Wannock and we're going to go back to SATA. We're going to go back to SATA because NVMe was such a great upgrade for us in terms of not just performance, but stability last time, thanks to the extremely low access latencies. Reason number three, and really the main reason for these drives in particular, is that we got a great tip from Wendell over at Level One Techs that Facebook was apparently flipping a bunch of these on eBay because they're upgrading to Optane. So these were a really great deal, just a hair over $350 for Intel data center grade drives that apparently are still under warranty on top of everything else. Now, they're not the fastest thing in the world by today's standards. 3.2 gigabytes per second reads, 1.8 gigabytes per second writes, maximum read IOPS of 645,000 and writes of 48,000. But remember guys, it's almost not going to matter at all, because I'm going to have 24 of them in the same server. And it's just a NAS anyway. Now let's meet our server. This is the Gigabyte R272Z32. And we chose this particular model based on the glowing review that was given to it by our buddy Patrick Kennedy over at Serve the Home. Oh, heavy boy. This is a 2U server, so that's to say it is two rack units in height. And it was designed from the ground up for AMD's Epic 7002 Roam platform. And the advantage we get from that is that it actually has compatibility with PCI Express Gen 4. Not all Epic servers are PCIe Gen 4 ready. What Gen 4 compatibility means for us in practical terms is not necessarily that much today, but in the event that we wanted to upgrade, it means that we could effectively double the bandwidth to almost the whole system. So let's take a quick tour here. We've got dual 1.2 thousand watt power supplies here. Those are redundant in case one fails. We've got our CPU socket here with support for up to 64 cores, 128 threads. We've got 16 memory slots that run in 8 channel mode, freaking incredible bandwidth. And on the subject of bandwidth, most of the internal PCI Express slots are taken up, of course, by the 24 U.2 slots at the front. So the way that they're fed is they're using their mezzanine card slot here to run four of them. You can see that managed over here. SATA actually goes to these two bays in the back, so that's what you would typically boot off of. Now, the CPU that we're using is pretty overkill, but AMD sent over a bunch of CPUs and unfortunately, all the ones that kind of make more sense for this project, like lower core count ones, I sort of have earmarks for other projects. So unfortunately, I am locking away my 132 core basically forever. That's okay, though. We really need the upgrade. It really says a lot about the efficiency of these things, that all we need is one of these basic passive heat pipe coolers thrown on here, one of these shrouds, and just the airflow from one of the chassis 80mm fans to keep it cool. And remember, guys, it's validated for up to the 64 core one. This is cool. For my boot drive, I had intended to use the SATA bays at the back of the chassis, but then I realized I've actually got two M.2s that share their PCI Express lanes with this 8X slot over here. So I figured if I have a couple of these old Optane M.2s lying around, what the hey? Might as well throw them in raid one and go full PCI Express on everything. All right, let's turn it around and oh, oh, that's funny. Gigabyte sent over a demo unit and it seems to be a little bit broken. That's okay. Oh, they've got the stupid cable management things that make it really hard to plug in. I mean, that also makes it hard to come out accidentally, which is nice. Whoa, hey, okay. Now, I've actually worked with this motherboard a little bit already for a whole home one CPU and it takes forever to post. So I'm going to fire it up while we check the compatibility of the sleds that our drives were pre-installed on and see if we're going to have to swap all those out. I really, really hope I don't have to. It's up. That's good. Let's get in the BIOS and see if everything's detected correctly. Come on, baby. Everything detected. Whoa, what about memory training error? What with the what now? That is not all of our RAM. That is 458 gigs of RAM. Oh, okay, Brandon. I've got good news and bad news. The good news is reseating those two memory modules. Boom, 512 gigs of RAM. So we're ready to rock as far as that goes. The bad news is that due to AMD's architecture of their epic processors, where there's no actual chipset with functionality like, you know, a RAID controller or anything like that, this has no support for hardware RAID either on the M.2 drives or the SATA ones. If you want to run RAID on your SATA drives, you have to put an add in RAID card in. And I asked Gigabyte, well, why didn't you guys just put a RAID chip on the motherboard? And they were like, well, it would have used up PCI Express lanes, which, I mean, as part of the design of this board, we were trying to reserve for all these NVMe drives. So I kind of went, okay, fair enough. Now in Linux, you can run your OS on RAID drives, but Windows has no easy way to do that. So that's just something we're going to have to consider as we build out this machine. I'm going to shut it down for now, though, because the last thing that I really want to know is if I am remounting all of these drives to the included sleds. Let's see if they happen to be intercompatible. That would be so cool. That would be so cool. Oh, oh, yes. Yeah, they are. Oh, it is so tedious. It's so tedious. Well, that's not a good sign. Only my two M.2 drives are in here, which means that maybe the sleds don't go all the way back. Oh, that would blow. All right. I put one drive in each one and lined it up and you can see it doesn't go back far enough. I got to swap them all. Damn it. And I've had people ask me why so many people work here. You thought I was going to shock all these drives by myself and put them in mounts? What one person can do slowly, four people can do also slowly, but less slowly. Pretty tedious. Now I just need to install a couple of roles and features, including file server. Then, yes, we're going to shut it down and put all the drives in. We're going to hope they all work. Now, really the most benefit for a solution like this is when you've got like a storage attached to network and you're using like banks and banks of these to act as the storage for like a whack of compute servers that are connected over fiber optics on, you know, the other end of the data center or whatever the case may be. But come on guys, this is Linus Tech Tips. So we're using it as an 8K video editing NAS. I really hope these all pick up. Worst case scenario though, we've got a couple of bad ones and the keen eye among you might have noticed. There's two drives not installed. These are either cold spares or to account for if a couple of our drives are just DOA. I mean, that's the thing with used hardware. You get a good deal, you might get a couple of bad ones. All right, that's a good sign. Every one of them has its light lit, but we won't know if everything's picked up until we actually see them in here. Boom. Okay, they're all there. Man, storage basis takes a lot of your capacity by default. Oh, actually, that's not that bad. It's about 70 terabytes. Okay, let's try that. As long as it works. Oh, you got to be kidding me. This stupid error. Like five years or whatever, it's had this problem. Man, they don't just allow you to choose your columns in the GUI. You have to do it via PowerShell. So dumb. Oh, you can hear it ramping up now, boys. 5.5 gigs a second. Well, that result is horrible. Worse across the board than old Wannock. Clearly, there's a configuration issue here. All right, so I did a simple mirrored space this time and we're going to try running that. We've got 44 terabytes here. That's better when it comes to sequential reads. Almost 11 gigs a second. Not even close to what we should be getting though. Okay, that's it. We're trying a simple virtual disk, which is basically just striped. If this doesn't perform well, then I'm at a loss. Wow, that is complete garbage town. It's even worse. How is that even possible? And Crystal Disk Mark is not the issue. There's a Microsoft tool called Disk Speed that you can use via command line to run sort of any mixture of loads that you want. And it seems like there's just this hard cap at around 10 gigabytes a second, which is like, I mean, obviously it's like good or whatever, but for this hardware, it's terrible. So we're going to free NAS. That's it. And it's been about four days. So we did manage to push Windows performance a little bit further. The drives needed firmware updates and they were doing some idle garbage collection, stuff like that, but we capped out at about 10 gigabytes a second, which to be clear is plenty for anything you're going to access over the network. I mean, we would need a 100 gigabit per second network card in order to saturate that. But that's not enough for us. We know this thing should be capable of so much more. So we moved on to Linux, Jake loaded it up with Proxmox and built a pool with four VDEVs, each with six drives in them running in RAID Z1. So RAID Z1 is kind of like RAID 5, but without the right hole issue, meaning that we should be giving up, let's see, four drives worth of capacity total, but with the resiliency of being able to lose up to four drives as long as we don't lose more than one in a single VDEV. So you would be, let's see, six. So four chunks like that of drives working together. Now that should have resulted in some rockin' performance, except it didn't. It was actually complete dog crap. Like, what did we max out at? Like a hundred megabytes a second or what? Maybe 200. 200 megs a second? Terrible. So it turns out as soon as ZFS was being loaded up, our machine was actually spitting out kernel panics. That's not a chicken emergency. That's like really bad. You liked it. Shut up. You liked it. That makes no sense. A single one of these drives, if I plug it into my computer, should be doing 15 to 20 times that performance. And we verified this. We checked in with our bud Patrick over at Serve the Home. He's got one of these boxes and he was getting great numbers with just 10 drives. So what could we do? Well, we called in the tech support for tech support. Wendell from Level One Text got on the horn with Jake, chatted it up on Discord, and between the two of them, they managed to figure out that it looks like there is some kind of compatibility issue between the particular Linux kernel that the latest version of Proxmox is running and ZFS in our particular system that's causing these problems. So we said, you know what? F*** it. We're just going to run a benchmark where we load up each of the drives individually and slam them all at the same time. Jake, hit it. He wandered away. He'll be back. Okay. You hear that thing ramping up? Oh my goodness. Oh my goodness. What's going on here? Is that over a gigabyte per second per drive? 28 gigabytes a second. At those kinds of speeds, we're at the limits of AMD's Infinity Fabric, and in fact, to get to those speeds, we actually had to overclock it a little bit. Thank you, Wendell, for the tip there. We pushed it too far. It's actually spitting out some memory errors. We're going to have to dial it back a little bit, but whatever. You guys want to see impressive numbers, right? Well, here they are. Now I'm running a second script that he created that writes, what was it, 15 gigs of data? So this is going to write to the drives this time. These numbers are nuts. There you go. Seven gigabytes a second that are right up against the theoretical limits of what can be done. So that's it, guys. That is basically the pinnacle of modern technology. Nearly 30 gigabytes a second of reads and just over 20 gigabytes a second of writes. Guys, to put that in perspective, your home network is going to be probably gigabit. This would be 300 times that to read it. You could read one full Blu-ray disk almost every second. Blu-ray, Blu-ray, Blu-ray. It's crazy stuff. Now we just need to figure out some minor details like how we're actually going to get an array loaded up on the thing because this is just, I mean, this is just theoretical stuff right now. And theoretically, I could have a good segue for a change, but we all knew that wasn't going to happen. Thanks to our sponsor, FreshBooks, for making this video possible today. FreshBooks is the cloud accounting solution that's built for how you want to work. And you can work not just how you want, but anywhere you want thanks to the FreshBooks mobile app. You can create professional looking invoices on the go, snap pictures of your receipts so you don't lose them, stay on top of important conversations and never miss an update. For example, being able to see when a client has viewed your invoice for the first time so you can get your money. Know what I'm talking about? Yeah, you do. So visit freshbooks.com forward slash tech tips to get your free 30 day trial. Today we're going to have that linked in the video description. So thanks again for watching guys. If you're looking for something else to watch server related, maybe check out the video where I lost all our data. You know, I was accused on the comments of that video of being a very bad actor. Brandon out of 10, how much was I acting in that video? Negative 10. Negative 10. That happened. That was a real thing.