 Ow, damn it. What is that a total 128 cores? 64, but good try. And a terabyte of memory. That's 256 gigs. Oh. Super Micro told us that we're not allowed to build this server ourselves. They have to build it for us. Naturally, we said no. So we are gonna be taking one of the thin one-use storage servers from the Petabyte of Flash project and seeing just how fast we can drag race it with a handful of Kyoksia CD6 drives Octane acceleration, 64 epic cores, the fastest we can get. And is this 200 gigabit per second networking? It's 200 gigabit. And 256 gigs of memory. The Optane's just for boot. This thing's gonna be crazy fast. Almost as fast as I can segue to our sponsor. GlassWire, are you having poor quality video meetings? Use GlassWire and instantly see what apps are wasting your bandwidth during your meeting and block them. Get 25% off today using code Linus at the link below. In a way, a server is a lot more like a laptop than it is like a commodity desktop made of off the shelf components because they tend to be way more tailored to a specific use case. And they're not really as flexible. Say you wanna build a storage server. There's a dozen different ways to skin that cat. Pardon the expression. For example, here at Linus Media Group, our primary concern is getting as much capacity as possible at the lowest possible price. So we're willing to give up some compute in favor of stuffing more drives into a single chassis. That's why our storage servers tend to be this thick or this thick. The reality is that 4K or even 8K video editing is pretty demanding, especially if you've got 10 or a dozen editors working off the server at once. Compared to enterprise or scientific applications, it's not even close. So that is why any good server deployment starts with the chassis. This right here is the SuperMyPro Super Server 1124 US. And it is all about density, not storage density. Because if we went with a 2U, remember that dual layer one? Oh yeah, old one. They absolutely could pack in more drives, but they choose not to, because you're going to run into performance bottlenecks if you don't have enough compute. And that's the density that we're increasing here by going with these one use. Each layer of this, 12 drives, yes, but two CPUs. So no bottlenecks, right? That's the idea. SuperMyPro only sells these as a complete system these days, meaning that it must leave their warehouse with a minimum of two CPUs, four sticks of RAM, and at least one storage drive. And the intention there is for them to be able to ensure quality and compatibility. And then as a side benefit, obviously they make some money off the parts. But because of Petabyte of Flash project, we were able to get our hands on some bare bones ones. So let's take a closer look. Wow, built in on board, you've got dual SFP ports. Are those 10 gig or? All four of those are 10 gig. All four of these ports, our J45 and SFP are 10 gig. Dual USB 3s, we've got an IPMI management port, serial VGA, gotta love that VGA, as well as two PCIe 16x slots back here. And what do we got for power supply? There's more, there's three. There's three, oh, there's a third one. Oh, look at that. Oh wait, there's actually four. There's one more like hidden inside, we'll see that later. Oh, cool, okay, let's have a look at our power supply. Obviously dual up to 64 core CPUs. Wow, 1200 watt power supply. Change. Strictly speaking, they didn't actually send us a bare bones, they sent us a completed one. And we took it apart. Do I ever have the story for you? On a call for this project, the super micro guy was like, you know, taking out a PCIe card, that's easy. But you know, get to a CPU, there's pins and thermal paste. I'm like, bro, I have probably taken out slash installed at least 1000 CPUs. Yeah. Is it wrong for me to just love looking at thermal solutions for super thin systems like this? Really, you know what I'm looking at? The RAM slots. It's like 60% of the width of the server, it's just RAM slots. It's a forest of memory slots. Why aren't we putting more memory in then? Fetch me more memory. No, no, no, no, no. The thing with Epic is we wanna have all of the channels filled out, so that's eight per CPU. But once you add more, it can be harder to hit the same speeds and the same latency. And speed and latency of your memory is super, super important if you're running software RAID, which is exactly what we're gonna be doing with ZFS. We're using ZFS, right? Well, for now, just to test it, but the actual deployment is gonna be using WekaFS, which is a different thing that costs hundreds of thousands of dollars, but seems to be software RAID too, so yeah, I guess. Whoa. Oh, what the hell? That's cool. You dropped something. It comes out as one big fat mama of a module. I love it. I don't know where that's from. I don't know where that's from. Now, this is a fun fact. Small fans not great at moving a ton of air because I got these little bitty tiny little blades, but what they are really good at is generating a ton of static pressure, which is really important in a deployment like this. See, look at the front of this chassis. It's gonna be all full of drives in there, right? And in order to fill it with drives, you've gotta have a backplane for them to connect to. Well, that backplane has a hard PCB and you can see that there's only tiny little gaps in it wherever they were able to get a little hole to draw air through the front of this chassis. They need to generate enormous static pressure in order for there to be enough airflow to force over the CPUs, memory, power supply and PCI express cards. Did I say power supply? Power supplies, they're redundant in the event that one fails. And it's also super useful for connecting your server to two independent power sources in case your power source fails. Side note, Jake, I think this might be the thickest PCB I've ever seen. Holy. I mean, you want rigidity, obviously, especially somewhere where there's gonna be mechanical strain on the device. It's like two sticks of RAM thickness. Here, let's get a shot of this just for context. Here's a stick of memory. I think it's more like three, Jake. What? That PCB is almost an eighth of an inch thick. That's crazy. Oh, this is interesting. You can see that in order to avoid recycling any of the hot air back to the other side, they've got these little rubber curtain things anywhere where cables have to pass between the front of the chassis and the back. And that's not the only cable management trick it's got up its sleeve. Power runs up this side, but the front enclosures also need PCI express connections for the NDME drives. And all of those are flat connectors, check this out, that run right in between these memory slots to these sick, freaking PCIe connectors that go into the motherboard. And do they have any cards for them? No, they just all come directly off the board. Yeah, there's the little ones over here too. Of course you can add even more NDME storage if you wanted to. There's the three, excuse me, four PCIe slots here. Wow, this is- See, this little guy, he's right here. Oh, there it is. That's where we're gonna put our Optane. Oh wait, actually, no, it doesn't fit. Ah! Oh God. No, it's fine. Oh, cool, okay. So this is a dual riser on this side. You've got a simple PCIe 16X to 16X slot. And knowing AMD Epic, it's probably running at full speed. It's running at full speed. Actually, all of this is just gonna be full speed PCIe Gen 4. And then over on this other side, we've got, I believe this is a PCIe 32X slot. Oh, that is crazy, Jake! It is a 32X slot, and you can see, they've actually got the pins that correspond to each of the 16X slots silk-screened onto the PCB. Well, if you think that one's crazy, look up here. That's amazing, I love it. There's one 16, another 16, and then a, what is that, 8X? That's an 8X, right over here. So I think some of the NVMe's run off of this. Oh, you know what? The 8X is running these SFP ports at the back. Oh! No bottlenecks. And then one slot, and then those are two more 8X NVMe connections. They're just running to the front, yep. There's so much PCIe in the servers. And again, very purpose-built, right? Yeah. You can put a GPU in here though, look. GPU power. If you wanted like an A100 or something. Again, if you had a very purpose-built specific use case. Oh my God. I mean, we did see a storage deployment recently where GPU acceleration was used for RAID parity data. I have a bit of an update on that one. Wendell informed us that there could be some issues with that particular solution. We tested it, there is an issue. Basically we stopped the array, edited one of the drives. I think we edited 32 bytes of it to be something different, started it up and it didn't fix it. It just has no error handling whatsoever. It is depending on the drive to tell it that there's an error. Jake, I just realized something. I was trying to figure out why the front of the slot was over here and I was like, right, that's where the power pins are. Yeah. It's got normal size power pins and then these itty bitty higher density data pins. Now the goal today is to see how the system would perform if you were just to set it up yourself with something like ZFS. But even with a more optimized and actually specifically built for NVMe file system like WekaFS, you still need a lot of CPU compute to handle things like networking, the actual connection to the NVMe drives themselves and any sort of networking overhead. Fortunately, AMD stepped up to the plate and provided 12 of their 7543 epic processors. So those are 32 core each for a total of 64 cores in each of our six servers. These are configurable to a max TDP of 240 Watts and a max boost clock of 3.7 gigahertz. They're not quite as fast as the 75F3s we had in the G-RAID server, but they're still plenty potent for what we're trying to do here. So let's get them installed. I gotta prove super micro right here. I know how to do this, David. I swear. I swear I can put a CPU in. I don't know, Jake. Watch me screw this. Oh. Oh! Ah, you saw nothing. You know, my hands quite aren't what they used to be. All right, David, I'm doing the most dangerous part here. Thermal paste. Ooh, don't wanna mess this up. Oh, oh, I already messed it up. Is there treasure in the back? Look at these bad boys. It's crazy to think that this could handle a 280 watt CPU. Like there's just underneath here, there's gonna be a massive vapor chamber that just spans the entire thing. Now it's time for the tedious process of installing all of these sticks of memory. The DIMMs are made by Samsung. They're 16 gigs each and they run at 3,200 megatransfer per second. But the most important thing about them is that they're qualified by super micro for this particular server. And I get it. You know, who would wanna run on qualified memory in their mission critical server? It's like drinking from a non LTT store qualified water bottle. Crazy. I think the craziest thing about this memory setup is that it's not even that crazy. 256 gigs of ECC error correcting memory would be mind blowing for a desktop, but for a server, this is pedestrian. This is a storage server. We don't actually need to put enormous data sets in memory for these CPUs or GPUs to crunch away at. These are to make sure that each of our CPUs gets two full fans worth of dedicated air flow blowing through them. In our final deployment, as part of our petabyte of flash storage project, we're gonna have six of these acting as NVMe over fabric posts and it's kind of similar to iSCSI in that your storage is in one box over here and then it's connected via networking to your compute box. But NVMe over fabric was designed specifically with NVMe devices in mind. So it's way more performant, but to push that kind of speed, you need to make decent use of the drives, right? And that requires a lot of networking. A hundred gig? Can I get a ha? 400 gig is what we're targeting with dual Nvidia connect X six series cards. And I couldn't help noticing that one of these has a half height bracket on it. Oh, you want it on this side? It's for cooling, for more cooling, get them separated. It's just gonna hang there. The teamwork in it. I'll go at it from one side, you go at it from the other. Yeah, I think they called that a s***. Ew, no. I thought you wanted to put this one on this one. It doesn't fit. Oh. Yeah. It's fine, we can just stick it in there. Too much cable. Oh, careful. Yeah, we really need to not break any of these if we're gonna hit our petabyte of flash storage. And also we don't want them to be able to say, I told you so. They did tell us not to bother them. This may be the most overkill boot drive of all time. Especially like it doesn't, it's- It's not redundant though. So it's like actually not that great. Hi. In fact, many server motherboards, most even have an internal USB port that is exactly for that. That is what it's for. Really? Yeah, it's for just running an OS off of USB, but just using a cheap thumb drive and plugging it in. A lot of them also have an internal like little powered SATA thing. And that's what we'll be using for this one. Where is that? In the real deployment. Yeah. I took it out. Oh, well, where does it go? Oh, is it the super Dom one here? Yeah. Something about that doesn't look right. Yeah, I did not put this in right. Yeah. Oh, God, every time. You didn't screw this in. Oh, I forgot. Oh, I can still access it. Where'd the screws go? Just get it from here. Is there a through hole? Yeah, so like a- Oh, God. What about this side? There's a bit of flex. No, that one I screwed. Oh, I didn't screw that one in either. Last but not least, storage. The actual deployment of this cluster is gonna be making use of 12 CD6 15 terabyte drives per server. But because those drives already have like specific demo data assigned to specific slots, we had to be very careful about taking them out. Did you see my little diagram? Oh, no, I didn't. Oh. Oh yeah, baby. Oh my gosh. It's perfect. Okay, I mean- I labeled the drives. I didn't wanna screw it up. That's fair. That's fair. Because that would be catastrophic. So instead, we're gonna be using these seven terabyte CD6s that we already had laying around, and we're gonna be installing TrueNAS to run ZFS on them, just to see, like if you were to buy this server and these drives, how much could you get without spending $400,000 on a file system? Yeah, that. I mean, we're expecting really impressive results even without the fancy file system because these are PCIe Gen4 drives that are capable of an excess of- There's six gigs. What is it, like over six gigs a second? These are exactly six gigabytes. Six gigabytes a second of throughput. So put together, that's around 75 gigabytes a second. The interesting thing is the 15 terabyte ones are a little bit slower. I think they're five and a half gigabytes a second, so it works out to be closer to like 65 gigabytes a second, which is a lot closer to the 50 gigabytes a second that our network can do. Shout out Super Micro, by the way, for these tool-less sleds. This was so fast compared to when I built that Simply Double server that had- Did you have to screw them all in? All of them. Even with one screw, boy does that ever add a lot of time. But you kinda have to do two. So that's like, oh, 96 screws. He's not even doing it right and he's complaining. Oh, are we done? That was it. That's it. Wow, it's so cute. Freaking crazy. I mean, cute is like not giving it enough credit. 400 gigabit per second. Okay. Shall we plug her in, Captain? It's gonna complain that I only plug in one of these. And then we'll tell it to shut up. This one doesn't have a shut up port, but it does have a shut up function called unplugging the power supply. Oh, well we could just plug it in. Here, we're gonna plug the power supply into our server. Oh, that was a little rough. Oh, I got the wrong power cable. One moment, please. When I was living at Yvonne's house- Like with her parents? Yeah. I had a GPU make that noise when I forgot to plug in the PCIe power connector. Oh yeah, like an 8800 or something. And it made her dog throw up. This thing is surprisingly quiet for a OneU. I mean, I know it's idling, but still. I guess if it's not doing anything. Yeah, well it's nice to not have to hear it just whining. Some of them, the power supplies. Oh, nevermind. That's still not bad. Two 32-core processors. All right, we see 13 drives. Looks good, we got our boot drive and our 12 seven-ish terabyte drives. It's time to make our pool. Should we do realistic or should we do full send? I actually don't think a stripe is gonna be that much faster, honestly. We might be best to just do like two RAID Z1s. Two RAID Z1 pools would allow us to have two drive failures before we actually experienced any data loss. And the way Jake's going to configure it is with two six-drive VDEDs that we will then combine into a single pool. All right, what do you wanna call this? I am speed 69 TBB. You know what, it's even.84. Like that's double 420, you know? We're gonna make a couple of tweaks here. They've actually updated it so A time is off by default. Thank you. What is A time? It's like that it records the access time of the data. You only need that for like very specific use cases. Or diagnostics, I would think. Yeah, but you don't want it. It's not good for performance unless you actually need it. We're gonna go from 128 to one meg because that's kind of closer to our use case of like video, which is big files. If you were to host like a database or something where you have lots of random reads that are small, like especially like a text-based database, you would probably want a smaller record size, but for us, one meg is good. 128 is the default for a reason. That's excellent for a mixed use case. Yeah. Okay. And we wanna do one more thing. We're gonna set the ARC that is the RAM cache of ZFS to just be metadata only. If you use it for files as well, when you have such fast backend storage, you can actually lose performance. So setting it to metadata only gives us a little bit of acceleration from it, but not the same kind that ARC would for hard drives. So we're running an IO depth of 32, which is somewhat unrealistic, but two threads per NVMe, so 24 threads total at a 128K block size. You make the server go fast your way, I'll make it go fast my way. Here we go. Ready? Ah! Did you just unplug the network? Oh, maybe. Maybe? But I did it really fast. Yeah, I think you did just unplug the network. Cool. Well, it's fine now. 15, 17, 20. Now, we've done, you know, 18, 20 gigabytes a second on a ZFS pool before, but what you have to consider now is that we've done that on servers that were generally double the thickness. So in a cluster deployment where density is key, you're able to get effectively double the performance of your drives by having two one use by adding all of that compute. That's the point of this. And that's what made these ideal for our petabyte of flash project. We're ramping up, baby. Almost 30 gigs a second. Oh, wow. Look at the CPU usage. Those cores are pegged. I bet you if we switch our test to a 128 block size, leaving the array at one meg, this is gonna go even faster. Yeah, there's 22, 20. See if it ramps up even higher. Five threads at 100%. Well, there's more than that if you look at it realistically. So this is a write test, sequential write. We're looking at around 20 gigabytes a second as well. What that tells us is that we are still CPU limited because in theory, these drives don't write as fast as they read. These are more read optimized data center drive. That's actually really impressive considering that we're dealing with parity data here though. And is that a fast? Bump the threads up a bit. Man, I am excited to see what this thing can do when there's another five of them in a cluster. Yep. Okay, so this is a random read 4K block size. We're doing four threads per drive and a 64 Qdip. This is not only gonna be a petabyte of flash, it's gonna be the highest performance setup that I might ever see. Look at how dog crap that is. Oh, that's a shame. 150,000 IOPS. Individually, these drives will do more than that. They'll do a million each. That's that one meg record size kinda hurting us. Sheesh. Look at our cores. They're just packed. Wow. Yep. 47 threads at 100% right now. Poor thing. And what about a random write? These poor drives are just abusing them. Oh, that's embarrassing. 20,000 IOPS. This is literally slower than a hard drive, but a hard drive sequentially. If we were doing 4K random writes to a hard drive, it would be way slower than this. So that's something you gotta keep in mind about these numbers. It's not as simple as just megabytes a second. The kind of data that you're hitting your storage device with makes an enormous difference. Really, that brings us back to what was kinda the whole point of this video, doesn't it? Yeah. That servers have to be designed for the application that they're intended for. And these are absolutely perfect for what we will be doing with them, but not perfect for what we do with our regular servers here. Like we wouldn't replace new Nuonic with one of these. That's the thing with software, man. Random reads and writes are just not it. But you know what is it? Our sponsor! Manscaped. The new Manscaped Ultra Premium Collection is an all-in-one skin and hair care kit for the everyday man and covers you from head to toe. There's the two-in-one shampoo and conditioner, their body wash with cologne scent, hydrating body spray, deodorant, and a free gift, moisturizing lip balm. Ooh. Your man maintenance just got easier. And best of all, all Manscaped products in the Ultra Premium Collection are cruelty-free, paraben-free, and vegan. Visit manscaped.com slash tech, or click on the link below for 20% off and free shipping. If you guys enjoyed this video, go check out part one where we got into way more depth about the complete configuration, including taking a close look at the 8 GPU server that is gonna act as the head controller for the six of these that we're gonna have stacked up. I wanna do a video like this on that server and boot it into Windows just for shits. NVIDIA specifically told us not to mine on it, but could they hate us anymore at this point? Yeah.