 I bet most of you forgot about the million dollar PC here and Honestly, I don't blame you. It's kind of been a while since we touched the thing, but I swear It's not our fault right before we were gonna do the big demo it broke then it broke again And again, it's just been the worst kind of problem to troubleshoot sometimes our full petabyte of NVMe SSDs gets detected and other times some of them don't but I Wouldn't make a video just to tell you it's still broken would I the solution ended up being shockingly simple and Something that I bet you've actually seen before but that's in the past This is the now and it is finally time when this video is over We are going to have the largest and fastest storage server on YouTube at least until we have to find all the boxes pack it up and send it back to them Did I tell you they threw away the boxes? Sorry what? Oh my god. They're not behind me How are we gonna deal with that? How are we gonna segue to our sponsor? Gskill's Trident Z5 Neo DDR5 memory is built for AMD Ryzen 7000 series processors and one-click memory overclocking Learn more at the link down below The culprit was this guy but not this guy entirely four of the drives specifically out of the twelve had this erratic behavior where we would fire up the whole system and They wouldn't be there then we'd replug the cables and they'd come back and we could benchmark it and test it invalid And then there and then they would be gone. We tried everything from moving those drives to different bays to servers even unplugging and replugging the cables and it just kept freaking happening and In the end it was one of the most basic troubleshooting steps in the book Have you tried unplugging it and plugging it back in but not the drive? Not the backplane or the cables the CPU because what it's hard to wrap your brain around is that these are NVMe or PCI express drives and Through the connector on the back through the backplane through the motherboard to the CPU socket They actually connect directly all the way back to your processor which means that a loosely seated CPU could actually cause a broken link in that chain and Here's the thing after the servers have been running for a few minutes. That's just enough thermal expansion for that CPU pin That's just not quite touching. Bam now it's touching boom intermittent problem Yeah, of course with the system already on it's not gonna pick those drives up again So when we were rebooting it after reseating the cables they'd show up again It makes perfect sense now, but oh my god trying to figure this out You got to think outside of the box and also deep inside the box You got it. You got it. Everybody's got it. Oh, this is not screwed on Jake. What is not this top cover? Oh, whatever. Where are we working on this? Oh, you're there. Oh sure you have a screwdriver, right? I have a highly erect server. That's for sure. Yes. I have my high quality Ratcheting magnetic screwdriver LTT store calm now if we were smart folks We might have looked at what drives they were and traced it back to what CPU it was Yeah, that's what I think you but you know What I was trying to say is we're just gonna reseat both the CPUs. Damn it. Sure Oh, I don't have a screwdriver yet. I went I put a Phillips bit. So you're just gonna have to do this all yourself That's fine crap. I don't have a torque set. You know what sometimes hex works. Oh my god. Don't do that All right, don't do that. That's awful. You're gonna strip this and then we're gonna strip it You're gonna make this a very long video. I'm not gonna strip it. You could go grab another one You can make yourself useful. I could the funny thing Oh my god Why did you even take it out just Don't clean the thermal paste My god, really? Yeah reseated baby. We should read goop it. No No place is expensive in this economy, right? I mean left Correct. Correct. Yes. No, no, I was never much good at math. Why are these so finicky? Damn it? Oh, yeah, beautiful watch the server not boot now I know right that is a very real possibility with AMD's thread ripper and epic CPUs because they're so large It is pretty easy to accidentally install them a little bit cockeyed which can cause these sorts of issues whether it's PCI Express devices having intermittent problems or RAM? For Those of you who haven't seen the previous parts of these series or who understandably forget them at this point What you're looking at here is a one petabyte of flash storage server Which is a non trivial thing to build because aside from just having enough bays To put that many SSDs in if you want to get anywhere near the full performance of these kiosk See a drive you need a ton of compute so inside each of these six one new servers here is 12 15 terabyte drives so that's for a total of 72 drives each of these CD 6r enterprise drives is capable of a whopping 5.5 gigabytes a second reads Four gigabytes a second writes all of these work together using a file system called Weka FS That's designed specifically for MDME drives to achieve Unbelievable performance of course in order to measure this performance you actually have to put some kind of load on the system That's where this guy comes in it has two 64 core epic CPUs eight of Nvidia's A100 GPUs Those are critical to generate the load and it has a whopping eight 200 gigabit per second network connections and of course those run through this 32 port 200 gigabit network switch to each of our servers down below That's a lot of high-speed connectivity, but with great performance comes great power consumption And to run this thing full bore we needed to plug extension cords into five separate 15 amp 120 volt breakers and a separate 208 volt 30 amp breaker And did I mention they're overclocked? I mean I feel like I could tell from all the heat coming off of it It is flippin warm back here. It's very uncomfortable to stand here Another thing we ran into last time just to get the array started these servers were so out of sync just from shipping like their Time the time on the servers was sort of sick the real time clock You know it could be five ten seconds off Whatever that the array would not start and the easiest way to get them to sync up again is NTP network time protocol And of course without networking wasn't set up to actually connect to know the internet all of these have static IPs that are assigned So in our router I had to create a VLAN that has the same subnet information And route it from the server room all the way over here to here and plug it into the switch Had to go in the switch and tell it oh 10 gig is fine It doesn't have to be 200 gig and then they turned on no problem But I want to access the IPM is so we can see the power consumption sure so we've got to plug this bad boy into All the IPM I ports really quick. Oh, okay, so we need some patch cables cables are written the fact. Let's do it Oh, that's wild. Yeah, the management port for this one is actually on the front. Oh man It's so much nicer in here. I want to see all 72 drives is what I want to see you might be asking for a lot No, no Bare minimum when those four drives are missing the array just rebuilds. Oh god. Just give a second. Oh sick. It's working. Okay Oh the cool thing though when those four drives are missing the array like rebuilt itself like it was nothing It takes like five minutes because they're so fast Yeah, but we want the capacity. We want the whole point a petabyte of flash Okay, I hate to burst your bubble for some reason they only configured it with a 500 terabyte like drive only 500 terabytes I think we can expand it later, but for the purposes of the first demo I think let's just leave it. I don't want to break it. I'm ssaging into that the node Oh look at that. How many is that one two, three, four, five six seven eight good. That's the correct number That's a lot of GPU. It's a lot of freaking GPUs. Is there Nvidia SMI? Each of these is like Oh look at that 400 watts 3200 watts of just a GPU. We're not connecting to these things over anything like SMB because that would be way too slow We're like directly connecting with the Weka interface and we're gonna be using GPU direct storage, which is super cool We'll talk a little bit more about that later. Now this visualization is super cool. The green things I'm looking at I guess our drive and the gray things are cores or no the gray things are drives the green things are cores That makes way more sense actually. Yeah, and then blue is our network interfaces. That is a pretty neat way to visualize it Oh, yeah, you see this look it's they're pinning cores to specific drive slots It looks like there's some special sauce going on here. There is some manual ass shiz that went on in the configuration of the system Seriously though a lot of tuning because like I said before Making a one petabyte flash server easy making one that performs Near even actually we're still not gonna be hitting the peak You'll remember when we did the honey badger server. We got like a hundred gigs a second Yeah, that was directly writing to individual drives There's no file system. No, no raid parody networking. Nothing was going on Yeah, this is a usable file system across six servers. All right, you ready to see some big numbers? Yes, this one's not as exciting as you'd expect you have to you run it and then it shows you it like gives you a file With the numbers that's not that fast. It's not very fast. Let's give it a sec. Oh It's faster. You said this was gonna be cool. It's trying different sizes This is probably like 4k right now. The script is starting with GDS or GPU direct storage It sounds very complicated and realistically in practice setting it up was probably very complicated for somebody But the main difference is rather than taking the data from those NVMe drives putting them into the CPUs memory Yeah, and then into the GPUs memory. We're skipping that middle step It goes right from NVMe to the GPUs memory. Oh, there we go. 15 gigs a second. Was it CPU first then? No, it's doing GDS. But like I said, it's doing different block sizes. You ready to see some big numbers? Okay, so this is GDS Says GPU direct storage only read tests 106 Gibby bytes per second. So that's like a hundred and fourteen gigabytes per second Over a network with a file system. It's like two blu-rays per second like like full quality big boy. Like the whole thing per second Holy sh** and again, we've seen numbers like this before but not with a file system loaded on it We could take that you know, we could actually use this you want to like SMB for a second here I mean, well, we don't have a client that's anywhere near fast enough Which I guess is a perfect opportunity for us to talk about what this would be for Want to show us that visualization demo? Humanity's desire to look beyond the stars is nothing new It's not and one of the major stepping stones in the space travel is finding a way to transport people In all seriousness though sending people to another planet is an incredibly complex challenge, but one that modern computing can Help us with and one of the current proposed plans from NASA involves sending a six-person manned craft all the way to Mars After the ten-month journey the crew would transfer into a landing pod that is 16 meters in diameter So roughly the size of a two-story house due to the sheer size of the lander as well as the thin atmosphere on Mars It's not possible to use a parachute to slow down like they have on past landings like the Perseverance Rover So instead NASA wants to use retro propulsion Otherwise known as creating thrust in the opposite direction to slow the lander down from 12,000 miles an hour To let you know the ground in less than seven minutes Simulating this landing is a very important part of the research and development for the project and with the power of the summit supercomputers 15,000 NVIDIA GPUs over the course of a week. They were able to build a 100 plus Terabyte model of the landing that is over 1 billion points each with seven numerical values density vorticity pressure that sort of thing and I guess we have NASA security clearance now because we have that exact Model loaded up on the million dollar server and here it is and Jake's got it running right behind me now in the past engineers would actually take this data model and Render it out frame by frame into a video still super cool and very useful But if you want to tweak a parameter and see how it affects the model Guess what you're waiting hours for a new render with new technology like super fast NVMe storage NVMe speed file systems like Weka FS and the power of GPU direct well now We can have the graphics cards render the data directly from the storage bypassing the CPU Allowing us to do this in basically real real time manipulation That is I mean, it's a bit cinematic But we're getting we're getting like 5 FPS 5 FPS. That's I mean a lot better hold on hold on hold on 5 FPS Each of these frames is over 14 gigabytes of data Yeah, that's before it becomes a frame the GPUs have to process that so each second We are streaming between 70 and 90 gigabytes from that storage Directly to those GPUs. So that's one to two blu-rays each second can you like Move it. Yeah, I can hear let's yeah move it play it play it trust us. This is really neat I wish there was a way for me to like translate how much is happening in the background I guess we can look at how much power it's drawing. This is what you guys are gonna want to see How many amps are going through our PDU here and that's just for this top server. Yeah, why does it need? 17 amps of 200 4000 watts right now. What the hell? Oh, hey, this is interesting Look, so one of our GPUs you can see it's refreshing every two seconds Yeah, they're kind of drawing a little more power now, but still not anywhere near 400 watts. Yeah, that's interesting I guess this is just more storage dependent when it is actual GPU rendering dependent which makes sense. That's they're pretty OP GPUs sort of why they sent this demo It's about the storage like moving it around a bit. Oh, yeah, they got a little spike here You can see the utilization on the GPUs is like 50 60 percent Probably a lot of this is more the memory usage and less the actual core itself You can imagine the type of workflow improvement you would have from being able to just Mess around with this in real time rather than like rendering it waiting three days for it to finish rendering because you're at a university And you got a small budget works just like this segue to our sponsor Vessi do you hate wet socks as much as I do? Vessi footwear makes lightweight breathable and most importantly water-resistant shoes So no more squelchy socks their Dymatex material not only keeps your feet dry But keeps them warm in the winter and cool in the summer How does that work the stretchy design shows that comfort is at the forefront at times making you forget you're wearing shoes Vessi makes cruelty free products right down to the glue their shoes are 100% vegan Whether it's a rainy city or a rocky trail the herringbone tread design is there to help stop you from slipping around Your feet deserve a little treat so go ahead and click the link below and use promo code Linus Tech tips to save 25 bucks on your first pair today If you guys enjoyed this video, maybe check out the previous parts, and I don't know maybe subscribe to float plane Maybe we'll shoot an exclusive of packing them up. No, I don't think we'll do that I won I like I gotta wonder if there's anything else we could do I thought about video editing I would love to use these for the machine vision cat squirter turret project