 It's all here, look at this. It's been almost a year since we discovered the major issue that effectively put our entire video archive server offline. That's right, all the full quality footage since I was still working at NCIX gone, temporarily. After the painstaking process of recovering nearly all that data, love you Wendell from Level One Text, we finally have all the hardware in place. So today's the day we get our vault back up and running. We're talking sprinting, we're talking better than before. We're talking an entire 3.6 petabyte array managed by one single head server instead of the five servers that we have now. And I promise it's gonna be awesome. Like the segue to our sponsor. iFixit, looking to expand your selection of tools? iFixit has you covered. Learn more about the Manta Toolkit and it's 12, what? 112 steel bits, what? With two premium drivers, learn all about it at the end of this video, Manta. This is the one computer that is going to run the mother vault. And the astute among you have probably already realized that we are not going to be able to put 3.6 petabytes worth of storage inside it unless it's from Hogwarts. It's magic. Wingardium Leviosa in here. No, it's Wingardium Leviosa. Oh, sorry, excuse me. No, in fact, there are no hard drives in here. This is where the magic happens. The machine you saw just now, that's only the brain. So instead of trying to put 3.6 petabytes of hard drives into there, they're gonna go in separate enclosures like this one called JBODs or just a bunch of disks. And you can think of an external JBOD enclosure kind of like an external hard drive dock, except instead of one hard drive, they hold up to a whopping 90 hard drives each. We'll talk more about these a little bit later. Just because the 2024 US-TRT from Supermicro is shirking its storage responsibilities doesn't mean that it's not packing. Inside here, we find a dual socket AMD Epic Milan system that honestly has a bit of an unusual configuration. Typically, a system with this much CPU horsepower. So those are two 24-core AMD Epic 74F3s. And the ability to run up to dual 64 cores would be decked out with a ton of NVMe slots or some enterprise GPUs or something like that. But this guy only has four NVMe slots. And these are an optional accessory. The rest of the storage up here, and in fact, these four slots by default are, I kid you not, six gigabit SATA, three and a half inch slots, like that you would use for a hard drive. That's because this system is actually designed with a different purpose in mind. Say for example, as a virtualization server where you might need a ton of CPU that's gonna be split up to different clients, or if you're like us, a ton of CPU that you're gonna use to drive a boatload of connected storage that isn't inside the chassis. So instead of using up all those express lanes from those Epic CPUs, which is, buddy, 128 lanes each with like NVMe drives in the front, we instead want them allocated to PCIe slots in the back of the server. Back there, we could install high-end networking adapters. What is this? Dual 100 gigabit, this guy right here? Woo! Or the other piece of our puzzle, host bus adapters. Just like with an external hard drive, our JBODs back there, which the keen eye among you will have noticed that there are too, require some kind of connection to the actual server with the actual CPU in it and the actual operating system running on it. In a modern JBOD, this is usually handled by mini-SAS HD cables like these ones from Infinite Cables. We're gonna have these linked down below. So each of these ports on the back of our three cards can handle four lanes of 12 gigabit SAS. That's the enterprise big brother of SATA for a total of 48 gigabits per second per cable. Your average computer, even server, probably doesn't have any factory mini-SAS HD ports. So we're gonna be adding them to this machine with three of Broadcom's 9,500 16E host bus adapters. That means we'll have a full dedicated card for each of our JBODs. Oh yeah, there are three of these? Right, there's one more JBOD that's actually already in the server room. We will show you guys that one later. Now each of our JBODs allows us to connect up to three cables for increased throughput. So that's exactly what we're going to do on the subject of throughput. To push this thing to the absolute limit, we want to pack this machine with as much RAM as possible since the file system that we're using called ZFS can use RAM as a read caching layer to significantly improve the responsiveness of the array because mechanical drives on their own, yeah, even with this many of them, not that great in terms of access times. Luckily, the folks over at Micron just sent over one terabyte of 3,200 mega transfer per second registered ECC memory, which last time I checked, smells like performance. But the performance improvements don't stop there. We had some leftover Keoksia 30 terabyte NBME drives from the Dream server build, sorry Dream, and we figured they would be the perfect choice to configure as a special metadata device in our ZFS pool. This is a game changer for performance because it allows metadata like where the files are physically stored on those hundreds of hard drives, the allocation tables, and optionally, even small complete files to be stored on a discrete higher performance device like these NBME drives. This can make navigating through large data structures like the thousands of video projects that we're archiving much, much faster. And that directly results in workflow improvements for our staff. I'd say it's about as much of a quality of life improvement as our flow plane subscribers get with no baked in ads, higher quality streaming, and exclusive behind the scenes content. We're gonna be setting these up in a mirrored stripe, leaving us with a total of 60 terabytes of capacity, which is hilarious, given that it's a RAID 10 of just four drives, which we think should be an ample amount of fast metadata storage for our 3.6 petabyte array. Unfortunately, because we only have these four NBME slots, we're not gonna be able to expand that storage unless higher capacity drives come along in the future. And the performance compared to some of our other all NBME arrays is not going to be that amazing. Another problem is that this doesn't leave us any additional space for NBME storage drives for a level two arc read cache or an intent log, at least at the front. But there's a solution to that. Ah, yes. This cute little half height PCIe card from Liquid allows us to install four M.2 drives in a single slot inside the server. And this will be absolutely perfect for that role. What's the total capacity of this bad boy? I don't actually know what SSDs are in there. We don't know what's in it. There is one we have that is four four terabyte SSDs. Okay, so it might be this one. It could be this one. It could also not be this one. Great. It's a problem for later Jake. This is pretty cool Jake. Did you design this three and a half to two and a half inch adapter thing? No, we found it on Thingiverse. Oh, cool. Dan printed it. Shout out Thingiverse. Whoa! What? Well that came out. Which slot did you want the Liquid card in? You gotta change this guy and then put it right here. Oh man, can we find out if it's the right one before we do this? See, we're gonna have to pull the whole thing apart. What do you mean what's the right one? The high capacity one. Oh, no. Oh, come on, there's gotta be a test bench not far. It's a problem for later Jake. Oh, don't, it's like glue. No, there's none over there. There's none over there. Jake says there is none. Well, let's just go to my test bench upstairs. Logistics. I think it's important for you guys to know that Jake thinks you guys are utterly incompetent. That's what he said. Whoa! Those are his words, not mine. He said those guys don't even have a test bench over there at all, not one. What is this? Yeah, that's what I thought. Yeah. Yeah, Jake. Near minutes later. All right. Hey, we got a foreign dynamic disk here. These are what, three terabytes each? What the hell is this? Up, up, up, up, up, up, up, up, up, up, up, up, up. Okay, right click on that one. Let's see if it's. Toshiba three terabytes, 84. Yeah, perfect. These are the four terabyte ones. Thank you, fast and competent logistics team. I just want you guys to know that I defended you. I was the first to say no. I would never use those words about our logistics team. Not like Jake. Thank you. Goodbye. Get out of here. Ah, here's what we've got next. Now I mentioned that this was 100 gigabit, but what I didn't say, I don't think, is that it has dual 100 gigabit ports. It can do a total of 200 gigabit. It's PCIe Gen 4 by 16. That is a flipping ton of bandwidth. And it uses these sick QSFP 28 connectors. While Jake orders his lunch, I'm gonna go ahead and get our other two HBAs installed here. I'm getting mucho burrito. Editor, you better put this in the goddamn cut. No. Okay, I'm just gonna put this in because you're taking too long. What? What do you mean I'm taking too long? I'm not the one sitting here ordering lunch. Touche. This is amazing. So we've got our three HBAs. We've got our 100 gig network card. We've got our additional SSD, which is somewhere in the neighborhood of around 15 terabytes of storage for our slog and level two arc. Well, there's 3.84 each. Oh, I see. So like somewhere in that 15 point something. Whoa. And how spicy is that? You forgot about these. You forgot about these. What'd I forget about? The 120 terabytes of NVMe in the front. Oh yeah, those. Yeah. Are we ready to turn it on? I'm turned on. Okay. I could be more turned on if I was shilling for lttstore.com. Hey, check out the RGB diode shirt. It's like based on what the individual sub pixels look like, but then it's also just a cool design. Is that, do you want the top? Yeah. You can do a peel. Very graceful, Jake. You're doing a fa- Oh! Oh! Oh! That's right, I'm a monster. This thing is sick. Do we get to power it on now? Got a laptop? Got a laptop. Uh, hi? Jeez. This one is, just wait, it's actually really quiet. Yeah. Give it a little bit of time. Are you gas lighting me, sir? No, I would never. BMW is a reliable. Sir, this BMW is very reliable. Damn it, he beat me to it. Oh, that is so much better. And can we turn off that alarm as well? I have a solution. I don't think that's going to work. It does. Pulling the power supply outright makes the alarm for a di- Oh wow, it did work. This is a 1600 watt power supply. What? I mean, to be clear, that's only a 200 volt input. It's a thousand watts if you're from 100, 227 volts. Woo! That's what you wanted to do. I want to go fast! You said it was quiet, so I thought full fan speed must not be that bad, right? I had him log me separately into the IPMI just so I could do that. In other news, it's booted. We got our terabyte of RAM. Ooh. We got our, oh yeah, we got our Mellanox card in there, that's good. Did you put the NVMe's in the right slot? I mean- One of them is wrong. What do you mean, wrong? Oh, it's wrong. Try the bottom. Wait, wait, whoa, whoa, whoa, whoa. One of those is a boot drive. It might be good. There was an activity flash. Yeah, you're right. Okay. It's cool. And look, because the boot device is a pool, it's mint. Just immediately fine. I mean, it's probably mad. Actually, this is going to be kind of complicated because some of the drives we need to put in the right spot and not mix up because some of the arrays were too close. And some of the drives are in another J-Bot that Wendel's working on right now. So we can't really touch those ones. Uh-huh. It's going to be fun. Let's go. Start carrying. How am I supposed to move this? This is a four man carry. It literally has a label on it that says four people need to carry this. Does it actually? Yeah. Dude, these are like the best earmuffs. Put them on. Oh wow. Now with 80% less Jake. I can endorse these. We don't actually have to take all of these out. We could just put them in the right slot. All of these out, we could just take this one and that one and then just move this. So the head server will just go here and then we'll go one, two, three. We don't even have to move this one. So we just have to take the temp vault out. Should I let the editors know temp vault's going away? I already told them it was going to happen today. I think you can tell them it's happening now. I'll tell them it's happening now. Hello, hardworking and truly valued editing team. The vault that you have come to know and love, temp vault is now dead vault. Press F to pay respects. They don't care at all. All right. Bye. It kind of raises the question. Do they need a vault at all then? Why don't we just have no vault then? Forget it. If you guys aren't even going to miss it then I- What do you mean you copied the files? You guys aren't supposed to keep things on your local machines. We copied it to the active server. New, new on it. New, new on it. Hopefully this clears everything up and is not confusing at all. I think they're more confused. Look, somebody is reading files from it right now. Like Ed was lying. What the hell? Look, somebody's reading from it or they're writing to it. What are they doing? Oh my God. At 350 megabytes a second. This is a not insignificant data transfer. What are they- Look, look, look, look too. The chart's like nothing. And then somebody's like, whoo. You want to shut it off now? Okay, who's messing with the server? Are you messing with us, Ed? Somebody is vaulting right now. Did you copy things to the vault right now? I didn't. Probationary employee number 17. You cut and paste it? Terra copy. But it's still going. You can still kind of paste with Terra copy. You can paste or copy and paste. Dang it, guys. It's up. We're good, you're good. God damn it. Ignore the disks with errors. What? We've lost a few. The original vault's 10 terabyte drives are still like in production and they're like six years old now. And we've had problems too. Our cooling in that room didn't used to be super stable. There were times when you would walk in there and it was like, phew, wall of heat. Because we had this air conditioning unit that would just leak refrigerant. When we didn't have an air conditioning unit. And fail. Yeah, yeah, at one point. I remember when I started here, and I was just like, no, we don't need it, we don't need it. And then all the UPS has exploded because it was too hot. And then he was like, okay, fine. How are we gonna move these? Fortunately, these aren't really rails. They're actually like shelves. So all we really need to do is kind of put it on the ground and then just tilt one end and then slide it in and then tilt the other end. That's how we put the other one in there. Installation by two persons is recommended. If these need two people, that server needs 25. Think I can handle it. I wonder where my burrito's gonna arrive. All the drives are gonna have to come out of this thing before we lift it out. No, no, no, that's fine. How are we supposed to lift this? We can lift it, we've done it. Okay, well how? Strong. No, really though. It's not off the ground. I just use my feet like this usually. Your feet. Yeah, and then you just walk it out. Walk it out. Yeah. Three, two, I'm holding the heavy side. Yeah, yeah. Don't drop it. I didn't drop it. Ow, you just put all the weight on my foot. I didn't know it was on your foot. Cool. Well, that's all the hard work. Yeah, now that we've got Lab 2 over there, we actually have a purpose for all these additional store-neaters. Oh man, I haven't gone spelunking in the server rack in ages, Jake. There's so much pressure in here. You're giving everyone quite the view right now. Whatever they like to. Usually you gotta pay extra for that. Onlyplains.com. How do you like this setup? I'm taking a little server cabinet nap here. Got my little pad on the back of my head. Okay, bottom of the fifth. Not bad. This was definitely the best way to do this. The front of this shelf is out? Yeah, that's what it appears. That's why we screwed them in. How full is this? It's got 60 drives in it. Okay. What seems to have happened with this server, because we didn't screw the shelf in from the front, is this little tooth here that holds it into place seems to have slipped. So it's sagging down like that, which means that a ton of this weight is being borne by a part that's really not designed for that. Complicates our operation here a little bit, because we can't just pop it back into place from here. We can't really get at it. Looks like we need to shut this down, pull it off, put it back into place, screw it in, and then put it back in, which sucks. If I take the weight and you slide it forward or the other way around. Here, I'll use my leg. Is it up? There we go, it's up. Just put the rail in. Don't move. Wait, put what rail in? Put it in. Oh, I can't see it. Oh my God. Oh, I think we got it. I think we got it. Let go, let go. Yeah. This actually went pretty okay. Situations like this are when I'm so glad we developed this screwdriver. I got the nice strong magnet holding it in place, almost no back force required to spin the ratchet. Like, easy. Is it in there? Yeah, it's in. Oh my God. Wait, how are we supposed to get it from here down to there? We got it up there somehow, so. It's just like a really heavy storinator. Why don't we just put the other one on top? That says this cart can take 600 pounds. No, we're not doing that. Okay. I tried. What are the odds? That it just goes right in. No, the angle's still too much. Okay, I'm getting under it, boys. Yeah, but- Put the right side in a bit more. Okay, good, good, good. Oh my God, it's it. Yeah, it's on, it's on, it's on, it's on. Okay, go ahead. You can push it in. Hi. All right. Now for the hard one. Yeah. This is more of a Linus sized server. Yolo! Well, it's easy to Yolo when somebody else already did all the work. Yeah. Good work, Jake. There goes the brain. And what's really wild is that if we want more capacity, more than 90, 180, 270 drives, with the cards we have in now, we could do another one. And then I think there's room for one more card, right? I would say right now with no HBAs, we could probably add two and it would still be pretty good performance. All right, let's put in the hard one. Ready? One, two, three, go! Yes! What? Weak. All right. Oh, still weak. Now that they're in there, working on these is glorious. You just press the buttons, pull them out, boom. Funny story. When we first put this in, I was like, ah, it's so heavy. It doesn't really need to be bolted down, does it? I was told yes. In fact, it does need to be bolted down. And this is why. Because if we had a handful of these hanging off the front of it and it wasn't bolted into the concrete, woohoo! We'd have a bit of a problem. Bit of an update here. We shucked all those drives, put them in the service. I plugged a bunch of stuff in, some power, and we have blinky lights now. Do I get to press the button? Yeah, it's on the right side. Give them a sec. Everybody in the next square mile is gonna hear this. They're just, they're spinning up, boys. Wow, it's actually not as bad in here. Come outside. Let's shut the door. Yeah, sure. You can hear how awful it is still through the door. Oh yeah, we wouldn't be able to shoot with those things going. But they don't go. Okay. That's fine. Oh, you just gotta add another layer of insulation. There's no space, Jake. Well, this is the backup plan, right? Because there's a thermostat in here that will start up a fan that exhausts out the top and will pull air passively through this grill in the event that the air conditioning completely fails. Do you remember the shroud for this thing? I made that in metal shop when I was still in high school. That's hilarious. Okay, that was it. It's up. So far only 20 drives are showing up, but Jake tried a thing, which is a what? A domain change? We changed some cables, and I changed the JBODs from three zone to one zone. Are you gonna click it? Yes! Okay. Thank God. Okay, one of these. 145 drives. God damn, brother. I mean, 45 drives is nice, but 145 drives. Now that's walking around hard drives. I'm just gonna try. This is probably a bad idea. I'm just gonna do it. I'm gonna try to import the pools. It should find both of them. There's two discrete pools on those existing drives, and then there's a bunch of extra drives. So let's see what happens. What are the odds? This is just gonna be like, yeah, your ancient ass pool and just. Well, they're not that ancient. I think they've all been recreated now. Okay. But actually, the crazy thing is, I took like 2015 pools from a version of ZFS that was not recommended for production and had not been updated, and I put it in like 20, 22 ass, true ass. It just imported. It just upgraded. For some reason, it was super slow. Like, scrubs would take like a week, which is really weird. Yeah, but there could be kind of flaky drives in there. It wasn't the drives. The drives were fine. I moved all the data off, deleted the pool, made it, and then brought it back, and it was fine. It might just be like a fragmentation thing or. Oh, let's see. Did something. And. Oh, look at that. Go fast is the 12 drives, I think. Got it. I wish I saved these IDs before I did this. Let's see what happens. Does this mean vault is just gonna be like. Well, I'm back, baby. 10th vault will be back. The read only new vault because we're still recovering a few things. That's why one of the JBODs is still MIA. We'll be back just like that. And I can just recreate new shares. I mean, the IP will be different, but I can just recreate the shares. ZFS is cool, man. Don't put that there. This is a really nice shirt, though. Wow, is that from lttstore.com? It sure is. It will be. Oh, is it not out yet? I don't know. I don't think so. Have you announced it on Wanshow? I don't think so. 30 hours later. Wow, the pool's finally imported. Holy crap. Supermicro, massive shout out. Thank you for sending over the gear. It's amazing. We did have a little bit of trouble with setting it up with three domains. Yeah, it's for some reason, they're only working in single zone mode. That means all 90 drives are going over one cable. I don't know why, because we had it set up on the bench like a couple of weeks ago with three zones. Anyway, it's accessible. We're ready to copy some files, and hopefully that's something they can sort out either through helping us set up the configuration or through firmware or whatever else. Yeah, it's probably just, we're missing something. This is one of those solutions where it's kind of like, you just know what you're doing if you're doing it. You're an integrator who sets these things up every day, week after week, and that's not us. But, we're there. I copied a file. Let's do it. Look, I'm copying a file from it to this, like hundreds of megabytes a second. It's not the most impressive speeds ever. This is a SATA SSD, and we're screen recording to it now. So it's like, But hey, remember guys, this is with ancient degraded ass fragmented V-Devs. Once we strip all the data off of them, rearrange them, reshuffle them, set them up fresh, and dump all the data back onto them, we should be able to be in the neighborhood of a gigabyte a second, which is all we're gonna have. More, for sure. Well, yeah, but that's all we're gonna have. Yeah, that's all we're gonna, it's a hundred gig. It's a hundred gig. Oh, that's right, it is. Well, here, look at that, 147 drives. Never an actual production, that's amazing. And then we'll add 60 more, and it'll be over 200. And then we'll add 50 more, and it'll be 250. And then we'll add this segue to our sponsor. iFixit's Manta kit has 112 steel bits and comes complete with standard bits like Phillips and Flathead, as well as more niche options like pentalobe, hex, and tri-point bits. It also includes two aluminum screwdriver handles, a weighty quarter inch drive for full size applications, and a smaller four millimeter for precision work. Both drivers feature magnetic bit sockets, knurled grips, and silky smooth spinning tops. You can even use the tool kit's lid as a built-in storage tray. Best of all, every iFixit tool is backed by a lifetime warranty. So if something breaks, they will replace it for free. So visit ifixit.com forward slash LTT and pick up your Manta kit today, Manta. If you guys enjoyed this video, maybe you'll enjoy the one where we tried to deploy Wannock. No, no, no, that was awful. Okay, sure, yeah, yeah, sure, go watch that. Or the previous one where we just looked at Jbots because they're cool.