 Yeehaw, this is one buck and bronco of a server. Okay, that didn't work, but it doesn't matter because we have a serious problem and this, this is one serious solution. In spite of all the issues we encountered with our archival storage system dubbed the Vault, we've actually been able to recover nearly all of the data spread across the 2.4 petabytes of raw storage that made it up. Except that because of the recovery process and the sheer amount of capacity required to hold it all, that data is now scattered to the high winds across the five discrete servers that are required to hold it. Some of which are getting close to a decade old. We had no desire to return to the admittedly imperfect setup we had before, which means it is time for, ah, a mother vault. So called because this is one giant mother. The final deployment is gonna be not one, not two, but three of these, each capable of housing nearly two petabytes of hard drive based storage. Is it time to bring back, holy, did it ever go away? No, it didn't, just like our sponsor. Crucial, don't just work faster, work better. When you need to open, transfer, or download files faster, get some great RAM with less lag time and more efficiency with Crucial. Get your Crucial DDR5 RAM today at the link down below.<|tr|><|translate|> Before we get to the new stuff, we're gonna have to get you guys up to speed. Back in 2015, we built the first petabyte project or vault cluster. It was composed of two 45 drive 60 base store nator servers holding 100 and more servers. So we built a lot of servers, but we didn't have enough of a lot of power to get the best out of the system. So we built the first 50 drive 60 base store nator servers, holding 120 10 terabyte hard drives, giving us a total capacity of 1.2 petabytes raw. These were combined into a single network share using a file system called Gluster FS. A few years later, we built new vault or petabyte project two, which used 75 16 terabyte drives across two more 45 drive store nators. Since those two clusters were built several years apart, we ended up setting them up as two discrete clusters. That meant we had two separate network storage shares, which you can kind of think of as having two separate drives in your system. So if you wanna find a file and you don't know which drive it's on, chances are, you're gonna have to search through them separately. This was fine at the time, because we could usually guess which server a project was on based on how old it was. And honestly, searching just two sources is not that difficult. However, if we went to scale that up again and build a third, eventually a fourth cluster, well, it would start to get pretty annoying, not to mention inefficient for the staff who have to use the fricking thing. The other issue is that managing even a single system takes a good amount of time. And every time we scale it up in this manner, it's another set of components, CPU, motherboard, RAM, power supplies. Oh my God, this is really heavy. That's the heavy side. It's 130 kilograms? I think that's when it's full. Holy crap. You ready? Oh my God. I told you, you had the heavy side. Yeah, but what the hell? Anyway, more hardware means more potential issues, more potential service. So with that in mind, we've been looking for an alternative solution for some time. And while we could have continued to use GlusterFS and expand the existing clusters, we just don't really have anyone on staff who is familiar with it. And it's not that user-friendly. I mean, heck, even 45 Drives, the company who provided us with the original Petabyte project have actually switched primarily to a different cluster solution called Ceph. Now, Ceph is cool, really cool. We use it for flow plane, for example, and it's a bit better for scaling up, but it shares a lot of the same problems of our old setup. We don't have a ton of people on staff who are experienced with it, and it would have the same or similar maintenance requirements. So instead, we've opted to switch to a configuration that many home labs and data centers adore. One I personally have actually never used, the J-Bod. Oh, you can fit so much hard drive in here. This magnificent beast is the Supermicro 947HE1C-42K05JBOD. It's a 90 bay chassis that as the name J-Bod implies is quite literally just a bunch of disks. There's actually no computer in there, which raises the question, what is here in the back where the compute module or multiple compute modules would normally be? This is it. This is what passes for the computer in this chassis. And there is some computing going on here. We've got some kind of ASIC that is, I mean, is this an FPGA even? I'm not sure, it would be for SAS though. Yeah, it could be. Are they doing enough? Oh. Oh gosh. Let's, you don't want to break it. No, I don't want to break it, I just want to see it. There's a button cell battery in there. This chassis does have IPMI, so that's what the battery would be for, keeping track of time. So there's going to be some compute for that probably on the bottom board, but the rest of it should just be SAS stuff. There's that IPMI management port. And then the only other IO on this machine is these six mini SAS external ports. What are these? 88? So each of these is four lanes of 12 gigabit SAS. So that's 48 gig per port. And you can use up to three at one time for connection to the head server, which is what is it 144 gigabit. Okay. Casual. Perfect then. Oh, you can see the A-speed chip for the IPMI right there. Is it in there? There'll be some non-volatile memory on there too, somewhere. Might actually be this. That looks like a NAND package. These are some Chungus cooling fans. Look how thick they are. There's two in there, right? 80 millimeter fans? I actually don't think so. It's dual blade, but no, I think these are individual fans, Jake. Oh no, it's got to be two. It's got to be two. It's got two discrete connectors. Either way, I guarantee you they absolutely rip. And then aside from, I guess these are probably the SAS and that would be your power. That's pretty much it. Very, very simple. And that's with intention. It's designed to be reliable, right? Our other module here is, well, it doesn't have a whole lot going on to be perfectly honest with you. What on earth does this do? You've got a small controller board that has a couple SATA ports, a couple NVMe ports, and more power and whatever the crap this is. This is probably PCIe. I think when you buy the computer version of this, these two NVMe's would come over here somewhere if you just had a single compute node. But I'm not really sure. You could have a JBOD that's something called dual path. Now this is a single path, which means there's one path for the data to go from the drives to whatever computer it's plugged into because remember, there's no computer in here. It's got to plug into a different computer. But in a dual path, there's two connections and you'll actually have redundant SAS expanders, which is the part that splits into multiple drives. You'll have redundant connections. So there'll be six more back here and you'll have another IPMI. And the reason for that is you can have a high availability controller. So you could have controller one and controller two. And if controller one had problems, it would just fail over to controller two and you'd have better availability. The problem is that in order to take advantage of that, we need to use SAS drives. And that's a hassle. Well, it's not only is it a hassle, but we already have 3.6 petabytes of SATA drives. That's a hassle. That's like $150,000 hassle. Yeah. Not doing it. Now let's take a look at the SAS expanders that make this magic happen. Naturally, they're hot-swappable because servers. And in a nutshell, each of these takes in four of those SAS 12 gigabit per second lanes and then is able to split that bandwidth to up to 30 flipping drives. So we only actually need three of these to populate the entire 90 bay capacity of this J-Bog. By the way, Jake was talking about how you could have a dual path. Well, you'd need three more expanders then, but we don't need them. Jake, do you have any idea what these connectors are called? I have no idea. Shout out them, because these things are flipping cool. Look at the way these are joined back onto the board. That is dense with two S's. That's, oh my God, the pins on the bottom. Right? Holy crap. The next thing I need, Jake, is for you to get me the head. It's right there. Right here. You can think of your J-Bot kind of like an external hard drive, or in this case, many, many external hard drives, which means that as Jake alluded to before, we need to have a computer somewhere. Well, that is where the head or the controller server comes into play. This particular machine is a placeholder. Super Micro's actually sending us over a dual AMD Epic Milan server that's a little more tailored for this kind of application. So instead of being full of NVMe drive bays in the front, it's got its PCIe lanes allocated to allow for lots of HBAs and network connectivity in the back, because hard drives may be slow, but when you hook up enough of them, you can be pushing some serious freaking data. And this is gonna be connected to up to 270 drives. Not 255, I think that's what it works out to. 90 times three is 200. We don't have 90. Oh, but yeah, I mean, you can say up to. You could add more drives. It's just J-Bot. So cool, right? In the meantime, for the purposes of our demo, oh, we've loaded this machine up with three of these Broadcom HBAs. These things actually contain very little logic compared to RAID cards, which had to perform parity calculations on the data running through them. These do almost nothing. They just take your bandwidth from your PCIe slot, in this case, PCIe 8X Gen4 slot, run it through a SAS controller, and then that breaks out into four, eight, well, 16 SAS connections. Do you wanna see my wonderful cardboard? Yeah. Do you wanna see my wonderful car? Do you wanna see my wonderful cardboard? This math actually makes a lot of sense. If you have 90 drives, and we'll say optimistically, they're running at 250 megabytes a second, that's 22 and a half gigabytes a second. It's pretty fast. Conveniently, three sets of 48 gigabit SAS cables is around 18 gigabytes a second. And even more conveniently, PCIe Gen4 by 8, it's roughly 16 gigabytes a second in either direction. So we're gonna be giving up some of the theoretical maximum performance of our drives, but given that this is only under perfect conditions, with brand new high-performance drives, reading and writing sequential data. With nothing on them. This'll be lots. Yeah. And that's times three, because we're gonna have three JBODs. We're never gonna get anywhere close to this. Maybe one gigabyte a second. I'd like to point out something else that's convenient. What? Check this out. See that ratchet back pressure? Wow. I can start a screw with it. Whoa. And in spite of the silver shaft, look at that, I used it and my dick didn't fall off. I actually think the silver shaft looks better. Wow. I'm right. Whoa. He's actually been one of the bigger skeptics internally about this whole project. I'm not a skeptic. I'm just a little bit critical. But that ratchet. I want it to be good, okay? You can sign up for a notification when it comes in stock, lttstore.com. I'm trying to sound less stressed about it. In the meantime, we also have shirts and stuff. Cash flows a little tight. Now you might look at this machine and think, that's ludicrous overkill. Surely the one that Super Micro is gonna send is a little more pedestrian. But that's not actually the case. Even though we're not running super high-speed NVMe drives, when you're hooking up hundreds of hard drives. With the potential to expand to hundreds of more. You need CPU and RAM galore. More than you'd think. In fact, we actually have two 32 core Epic 75 F3 processors. Those are the frequency oriented 280 watt per CPUs. And we're gonna have around a terabyte of RAM with the potential to expand that down the road because we will be using ZFS. And ZFS greatly benefits from RAM for re-caching. We'll also have NVMe drives and stuff, but that's TBD. You'll have to get subscribed for the next video when we actually deploy the whole thing. Are we doing tiering? No. Oh, well maybe one day. It's possible that you know a secret. We might build a highly available Wannock and set this up as a tiered archival layer. Entirely transparent to the user. So just one drive letter? One drive. Oh my God, these are just drive sleds. I thought these were boxes of hard drives, Jake. No, those are the trays. I've been deceived. Do you wanna see what they tray like? We were talking about how we wanted to discuss. Yeah, we also need the high performance CPU for ZFS compression. I was like, oh, well we'll just dump a bunch of drives in here and then we'll talk about, okay, we've got this much capacity, but we're gonna use compression. And then Jake's like, oh, we've only got 13 drives to put in it for the demo. Come on, Jake. It's almost like all our hard drives are deployed holding data temporarily from the old vault. Yeah, almost. Okay, well I guess it's 13 drive demo. That seems lucky. If you want Ed to yell at you, we could take apart Delta 5. That's got new vault on it, I think. And some old vault. When Ed yells at me. Wendel's still working on Delta 3, so that's not available. Temp vault. We could do temp vault, but then Ed would also yell at you probably. It's not fit. 13 drive seems fine. All right, we found another solution to some test hard drives. This here Stornator was full of 60 Spinarinos. And now they're in here. It has made the balance of this system super sketchy. Like, oh, it gets worse. Look, look at that. Look, it's, oh, oh. Oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, God, oh, I can't reach them. Okay, there we go. But there you go, there's 60 hard drives. They're kind of in a weird orientation because there's a kind of a poop mix of drives in here. There's some tens, there's some 12s, there's some 16s. I think there's even 120. So I just wanted to kind of separate those, but this is our test. Now these 60 drives do already have an array on them. And I think it's about 70% full. This is some of our existing archival data, but they're gonna be set up as four V-Devs that are arrayed Z2, 15 drives wide. Not great for performance. I don't think we're gonna see anything crazy in terms of numbers out of this machine, but it is kind of indicative of what we're gonna have it set up as. We're probably gonna switch to 10 drive braid Z2s just to make it a little bit faster, but this should still be pretty dang fast. Now in order to get a connection between our server and our JBOD, we have these cool mini-SAS HD external cables from Infinite Cables. I think SuperMicro also sent us some. By default, this JBOD's configured as a single zone. Now you can do kind of cool stuff with zoning in a JBOD. For instance, you can split this up into two zones and have two controller servers being serviced by one JBOD. So you'd have 45 drives on either. You can also do three zones. So that'd be 30, 30, 30. Do three controller servers, but they can't access the other zone's data. You'd only get the zone that you're actually physically attached to. So you'd have A, C, or E on the back of this server. Now in our case, we're gonna be using three zones, but to one controller server. And the reason for this is it allows us, instead of getting one cable, which is 48 gigabit of bandwidth, we now get two and then three for 144 gigabit, which again is probably overkill, but what do we do that's not overkill? Hi, you might need these, like for realsies. No, no, just wait. This is just the power supplies? That's the power supplies. That's like full tilt though. Whoa, this is ludicrously loud. So I put it in the server room with the door shut. You can hear it in the bedroom. But this is like, they're actually at 100% right now. This has never run at that speed. Never. And with the 60 drives, I already tried it, like running a load and they barely even spin up. Okay, that's good. Yeah, I got a little worried. Is it the air conditioning ever fails in the server room? We'll know, because we'll hear this everywhere. There we go. Give it a sec. Okay. Coming back to planet earth. That's a lot more reasonable. I mean, I guess that's what fans that thick will do for you. Well, I mean, when you have 90 hard drives, you're talking like at least a thousand watts. Yeah. That's a lot of power. I mean, what are these power supplies? They're 2000 watt on 208 volt. 2000 watt power supplies? So to have like actual power supply redundancy, you need to be running 208 volt. Wow. Otherwise you'll probably use enough power that it needs to split across the two. So we might actually have to rewire our server room to be 208 volt, because for some reason, it wasn't wired that way in the first place. I don't remember why Brian told me. There's probably a reason. Yeah, there's a reason. And I bet you there's a way to fix it. Oh my God. Yeah, you can fix anything with money. You have lots of that right now, right? No. Oh, well that's fine. We'll figure it out. You know what? No, we should just put these in lab two. Cause that one will be 208 volt right out of the gate. It might be too loud though. Time to put more insulation on the walls. Jesus. With nails. Yeah, I'll show you that quick config here. This is where you can change the zone. So there's single zone, two zone or three zone. Again, like I explained before, you could use that for multiple controllers or to increase throughput. We're doing the throughput route, of course. That's pretty much all you can do in here. There's no remote control. Like there's no, there's no screen. It doesn't have a display output. Right, yeah. Power. It's just a big, dumb, big discs. System critical. I don't know why, probably because you pulled the power spot out. Probably. Plug it back in. I'm sorry about that buddy. Here you go. Yeah, there's your medicine right in the butt. Suppository medicine, power cable. Now tell me something. If we decided to go high availability in the future, nothing would actually prevent us from adding a second interface unit back there. Yes. A second head server. Yes. No problem, right? We need SAS drives. Sorry, not high availability at the, oh, cause dual path. Oh crap, okay, yeah. The thing about this approach though is we've already proven we don't really depend on this. Yeah. It's been like, it's gotta be like eight months at this point that we haven't had it. It's funny how many people were listening to us in the video saying, yeah, it's really optional retaining this data. We lost it because we don't really care that much. And they were like, lies. Like, no, we haven't had it for like almost a year at this point and it's been totally fine. It is nice to have, but here's the thing. If this controller server were to poop the bed or to have some problem. Well, everything here should still be fine. We can just unplug this. We'll have some other server temporarily. So like a cold spare. Put the HP, yeah, basically. A cold spare server. I mean, you're talking maybe an hour of downtime, like realistically. I was looking at this and I thought maybe this was, I saw four VDEVs. I thought that was the four drives. I just saw it out of the corner of my eye and I thought you had put the 30 terabyte NVMe's in here. We might though. Wendell's been talking about the special metadata device for a while. And basically he was saying he has like half a petabyte with 24 terabytes of special metadata device in ZFS and it fills. So I'm like, we might as well just put four 30 terabytes in there and that'll give us 60 terabytes. What does special metadata do? So you know like file metadata and like the directory structure. Does that make searching really fast? It makes searching way faster. That's cool. So it stores metadata like where files are. And he's talking about it right here. It's literally Wendell's blog post. First thing that comes up. Hi Wendell. But here's the thing is we can have a level two arc. We can have a log device and we can have special metadata. Yeah. So he's saying 172 terabytes of space is five terabytes of metadata. But here's the thing. We don't have to have all of it metadata in there. Like it'll pull it when it needs it, right? Yeah, by default this includes all the metadata, the indirect blocks of user data and any deduplication tables stored on that device. So maybe deduplication. If you are running deduplication, this is probably really important to improving performance. Yeah. But either way, we'll try it. As you can see, it's about 70% full. Not ideal for performance. And on top of that, it's an existing pool. It's set up in four VDEPs, right? So they're 15 wide. So really I'm expecting like maybe three to five gigabytes a second. Okay. Sequential. Like I would not be surprised if that's where we end up. Because when we move to the final deployment, we're gonna do 10 wide, which will be better for performance. We'll do 10 wide RAID Z2s, which is basically the same as 15 wide RAID Z3s. Yeah. And then we'll obviously have a lot more drives. If you've got 10 drives of which you could lose two of them before you have any data loss, that's kind of the same as having 15 drives of which you could lose three in terms of the ratio of- Like the whole pool, if you look at it from a wide perspective. But it's probably slightly better to have more VDEPs if you run the probability analysis. Anyway, so this- No. No, no, no. Okay, we gotta explain. So this reset is set up with caching only metadata, which is what we want. The RAM cache is turned off. Oh, that's give it a second. There we go. We're getting like, let's just give it, give it, you know, just warm up there, bud. Come on. Okay. That's about two gigabytes a second. Come on, you can do better than that. I didn't even look at- I don't think it can. Come on, three. Three. Yeah, it's not three sustained. That's for sure. 3.6. You can't just read the high numbers. I'm reading just the high numbers. That was 3.7 there for a second. Yeah. You know, it really doesn't work like that. That's 579. Well, there's 4,000 for a second there. Yeah, but there's 0.3. Probably has something to do with the fact that the drives are pretty full already. Remember, we have no NVMe caching. We have no RAM caching. Like this is raw dog, giant V-Dev, just disk. Now tell me something. Hold on a second. Wait, here's something I don't fully understand. This is really cool for maintenance. The fact that you can have the system running. You don't have to pull all your cables out at the back in order to slide the drives out. Yeah, the cable staying in the same spot. Super nice. But how is all of this still connected? These drives are blinking. These are doing things. Oh, it's just like a ribbon. Yeah, giant ribbon cable in the back. You see it back there? Oh, that's super cool. Man, that's amazing for maintenance. So you can just go be like, okay, yeah, I've got a bad drive in Bay, whatever. Pull this out. You don't have to slide any servers out. See you later. Cause this is internal, right? You just pull it out and you grab that one. We're gonna have to label them. Oh yeah. Or I might just do a spreadsheet, honestly. Yeah, that would be fine. If they were SAS drives, you have like SAS enclosure service or something like that. SES, whatever. Anyways, you can go and be like, what's in that slot or by drive, you can say, what slot is that drive in? And it'll tell you. But since they're SATA drives, they all just show it's not connected. Let's put this back in. This is making me anxious. Let's do that. I'm like, I just. Oh boy. Okay. Yeah. So we're doing 16 jobs in IO depth of 16 block size one man. It's gonna be probably basically the same. Oh, 2000. Woo. Blow my skirt up. More consistent though. Yeah. Like nope. There it goes. See you later. In the grand scheme of things, we're gonna be writing to this thing, like basically peak one gigabyte a second ever. And it will be better when there is more V-diffs. Okay. Then I think we're good. Good to tell you about our sponsor. Vessi. Vessi proves that waterproof shoes don't need to be ugly or uncomfortable. Thanks to their DymaTex technology, you can keep those toes dry while remaining sleek, snug and stylish. Need even more comfort and breathability? The everyday move line has added padding at the midsole and a looser knit to keep up with your busy active lifestyle. Do you hate tying laces, tired of chasing the rabbit around the tree and down the hole? Well, fear not because Vessi's everyday move comes with handy pull tabs for easy slip-on action and are made 100% creature free. So you can be walking around on fluffy clouds with a clear conscience all day long. And when Vessi says everyday move, they mean it hot, cold, wet, dry, stay comfortable in any weather. Just think of all the freedom you can get with your new shoes. So treat those feet of yours to Vessi everyday move shoes and save $25 with offer code LinusTechTips at Vessi.com slash LinusTechTips. If you guys enjoyed this video, you might also enjoy the one where we explain what happened to the old vault and how we lost nearly all of the data on it. What? We got it all back. Well, yeah, I know. But first we lost it. Well, like sort of.