 Back when we installed a petabyte worth of hard drives in our server closet, we were sure that with that much storage, we'd be good for a long time. And in fairness, I guess two years worth of red footage is pretty good. But, it finally happened. We are critically low on space. I've got less than 5% available on our main editing server, but with only 20 terabytes available on the Vault, there is nowhere to dump it to. Fortunately, I've got a Band-Aid solution. The very best kind of solution. Seagate actually sent over these 15, 12 terabyte Iron Wolf Pro drives for a totally unrelated project that, well, didn't actually go very well. So, instead of using them for that, we're going to use them to add more capacity to the Vault. Speaking of the Vault, I keep all my segues in a Vault. Ridge Wallet is the sleek way to keep Wallet Bulge down with its compact frame and RFID blocking interplates. Use the offer code LTTSEPTEMBER to save 10% and get free worldwide shipping. [♪ Music Playing! [♪ Music Playing! Alright, so while I wait for Anthony to come down, I'm going to run through how this whole thing is going to work with some really crude diagrams here. Our petabyte cluster uses a file system. It's open source and it's called GlusterFS. That's what allows these two independent servers here to present to the rest of the network as a single large share. Now, we could increase our capacity by adding more server boxes, but that's a project for another day. Literally, that's a project for another day. We've got a video coming with a full petabyte of storage in a single server rather than two. So make sure you're subscribed so you don't miss that. Today, we're going to take the empty 15 bays in server Delta 2 down here, and we're going to expand our storage with another 180 terabytes of raw capacity. We don't get to use all of this capacity though. Our GlusterFS implementation is geared towards raw capacity rather than redundancy. So the only fail safe built into our local network here is the RAID Z2 VDEVs in each of our machines. What that means is that out of our 15 drives, only 13 of them count towards our capacity with the rest. These two taken up by parity data that protects us from data loss in the event of a physical drive failure or a cable failure. Speaking of which, how about we replace that bad cable in Delta 1? Oh, that's going to be quite a project. Guys, the vault is going offline for probably about two hours, TM. Just vault. So Wannock is up. And these servers are so heavy, so we're going to have to empty all the drives out of them onto one of these carts, then take the server out, put it on the cart, and wheel it over to the island where we can work on it. I'm still super proud of this cabinet door holding open innovation. We drill the hole in the side of the keyboard tray, and now we can get the servers out super easily. You know you can just take the door off, right? No, the door has a filter. Like that's the purpose of the door for us anyway. Yes, I know I could take the door off. I mean, the sides off, the backs off. Clearly we figured that out. Thank you, YouTube comments. It's quick to label a drive. It's slow to label 60 drives. And this actually has to be legible. That's important. Why don't you actually work your way from the front, and I'll work my way from the back. Okay. And we'll do this like, you know, lady in the tramp, eating the spaghetti style. That's a strange mental image, but okay. Speaking of terrible segues, sure is cold in this server room. Good thing I'm wearing my LTT Swack-It. It's a sweater, it's a jacket. LTTstore.com, don't worry about it. I don't think most people feel bad for me struggling to stack all my hard drives that are very time consuming to stack. Why can't I stack all these hard drives? If you could grab this end though, and help me up onto the cart, that would be swell. Whoa, oh, it's hooked on my fly. Okay, I got it. That's fine. One thing we wanna do as we're carting this over to the kitchen there, is be really, really gentle with the way that we're moving this. This is about what, 350 terabytes of our company's valuable data on here right now. And I remember Patrick from Serve the Home telling me that Yahoo had an incident where they moved their data center like across the parking lot, rolling hard drives on carts, not unlike this one, and all the vibration killed like a significant portion of their drives. So we could lose up to eight of them depending on, oh, seven, because one of them is already degraded. But we don't wanna do that. So I guess this is the part of the video where we explain what happens when a cable fails in a store-inator. Now, most bulk storage servers with like lots and lots of hard drives use what's called a backplane. So they'll take fewer connections off of your SATA or your SAS adapter or your raid card or whatever the case may be. And then they will take those and they will split that bandwidth across multiple drives. 45 drives takes a bit of a different approach. So they wire every single drive up individually across less of a backplane and more of an underplane. The advantage is you get the full bandwidth. Another advantage is that in the event that a connection fails, you're not replacing an entire costly backplane. But the disadvantage is that if a cable fails, you are digging this entire apparatus out to replace one flaky cable. So there were two drives that were dead. One of them, it was fine when we replaced it. It came back up, everything rebuilt, and everything was rosy. But then there was the other one. We replaced it a couple of times. We actually replaced the controller cards itself and tried different ports on different controllers that we knew worked, but it still had the same issue. And what's weird is sometimes it would kind of work and we could start rebuilding the data on it. But like, what kind of data speeds were we getting? It would start out at like, you know, 300, 400 megabytes per second, which is kind of low, but fine. But then it would go down to like 10. And the ETA was like a year. Come on. By the way, evidence that my dust filter works just great. Yeah, it looks brand new. There is one little bit here that I noticed, but that's it. Okay, so let's not hate on my filtered front cabinet door there, okay? Okay, so this guy needs to come out. Now this looks a little tricky. Does that entire plane need to come out? Oh, I hope not. Yeah, cause like, I'm looking at this and this needs to go up under there in order to get screwed in there. And in order to do that. This like straight sucks. This is exactly why I haven't had time to do it until now. How long did we tell the editors this was gonna be down for? Two hours. One thing I did notice though, is that with how tightly integrated it is into the bottom of the case, I don't know how, I can't really get at it very well. Okay, they're shooting tech link now. So we're gonna have to do ASMR server upgrade. So I pulled these out and I can see where the cable goes. So yes, I will in fact have to pull out this and this. I don't see anything obviously defective about it. That's an angry episode of tech link. Someone must have removed some headphone jacks. I hope it wasn't Samsung. Okay, so this is it. We begin the funeral procession again. And this way. It's an open casket funeral. Oh yeah, should we close this server? Maybe? Like now is as good a time as any to do that. If we're gonna do full yellow and assume that we have everything right, then yes. Otherwise no. Otherwise no. Let's go no. Let's put some hard drives in. Yeah, having to keep track of everything. Sucks. Should have just got a jellyfish. So that drive with the rubbed off label that I wasn't 100% sure about, I got to the end and the only thing left is 131, which is clearly not that unless I have a wicked case of dyslexia. And I put 131 over here. So I think it's time for a sanity check. Here we go, ladies and gentlemen. Okay, Anthony, you got the drives? Do you want me to turn it on first? Should we do it? Yeah, it's not hot swap if it's not on. All right, we'll hot swap it, we'll hot swap it. I'm turning it on. Okay, here we go. Anthony, you want to take the wheel here? Sure. All right. Let's see, Z pool status. Okay, so guys, you can actually see here what was going on with one of our RAID Z2s. So each of the 15 drives is a RAID Z2 V-dev. So this RAID Z2, RAID Z2-0 is online. You can see the whole Z pool is degraded though. That's because RAID Z2-1 here, drive 117, the one that we just replaced the cable for, is unavailable. And then these are all the previous attempts at rebuilding it with different drives. Now we're gonna try again, but with a new cable. So we fixed 117, but now we've got four drives offline. Oh balls, that's like way down there. You want to let me know if anything changed? Five, six, seven, and eight disappeared. They're unavailable. So that's the wrong one then? Yeah. Damn it. Okay, so we're back and all the drives are here, but they're not showing up with their, like their 45 drive store nater friendly IDs here. And also five of them are resilvering. That seems bad. How do you resilver five drives? These are resilvering because they got cut without being offline first. In the meantime, should we add the other 15 drives to Delta too? What could go wrong? What could go wrong? So do we need to make a brick? Cause that's blusters crap. No, we don't need to do that with cluster. It's all in under slash Z pool. Pretty sure. Okay. I don't see anything here that looks like just making a VDEV. Yeah, me neither. Oh, okay. So we just do Z pool, create Z pool, raid Z two, and then the paths to the disks. But if our disks don't show up, okay, hold on. So let's do this. And it is online. So we might need to restart. Oh, okay. So maybe they're not hot-swappable. Should be, but. Maybe not. I don't know if there's a- They may have also configured the server to not find them. Yeah, their special driver might actually not do that. Okay, well let's see. In the meantime, we can check in on Delta one and see if it's resilvering a little faster now. It is not. So now what? We've replaced literally everything. The drive, the cable, the controller. I mean, do we wanna just pop the drive out and pop it in one more time and see what happens? We can try it. Okay. Brand new drive. Okay, so Delta two is rebooted now and we've got one dot one. Okay. So I guess throw them all in. Yeah. And see if we get them all. And if so, we'll create the raid Z two. And at least it may be degraded, but it's bigger. So all 15 drives for the expansion are in Delta two now. And I switched back over to Delta one and I have good news. Our resilvering is going at 1.91 gigabytes a second, which is pretty sweet. That means it should only take a few days. So 117, it's not there, is it? It's there. Okay, but? But when I try to replace it, new device is a different optimal sector size. What the crap? So I need to figure out what that is. Well, hold on, hold on. That's not like a 4K sector drive, is it? It might be. Cause that would probably, I don't know this for a fact, but I suspect mixing 4K sector drives with 512 sector drives in some kind of an array is probably super terrible. So do you want it to soft or it's offline anyway. It's already offline. Okay. Did we accidentally buy the wrong drives, Anthony? Oh my God. That couldn't be like the problem, could it? They're not. Are these advanced format? Yeah, yeah. Wait, you have got to be kidding me. Is that why these replace operations haven't been working possibly? You know what? We can check the original video, unboxing the petabyte project. It's a good thing our entire life exists on YouTube, at least mine does. Okay, we're some B roll of a hard drive here. Oh no, they are advanced format. So they're all there, but they're not there. There we go. New RAID Z2, RAID Z2-3. Includes 15 drives, all online. 117 is now back online, re-silvering, and it's doing it at almost 900 megabytes per second. So we're good? So I did notice, curiously, that the vault is still not any bigger. Right. I believe the GlusterFS service either needs to be restarted or I might need to adjust the config real quick. Okay. So let me just quickly look at this in Delta. Wait, is it one of these, is it dist.config here? Ah, yes. There it is, woo! Hey, I helped. Did I help on the Linux thing? Yes or no? You did. I did. Usually I'm not nearly as helpful when it comes to server stuff. Okay, so final update, guys. It was just a matter of getting volume four's brick integrated into Gluster. So you can see everything's here. Volume one, two, three, one, two, three, four, four. Sure, whatever, doesn't matter, no big deal. So now we're gonna fire over to our other server. This is Wannock server, runs Windows, whatever. Wanna fight about it? We've got our Z drive, which is actually even tighter for storage than it was before. We have less than a terabyte of space left, but that's no problem. The vault, 163 terabytes, ready to freaking go. And as for Delta one, it is re-silvering at 375 megabytes a second. I can push this back in very ever so slowly and gently. And we're done, bud. By the way, if you guys liked this video, we actually did a video rebuilding our water-cooled render server down here. You can check that out up there. And if you're not into that, you can check out our sponsor for today's video, iFixit. The iFixit Essential Electronics Toolkit is compact so it can go anywhere you can and help you fix almost anything. It includes their most popular precision bits and they're held in place with high density foam so you can throw it around without any of the bits falling out. And of course, it comes with iFixit's lifetime warranty. It's just 24.99 at iFixit.com forward slash Linus. So go check it out today. That's it for this video. I will see you guys next time when we're installing the single box petabyte project.