 Come here guys, come on. This is 1.2 petabytes. And the last time we built a petabyte storage solution, it had to be spread out over multiple servers. Not anymore! With Seagate's new 20 terabyte Exos X20s, we are gonna be building a single 4U server that can hold over a petabyte of raw data. It's gonna be flipping crazy. And truthfully, it's hard to be that excited because the real reason we're building this is because of a data loss in our previous petabyte project that is making it so that we have to build a new server to copy all of the data off that server while we attempt to recover it. So I'm gonna be having fun, but in a all of this is very sad and stressful kind of way. What I'm never sad about is telling you about our sponsor. Thanks to SignalRGB for sponsoring this video. With SignalRGB, you can control and sync your favorite RGB devices all from one app. Best of all, it's free. Download SignalRGB in the link below. I mean, it's mind blowing if you think about it. 20 terabytes of storage in a single drive. The original Wannix server, which was hard drive based by the way, had about 20 terabytes of storage, except that it was an entire 4U chassis like this one full of one terabyte drives. Man, Wannix server was 20 terabytes up until like two years ago. One freaking drive. And the craziest part is last time we did a petabyte project, we wanted to get a full petabyte of addressable storage into a single chassis, but we fell short because I wanted a 75 drive version of the chassis. So it would be actually five of these rows, but there was a bit of a snafu and we didn't end up getting it done. The crazy thing is since then, we've gone from 16 terabytes maximum capacity to 20. So we are gonna have exactly what we were targeting, except in 60 bays instead of 75. Now I couldn't help noticing Jake that none of the hardware is in this. Well, it's kind of a long story. Does it involve these bins of hardware over here? Yes, yes. So in there is actually the original hardware, if you look there. So I think it's a 26 core Xeon gold, I think. 512 gigs of RAM. It would have had four HBAs in there at one point, but in between the process of this server being built and it arriving to us, I kind of decided I wanted our cache drives to be NVMe, not SATA. And unfortunately there's not really a lot of PCIe on here. There's probably enough. We probably could have put like some jank four splitter bifurcation thing in there, but I figured we should go Epic instead. We're going Epic, ladies and gentlemen. Need that PCIe. It's kind of crazy how like as an enthusiast or a gamer, you don't really think about that as AMD's advantage over Intel in the server space. You kind of think of it in terms of performance or core count. I've heard that's important. Eight channel memory, right? Like these are all things that as gamers we understand. But 128 PCIe lanes, when you're plugging this kind of shiz into your CPU, matters a lot. Yeah. What we ended up with is actually the XL 60s turbo variant. So it's got a little bit bigger CPU than the base one. But if you wanted like us, you could probably ask 45 drives and they maybe make you an Epic system. It's just not something that they show on their website. You probably don't need to, as long as you're just running mechanical drives, but because we wanna put a bunch of NVMe drives in, we do need the extra PCIe. Now at this stage, we've got a couple of options. We can work on things together or we can divide and conquer. But the one thing I'm not flexible on is we need to take this thing and put it on one of those carts. Because if we load it up with hard drives and we have to move it again, I'm gonna lose my. I already brought it downstairs. It was fine. No, no, but full of drives. Yeah, brought downstairs full of drives. Also it's better to carry it than put it on a cart. Cart is bad. It was IBM study. Look it up. You gotta have like. That was across like parking. Parking still. There's lots of bumps and stuff. Oh. It's gonna be fine. Look, I get it. You're kind of a small guy. You know, it's okay. It's gonna be okay. If you don't wanna carry it, we got other people for that. You know, you employ like 50 people. It's not saying anything. He's just putting drives in it. He's giving up. Should we maybe, wait, hold on, hold on. Should we do. Oh, now he doesn't want it to be heavy. Oh no, I'm sorry, Jake. I'm sorry, Jake. Is this not how you wanted the hard drives to go in? Oh, oh, oh shoot. Oh man. Oh, I'm sorry. I just think we should put the motherboard in and make sure that's working first. Oh, they just keep going in there. They just slide in so nice. Love 45 drives mechanism. Look at this, 20, 40, 60, 80, a hundred terabytes just like that. Oh yeah, bud. Let's actually talk a little bit about the drives that Seagate sent over. These are their data center enterprise grade drives. They run at 7,200 RPM. They've got 256 megabytes of cache each and a SATA six gigabit per second interface. They're gonna actually be pretty similar to the 20 terabyte Iron Wolf Pro Pro Sumer NAS drives in terms of performance, but these have double the rated mean time between failure and almost double the rated annual workloads. So they're just designed to run a little heavier. They're also $20 more expensive at MSRP, but a funny thing that we've noticed in the past is that these Exos drives often end up being less expensive once they've been on the market for a little bit. So these ones have only been at retail for about a month, month and a half. So the cheapest that we could find these specific drives was $571 each from a New Egg Marketplace seller. That puts this at, wait, $35,000 worth of drives. And the funny thing is they're actually gonna send four more so we have some cold spares. So that would put us like 3750. That's a good thing to have with any storage setup that has more than, I mean, realistically, a handful of drives because- Realistically, we should have a hot spare. Mechanical devices do fail. It's something that happens and it's something that you have to be ready for. Holy crap, that's sexy. What? This is like one of my favorite Epic boards for sure. Seven PCIe 16x slots. Is that a USB type C on a server board? Yeah, it's also, it's all Gen 4 too. We originally got this board for the sensible jellyfish fryer, but I think they originally sent the OG Epic version. Yeah, the non-Gen 4 one. Yeah, and then this is the Gen 4 one that does Rome and Milan. Sorry, Yvonne, I'm filing the papers. This is my new wife. What happens when the new one comes out? Well then- More papers. Off with the old one with the new. Jesus. What kind of Epic are we putting in here? Hmm, okay, so it's not a Milan because we still can't get any of those. We have two, we finally got two. This is not one of them. This is a 7402P. It's the previous generation Rome chip. It's 24 cores, which is probably excess for this, honestly, but it should be pretty good. Once it's paired up with the 512 gigs of RAM we take from the old board. More funny things, we don't have a CPU cooler for this that fits in there. We have this one, it's a U12. It's a little big though. We'll just run it hot rod for now. Yeah, I think we'll just leave the top off. Can we just take a moment and talk about how the Molex connector has apparently been doing Roids? Have you seen these? Wait, plug it back into the right one. What is this? It's all about having a steady hand. If I just blew right now? No, you wouldn't do that. I would ruin both of our days, so I'm not gonna do that. Just a quick- He was telling me what a great job I did. You mean like what a great job I did, right? Well, someone did a great job. Whose name is on the channel again? Oh, okay. Hey look, you started it today. You 100% started it. Fix your petabytes yourself, damn it. You started it. I'm done, I'm through. You started it. Oh God, oh no. What, what, what, what, what? Okay, so to use all the ports we wanna use, we have to move a jumper. Oh no. So right now they're both on the right. Two and three, yeah. Two and three, two and three. So that's where we want it actually. Perfect, well that was easy. I don't know if I believe that though. That's fine, we'll find out soon enough. Worst comes to worst, we just have to switch it to the other side. Okay. Because then it would be on one and two, one and two. Okay. Easy. We're hoping that that's right. Okay, now Epic has no built-in RAID functionality, so fortunately, True NAS has a built-in redundancy function so that you can just install the OS simultaneously to both drives, and it's not RAID, but it's a backup, which is better. It's a ZFS mirror. Is it a ZFS mirror? It literally just makes a simple- Oh, I didn't even know it was using ZFS. Oh, cool. This does not feel fine. It feels perfectly fine. Oh my God. Can you imagine though? No, I cannot imagine it. I don't think this is built to be tipped like this. Okay, yeah, everything's fine. That's horrifying. So this is cool. These connectors right here are super cool. They can be used for SAS, SATA, or PCI Express. So it can cause some confusion because I thought they were U.2, which would be PCIe for NVMe drives, but actually they are, are they SAS or SATA? Probably just SATA, actually. Yeah, I think they're SATA. So we're just gonna connect this little breakout cable and bippity boppity, we'll plug these into our drives here, and we're good to go. It's cool. It passes through the SATA power too. You see that? Yeah. So we actually do need four because story about our NVMe drives. They're not here yet. In fact, not even two of the cables we need are here yet. The Oculink, wherever that is, to U.2 cables are not here yet. They were supposed to arrive today. Maybe they're just sitting in logistics. I don't know. Anyways, yeah. So we're gonna do two eight terabyte SATA drives that are from Micron, and we're gonna do two one terabyte NVMe's. Those are gonna be our intent log for CFS, which is almost kind of like a write cache, not really. And then our level two arc, which is just an extended read cache that adds on to the RAM cache that CFS uses by default. Why didn't you take the... Well, they're there for in case. They're there in case. Get it? This is the case and they were in it. Andy likes it. Andy likes it. Andy likes everything. You pay him to like it. I pay you to like it. Well, I'm just a bad guy. It's just one of you is better at their job. Oh yeah. Hey, thanks Linus. Put her in the first slot there, bud. There it goes. Oh yeah. Look at all that PCIe baby. We still got two more slots. Well, one more slot that works. Yeah. Wow, you mangled this cooler. What the heck? I did nothing to that cooler. You know I didn't do anything to that cooler. Oh, look at that part. I didn't touch that. This was brand new. Was it really? What happened? I don't know. Probably when you were fumbling it out, probably both of us messed it up a bit. We might have a big problem, Jake. I don't know if I can plug in the front IO. It's right angle on this board. Oh, who cares? I can't get it in. Oh, who cares? Okay. We could do it. I have a new idea. We just take the pins and we just bend them up. I love it. Okay, I'll be back. Did you just pop that out for no reason at all? I didn't know the screw wasn't in yet. What do you think I was doing? Picking my nose? I mean, usually you're not doing too much, so. Really? Mr. Stands there and- We hit a cord here, boys. Walks up and unplugs what I'm working on. I did the math. And lttstore.com is a great deal. Check it out. We've got this new hoodie. Is it even on there yet? It will be by the time this video is up, I think. Sorry, you can finish your thought. I don't even remember what I was saying. Doing the math? Oh, I did the math. So at one gigabyte a second, which I think is a little- Optimistic. Optimistic for the old vaults, especially in a degraded state, it would take seven and a half days if it sustained that the whole time to move 700 terabytes. So let's say 500 megabytes a second, it'll take 15. I want to hear you say it. It's going to take a month. You're supposed to tell me I was right. You said it was going to take months. I said it was going to take weeks. Well, to move everything. Yeah. To do it three times. Three times? Twice. I don't want to have new vault stuff on here. I want to move it off and then move it back. Is that like, is that unreasonable? I don't know. But I think we're ready to fire this thing up. Really? We ready? Are you ready, boys? You plugged the management port into it, I see. Are you ready? Let's see what you were saying. You plugged it into management. Yeah. That's what you wanted to do, right? Yeah. Okay. Yeah, I wanted to plug the crappier one into the management port. I thought you were just stupid. Smells like fire. Does not. Is it still post-coding? I mean, it's a server, so like- Yeah, it could take a minute. ASRock boards are sort of middle of the road as far as that goes though, right? Speed? I think it's usually pretty quick, actually. Yeah, it's probably got to do memory training though for 512 gigs of RAM, so. Oh, it's booting. Nice. 170 seconds it's been going. Hey, look, look. 63, I think that's right. Oh, wait, no. That's not right. We're missing some NVMe. So we got our two NVMe's. Oh, the SATA drive, 6.99 terabyte. Yeah. So we're missing one of those. So we're missing one of the SATA. And then we're missing two of these? Yeah, theoretically. Oh, bananas. Oh, I bet you it's those two. You suck at this. You put those in. This is your fault. You distracted me. Those were very out. Oh, I can feel them. They're doing stuff now. Okay. I don't know why this boy is not working though. Okay, 65. We're at 65. Okay, we have all our hard drives now. We're just missing. Okay, so tell me if that's the bad one. I don't know. I guess we could have checked the serial. I don't actually know if it'll, oh yeah. It could also be that this cable is bad. These cables suck. I've seen like tons of them fail, so. We have another one of them. Serial number mismatch it says. So it did pick up the drive then. Says SATA link is three gigabit per second. Oh no, it picked it up. Micron 4300. Okay, try to plug the other one in for a second. Yeah, that cable might be bunk. Oh, it couldn't be the cable Linus. Must be a bad Micron enterprise drive. Yeah, for sure. You know, with all the crap that has taken place that did not make it into videos, it's probably just as well the vault gets white points in a while. You back up. Look at that. 26 baby. Nice. So we got 900, 900. Those are two NVMe's. Let me just switch this to a hundred. We got 6.99, 6.99 and the rest. Okay. Now we have to make a pool. Okay, let's do it. Self. Oasis. Sure. Let's call it Oasis, the data Oasis. Good gravy. You have to select them one by one. Yeah. I mean, I guess that's a. Think of how often you're gonna do this. Champagne problem. I think you can also hold control. I'm just, you know, whatever. RAIDZ 2 gives us two drive failures per VDEV before we actually sustain any permanent data loss, which from our recent adventure we found is not necessarily the be all and end all, but because it doesn't save you from bit rot. Well, look, TrueNAS sets up a scrub for all pools by default monthly. Yeah. It just does it automatically. CentOS had new such functionality. You can't screw this up. You can, but. Yeah. All right. Anyways. Should be fine. I want to listen to them initializing. Formatting. They're already initialized. The server's on. What are you doing? They're not doing anything. Oh, they're going now. That sounds terrifying. I know it's normal, but. Like enterprise drives are. We are doing data things. Wow. These are ones are getting kind of warm actually. Maybe we should put the top on. Yeah, we probably should. Hey, look at that. Okay. So we have available 895 tibby bytes. Oh. 984 terabytes. Okay. So should we remake the array? Just RaidZ1 so you can see more than a petabyte of usable. No, that's okay. RaidZ2 is bare minimum for this mini drive. We still have 1.2 petabytes of raw storage plus 32 terabytes of NVMe once it's done. So one point of clarification. Each VDEV. So each grouping of 15 drives is running in RaidZ2. So that means we could actually theoretically lose up to eight drives before sustaining any kind of permanent data loss. But you'd have to get really lucky. I'm going to call it Vault3, okay? New, new Vault. No, I'm not doing it. I have power in this manner and it's Vault3. I don't care. One day I'm going to come in on a weekend and just change it all. Okay. We got a network interface. Look, these two are a hundred gigs. Sweet. One thing that's really smart to do on your storage service, especially one with this many drives, is you can actually set up a scheduled smart task. Like on all of your drives, do a quick smart test like once a month. And you can usually tell like if a drive is going to die, just based on that, it'll like email you the results. That's one nice thing about mechanical drives is they tend to spit out some warnings before they actually completely give up the ghost. You can too, like reallocated sectors. Not always though. You got anything in here that's bad? I don't hope so. Spam log, Pickle Linus. How's it going Pickle Linus? I was making a thumbnail for one of my just like random troll streams on Floatplane. So I just put that up there. I killed him. Oh, there we go. There we go. Whoa. There it is. 95 tibby bytes, I think. Anyways, the moral of the story is, holy that's a lot of storage. Is this a holy now? It could be. Now it's time for us to drop it in the server room and start a weeks long data transfer. It's probably gonna be like two weeks. In a use case where you need more IOPS that is operations per second, maybe if you're video editing or you have a database running on this, you might wanna do a setup that has more VDES because typically it means you're gonna have better IOPS performance. In our case, basically everything going to this is sequential AF. We're moving like 500 to a terabyte of footage, of just straight footage in one go. So it doesn't matter for us. If anything, we just want more capacity while still being safe. And this is our middle ground. Actually, it's not really middle, it's like on our side ground. You know what I mean? Thanks to privacy.com for sponsoring this video. Valentine shopping is well underway and privacy.com is here to help. With the growth of technology and ever increasing online fraud, keeping your banking information safe and secure is becoming more and more of a concern. And privacy.com lets you shop online even at unfamiliar stores with virtual credit cards that offer way more security and control than conventional cards. You never have to worry about being billed twice, getting upgraded to another service without your consent, or having to change your card information in the event of a hack. Privacy.com also allows you to set spend limits so that you can take control of your finances and stick to your gift budgets. Their free service offers you up to 12 cards and by upgrading to pro, you'll be able to get 36 cards, more security features, and even 1% cash back. As a special bonus, if you sign up today, you'll get five bucks. It's five bucks. So check it out today at privacy.com forward slash Linus. That's privacy.com forward slash Linus. If you guys liked this video, maybe go watch the video on the original Petabyte project. It was a cluster. We actually needed a cluster cluster, two different machines in order to reach this same capacity. It's come a long way.