 After months of performance hiccups, several blue screens, and us literally being able to bring down the entire core of our operation just by ingesting video footage to our storage server, I am finally ready for a change. Admittedly, maybe running new, new Wannock, our main video editing server on Windows, was a mistake. I had my reasons, some of which were valid, but it's clear at this point that you guys were right. The advantages of running Windows are far outweighed by the problems that we've encountered. So, it is finally the day so many of you have been waiting for when we remove the last Windows machine from our server room. We're going Linux, baby. We're doing it right. Thanks CableMod for sponsoring this video. CableMod allows you to personalize the look of your PC with custom-colored sleeved cables. Try out their configurator and build your cables exactly how you want them with their realistic cable preview. We're going to have it linked down below. This should be easy. I've got my Linux install USB right here. All we got to do is slap it into Wannock server and boom, we're after the races, right? You see, the thing is, unlike most of our videos where the hardware is just going to get torn down immediately afterward, Wannock server is our one and only production network editing server. We can't just shut it down, format it, and reinstall anytime we want because A, we wouldn't have anything to store our videos on or edit off of, and B, we would lose all of the work we've done on our active projects. That means we either have to do this whole process on a weekend or we have to migrate, at least temporarily, to a different server. As you can see, it is not the weekend. So the plan is to use my home NAS machine as a test bed and then eventually migrate what we've done over to new, new Wannock. Wannock 3. New, new Wannock. My intention is for this to eventually end up at my house, but that may not end up happening for a number of reasons, not the least of which is that Jake has become very enamored with the kind of performance that we get out of these Gen 4 Kioxia NVMe drives. He and I are going to have to fight that out a little bit later. In the meantime, let's build up a server, shall we? Actually, that's a bit of a lie. This server is already built. So what we need to do is we need to upgrade it. We'll start by taking out this Epic 7402. I got it. 24 cores is plenty for what I'll be doing at home, but as we learned the hard way, you basically can't have enough CPU performance if you want to get the most out of these Gen 4 NVMe drives. Oh, and if another eight cores doesn't end up alleviating the bottlenecks we've found, we've got a 7742 64 core that we can play around with later. Another thing you can't have enough of in a storage server like this is RAM. And I mean that both in terms of capacity and in terms of speed. When we were deploying this at my house, we put the smallest, crappiest kit that we possibly could. I mean, obviously without compromising on reliability. So this is 64 gigs of 2666 DDR4. Truthfully, the sticks I'm putting in now aren't really ideal either. They're running at 2666 mega transfers per second compared to Epic's maximum of 3200. But we're going to have 256 gigs rather than 64 gigs, which is going to make a big difference, especially if we're using ZFS, which we are. As much as there might be some kind of GPU accelerated storage something, I've never heard of it. So we're going to pull this 1050 Ti out here. That was supposed to be for Plex transcoding. And we're going to replace it with this gorgeous Mellanox ConnectX 6 100 gigabit per second network card. Do we really need these one terabyte drives for boot? We have like a dozen of those Optane M.2s that are like 32 gigs and they're perfect for server boot drives. Oh really? Oh, all right. Well, I guess I keep using them for stuff like this where it just really doesn't matter how much capacity you have and you just want something reliable. Speaking of our installation media, there was a fair bit of deliberation when it came to choosing an operating system. Ubuntu server seemed good, except that the current long-term supported version, or LTS for short, doesn't have OpenZFS 2.0, which features a bunch of performance improvements for ZFS, which is the file system we intend to use. And then DBN 11 sounded good. It has OpenZFS 2.0 and a version of Samba, the software used to host Windows compatible SMB network shares from this year, which is great, except it's from March, which is not great. So the beginning of this video ended up being a bit of a lie because while we could have still gone with Linux, just find a distro that's got a new enough kernel, potentially build OpenZFS 2.0 and Samba from scratch, there is an easier option. TrueNAS, it's got OpenZFS 2.0, a very recent Samba, and a really nice web interface for managing your storage. It's not Linux, but it is based on a Unix-like operating system called FreeBSD. So we're back on the used to be FreeNAS, now TrueNAS train. I don't know what it is about their product, but pretty much every time I go to use it, something blows up in a way that is totally inexplicable. I've ended up escalated to directors and VPs of engineering multiple times with nobody ever able to explain why the product just won't work properly for me. But hey, fifth time's a charm. Works properly for Jake. Yeah, as long as it works properly for Jake. You know, our backup servers are running TrueNAS for like a year now. I know, I know it's a good product. I've recommended it to countless people. I've just never been able to get it working the way that I need it to work. Did it just turn on? Do you have it set to just restore power on AC? Probably. Let's just give it a sec here. That's what servers are supposed to do. Okay, where's that USB? We don't need it. Oh. We're gonna do it properly. I thought you told me we need a USB. Just for visuals. Are we gonna go IPMI AF and use an image and everything? Look at us doing things properly today. Yeah, I didn't download TrueNAS, but I will now. Wait, what happened to my laptop? I put it away, you don't need it. Oh, you told me to bring it. I didn't realize there was a keyboard here. The problem with Mac IPMI is I can't like press F10 properly. Like doesn't work. Press F10 to pay respects. Doesn't work. I can screen record on this thing for like an hour and only use like 30% battery. And it's touch pad is the equivalent of being hung like a horse. It is, look at it. Look at this, let me show you my touch pad. Okay. Look, it's a touch pad. Actually, what I was really gonna show you was my underwear from lttstore.com. lttstore.com. It comes with cat hair. Comes, no it doesn't, that's from your cat. I just walk out in the hallway and she's like, what's up bro? Virtual CD-ROM, enter. Here we go boys. This is the smart way to do it. You shouldn't actually ever have to plug a monitor into a server. You ready? Install. Wow. Wow. That was inspirational. Look at it go. Okay, hold on. Hold on. Pick our drives. Hold on, yeah, we don't wanna install the wrong drive. Now this is cool. Even though AMD EPYC servers do not have any innate support for RAID, that means you can't even install two boot drives like we did and then just configure it in the bios to run RAID freaking simple one. True NAS has a quick and easy way for you to install your operating system to two drives. So in the event of a hardware failure, you're just ready to go again immediately. Yeah, it literally just shows up as a separate boot entry. You can just pick which one to boot from. Okay, time to go to lunch. See you later. I'm asking if I can turn it on. Well, that's too bad because I already did. Ah, roasted. Yeah, what's up now? What are you gonna, hard power it off? Yeah. What's up? Ha, got him. Now who turned it on? This guy. Oh, I turned it off again. I'm turning it off first. This is bad hardware practices. Hey, wait. Mother. Man, it's not plugged in all the way. Oh my God. Really? It snapped out. It must've been when you unplugged the power. I'm turning it off first. Why is there no- Is there blinking on the port? IP. No. We have to take it to the server room later anyways. Let's just take it there now. This is the way to do it. I mean, did we say we were gonna do everything properly today? No. Okay, cool. Because that's definitely not what's happening. Hey, I got a hundred gig. You won't believe how much gigabytes you can put in this bad boy. There's a bunch of hard drives under there. Stop slapping it. Let's go back over there. I don't wanna wait. Okay, let's create a pool. Yeah, name. Name. It's called Lambo. Lambo. Don't do all caps. So do I click them in order to add the video? No, you don't have to. Just click all and then unselect that. Yeah, we don't want this boy. The DAO. I think that's the virtual thing for my PMI. 12 selected. Let's click over. So I did a little bit of testing beforehand. It doesn't seem like it makes a difference between having one VDEV or two for our workload. Okay, RAID Z is fine. RAID Z is gonna give us one failed drive before we actually start to lose data. I'm not expecting to lose a ton of these drives. And this data is gonna be replicated using our snapshot to another server that's over in unit 101. And then it'll be replicated again offsite to Camloops once Jake gets around to fixing the new Camloops server. Yeah, there we go. Okay. 72 terabytes, that's not bad. That's not bad usable. Okay, so we gotta make a, oh here, first thing, go to that. Yeah. We have to turn auto trim on, pool options. Oh, why is that not by default? Well, most people don't use SSDs with TrueNAS. That's fair. We gotta make a new data set, add data set. And then let's pull this Z drive. Turn compression off, we don't need that. Pretty much all the data we're gonna write to this thing is incompressible video footage, so there's just no benefit. We could turn it on later because there are other things like Word files. But for now, because we're gonna do some performance testing too. Let's leave it off. Turn A time off, we don't need that either. That's like the access time. Is that a festival store? The last time a file was accessed, you might use that. To see if your kids have been looking at your. Share type SMB, that's gonna set it to be case insensitive. Okay. And we're gonna leave the record size at 128 KB. Now you could theoretically change it to one megabyte. That might be better if you're like have a Plex server because those are bigger chunks, which means less overhead. I mean, our chunks are pretty big. But for whatever reason, when I was playing with Premiere, it seemed like it was a little worse. So. At least in scrubbing. Cause it will increase your latency. Oh, we gotta do some ZFS tuning. Yeah, because why would you want things to just like work immediately? Look, it's not made for NVMe. We're kinda doing it dirty right now. Primary cache, all one word, equals metadata. And then we'll do space, Lambo slash Z drive. Okay. Enter. Aha! Okay. So what that just means our RAM now stores only metadata? Yes. Which makes sense. Cause our storage is so fast that there's pretty much no point using our system RAM as a more conventional cache for it. For most use cases though. I haven't actually played with it in Premiere. It might actually like the lower latency. And when we deploy to actual Wannock, we'll have a terabyte of RAM. So maybe it might actually make sense. But for now we're just gonna leave it off. Okay. Wow. It's really poorly formatted, but Lambo Z drive users metadata. On the next line. Cool. It worked. All right. Now we got a Tomb of Samba. So on TrueNAS SMB slash Samba is not really configured for the type of performance that we're going for. We're talking like 20, 30, 40 gigabit on our 100 gig card, not gigabit or 10 gig. So we have to make a few changes. Now by default for some reason, at least according to this dude on the internet that has this article about TrueNAS and the testing that I've done, it seems like they set the default threads for SMB multi-channel to like one read, one write. Whereas like the default in Samba is like a hundred. There's probably a reason for this. And maybe they've changed it back, but from testing before changing it to testing after, it was a huge difference. I wonder if it's one of those things where for the average home user repurposing an old machine and just like throwing a couple of hard drives in it in the closet. Yeah, maybe it like overwhelms, you know, that the Athlon XP. This is also like the community edition, right? So if you were to buy a machine from TrueNAS, one of their pre-builds, I imagine they would already configure this stuff for you. Anyways, we're just gonna copy paste this stuff into our config. Just need one more zero in there. A lot of bytes. Yep. Now the thing is these drives are capable of like tens of gigabytes a second of raw reads and writes. But as soon as you start trying to run a file system on them like ZFS, performance takes a bit of a kick in the balls. Especially when you have to calculate parity, which is a whole other thing. That's why we have created the CPU. Hopefully we can get a little more parity in. Let's see where we end up. Okay, we're gonna do a write test first. 50, 80. Here we go. 12 jobs because we have 12 drives. Laying out some files. That's pretty good. Wow. That's more than 100 gig. It kind of fluctuates a bit, but that's pretty damn good. That is not bad. As long as we're above 100 gig. Yeah, then we can't possibly be bottlenecked by the drives. And I think we're bottlenecked by CPU right now too. I bet if we go look at the CPU. Oh yeah, it's probably getting absolutely crushed. 100%. 41%. You know, it's probably a little bit slower in terms of writing that. I bet you if we do a read test, it would be 100%. But I think it's just because it's calculating parity and whatnot. TrueNAS is not built for these speeds. It's really not. We're using them for these speeds. But properly. Yeah. All right, read eight gigabytes a second. You know, I was getting a little higher than this before. I'm a little surprised. I wonder if the CPU is slower. So far beyond good enough. Now, something to note is that when you're benchmarking your desktop system, changing the Q-dev to get like massive numbers is not representative of the real world because an individual user sitting in front of a computer could never use it hard enough to reach Q-depths of, you know, eight probably, let alone 16 or 32, which you might do in benchmarking. However, for a machine like this, where literally dozens of people are accessing it, that is conceivable. And that is a reasonable way to test it. But we were only testing with two. Yeah, so that's why I said it's turning it up. It's not a hack to just show you guys a bigger number. It actually is applicable to this use case. I wonder if we try more threads. We have more threads. Let's try 24, two per. Give me all the threads. Give me more threads. I want a bigger number. Whoa! Hey, there it is. 18 Gibby bytes per second. Interestingly, CPU usage doesn't really go up. TLDR, it's very fast. Very fast. It's way faster than we ever need and it's way faster than a lot of people. So let's try it. You know, it's good that we actually left the one terabyte in there because of those ingest issues. So that was another problem that, actually that was what really prompted me to go, okay, yeah, forget it. We're done with this. Because when we would ingest footage from the stations over there, if it was greater than one terabyte, it would fill up the RAM. No, it's less than that. It basically like when you write to storage spaces, it just goes to RAM and then it flushes to disk slowly. And when we had the URSAs, those drives could do like one and a half gigabytes a second, you know, times a few. If you ingested fast enough, you'd fill up the RAM and storage spaces only lets itself use half of your system memory. And then at that point, it would just crash the network share. Like for everyone. And there's no evident way to turn it off. Because our disks are fast enough. We could just go straight to the disks, but we couldn't turn off the stupid RAM caching. And especially when you've got an OS that blue screens every once in a while, having 500 gigabytes of data, potentially in a RAM cache. Yeah. One does not simply. So this is kind of dog poo. What we need to do is go into one of the editor stations that is not Windows Server running storage spaces and like copy these files or something. Okay. So I actually, oh my God. I didn't touch it. I'm sorry. It's been a stressful week. What's up? You want to drive for a sec here? Of course. Okay. I'm gonna drive for a sec here. Okay. So I want to copy this to my local drive. Huh. That is not very good. Jake. It's possible that the issue is just Windows file transfer being Windows file transfer. So we're pulling out the big guns here, bringing out cho easy copy. Look at that. 20 gig of it. Oh, well that's better. Where are you going to and from right now? I'm going from Wanek to the other server. The test one. Yeah. And I'm sure if we open this, lots of blinky lights. So there are. Okay. Yeah. There's your CPU usage right there. Okay. I'm copying over a red mag. Boom. Pinned to 495 megabytes a second, which is the speed of a red mag. Okay. What else we got? Another red mag. Brandon is, what's yours going at? 499. Okay. So we've got a total of a gigabyte a second heading over there. I need to find some more media to copy here. What you'll see is sometimes it'll slow down when it's going between clips. Red footage is a bunch of clips. So between each clip, Windows file transfer will like dip for a bit. Yeah. We're definitely running into some kind of bottleneck on this system where it's maxing out at 500 megabytes a second, no matter what we plug into. But still overall, the ingestations are performing better than with old Wannock, which is new Wannock. So this will be new, new, new Wannock. Anyway, the point is I paused all the transfers and we're going to have three or four editors now try to edit. And then I'm going to start these again and make sure everyone's smooth. Our guinea pigs are going to be Ed, Mark, and Emily over here. And you guys are pretending to edit, right? I mean, that's basically all everyone in here does, right? Is everyone, A prime, pretending to edit? What? Are you pretending to edit right now? I'm not pretending. Oh, okay. Well, we've got at least, okay, we've got at least one editor actually editing. Fantastic. Oh, oh, oh. Hoffman, that's a beautiful picture of David. All right. So guys, keep pretending to edit. Andy, you stay here. I'm going to go turn on all those file copies. Just like before, we got about 10 gig on the one on the right, five gig on the one on the left. And how are your projects going? Does it seem fairly normal or very normal? Or better than normal? It's not better, but it's normal. Not better, but normal. Yeah, it's pretty much exactly the same. All right, thanks, Ed. It's good. It's acceptable. Is it the same? It's not worse. That's good. But it's not better? It's hard to say, to be honest. Okay. Yeah. Okay. I could edit with this, yeah. Okay. Really wasn't the goal today. The goal today was, hey, this is going to be perfect and amazing. You guys are going to love it. So, No, no, no. Is it? No, no, you had your chance. The goal was the same. Okay. Ed, how's yours doing? I think this is doing well. Historically, I think it's about on par with what performance was like before. But as of late, this is a lot better. I used to have runaway footage pretty, pretty often in the last little bit. For those of you not familiar, that means when you stop the play head, it like keeps going for a bit or? Yeah, yeah. Okay. Like the last couple of weeks, right? Yeah. The last couple of weeks when I would play back footage, especially the footage from the Sony's, it would just like run away for me for a few seconds. It's hard to make an accurate cut that way. Yeah. Okay. Now this seems fine. Okay. All right. Confirmed. We're going to move ahead with it. And confirmed, we're going to tell you about our sponsor. Thanks to Current for sponsoring this video. Current is a mobile app and debit card that helps you spend and save better so you have the freedom to do you. Their app makes it easy to see where your money's going and set different savings goals to keep you on track. You can withdraw your money without fees across all 40,000 Allpoint ATMs in the US. And there are no overdraft fees up to a hundred dollars. You can send money easily to friends and sign up takes less than two minutes. So head to current.com slash LinusTechTips and sign up for Current today. If you guys enjoyed this video, maybe check out some of our previous server blogs. It's the last time we try to deploy this server. How about that? Yeah. Someday we'll get it right. Maybe this is it.