 I called it! Called what? Okay, but I did though. I mean, sort of. I'm pretty sure it was me and I'm also pretty sure it existed before that video even came out. It doesn't matter. The point is, meet the Supreme Raid SR1000. It looks like an Nvidia T1000 workstation GPU. In fact, it even has the letters T1000 printed on it and the same mini display port ports are in there, but they're blocked by, ooh, solid metal. That's because this GPU is not meant for graphics. And before you say you know where this is going, no, it's not for cryptocurrency either. So what the heck is it? Through some kind of software funkery, G-RAID is using this GPU to act as a freaking storage accelerator. And if they're to be believed, which I'm not sure if I do yet, this thing with the right array of NVMe drives can supposedly sustain transfer speeds of over 100 gigabytes per second of sequential throughput. Holy is what you might say if I didn't segue to our sponsor. Kioxia, their BG5 NVMe SSD brings PCIe Gen4 performance to an affordable price for systems and notebooks. They even make a 2230 sized one so your PCs can be lighter and smaller than ever. Check them out at the link in the video description. I have so many questions about this, but before we can even start to answer them, we need a little bit of background. Combining multiple storage devices has been a staple of computing for decades and generally falls under the umbrella of technologies that we call RAID, or redundant array of independent disks. RAID can serve a variety of purposes, improving speed, data protection, capacity, or usually some combination of all three of those compared to a single drive. Now, traditionally, high performance RAID required dedicated co-processors typically found on hardware RAID cards. You would slot one of those bad boys into your motherboard, connect all of your drives to it, and it would handle both the high throughput of these many disk arrays, as well as the parity calculations that are required by popular RAID configurations like RAID 5 and RAID 6. If you wanna learn more, we actually have a tech quickie on this subject from almost 10 years ago. But RAID cards have a problem. As we've transitioned from mechanical drives to solid state and then to NVMe, storage devices have gotten so fast that RAID cards haven't been able to keep pace, turning them into a performance bottleneck. So the current meta is to connect your storage devices directly to your CPU via PCI express. This improves both the throughput and latency, but requires the CPU to handle those parity calculations and any other overhead. This is called software RAID. And in some ways, it's actually kind of a big step backward. First of all, CPUs are freaking expensive, and in more ways than you might think. A lot of enterprise software is licensed according to how many CPUs or how many cores are present in your server. So you better believe that big businesses are all about squeezing the absolute most out of every box. Also, CPUs are generalized processors. I mean, you can brute force it. Here's us hitting 20 gigabytes a second in software RAID. But the issue is that even with a 32 core epic processor, we're looking at a lot of utilization here just to manage storage. And if you compare that to the theoretical combined read speed of around 75 gigabytes a second for our 12 KIOX YSD6 drives, you can see that we were leaving a lot of performance on that table. If I'm a server vendor, that's too many wasted CPU cycles that my customers now can't allocate to something useful or can't rent out to their customers. That is where G-RAID comes into play. The most obvious difference here right out of the gate is that there is no port to plug a drive into, nevermind eight or 16 drives. Instead, the drives connect directly to the CPUs PCIe lanes just like they would with software RAID. So this server from Gigabyte handles all of that through a back plane here in the front of the chassis. Then the G-RAID card just plugs into any available PCIe Gen4 slot. We'll do that a little bit later. And all the storage communication happens over the PCIe bus. No direct connection between our RAID card and our drives. Weird, I guess we won't need any of our new cable ties from lttstore.com. Although they are available in so many colors. And you might be thinking, gee Linus, even at Gen4 speeds, a 16 X PCIe slot can only push around 32 gigabytes a second in either direction. How could this thing possibly do over a hundred? That's the special sauce. None of the storage data actually goes through the card. That's the old way of doing RAID cards. This card only handles the RAID calculations and directing the system where to read and write from. All of the actual data flow just goes directly between the drives and the system memory. No man in the middle. And it does all of this while using barely any CPU resources. Or so they claim. What I think this means is that we could plop this G RAID card into any system and it would just work. We are definitely going to try that later. But for now we're gonna stick to the validated server that Gigabyte sent over for performance testing. Our CPUs are a pair of epic 75 F3 monsters. They're only 32 core, but they'll hit 280 Watts of max power consumption and will boost to four gigahertz. And then we paired these with some equally monstrous memory. Micron sent over a metric whack ton of 32 mega transfer per second ECC RAM, giving us a total of one terabyte of system memory. Shouldn't be a bottleneck, right? I don't think so. Should be fine. Then for our drives, we're using Kyoksia CD6Rs. They're a well-balanced enterprise Gen 4 NVMe drive. And with 12 of them in here, we should be looking at raw sequential performance of around 75 gigabytes a second. Before we can set up an array though, we need to install the G-RAID Supreme RAID software. And also we need to like finish putting all the RAM in this system. It still blows me away how low profile of a cooler they can use for these 280 Watts CPUs. But that's the thing is under this giant heat spreader, all the dies, right? It's a chiplet design. So they're actually like freaking spread out. Like it's huge. It's a lot of surface area to transfer that heat. Brother, these are 80 watt fans. Consume 80 Watts each? 12 volts, seven amp. I think that might be part of the equation. Wow, that is a heavy vapor chamber. I love it. For how small this thing is, like, oh, it's a heavy boy. Oh my God, this is a lot of memory. It's gorgeous. A freaking terabyte. Why are you whispering? Cause it's a terabyte of memory, I don't want to wake it up. As of recording the video, this only runs on Linux server operating systems. So we're going to be firing it up with, do we have an SSD in here or something? Yeah, back here. Oh, cool. Yeah, just a little SATA SSD there. Cool. So we're going to be running Ubuntu server 20.04 LTS. Jake has already gone ahead and installed that onto our boot drive, as well as the required Nvidia drivers and Supreme rate itself. This is cool. I am liking this like super over-engineered airflow director here. I'm pretty sure they just ripped it off of the Dell, but. Oh, all right. That goes, hey? That's not even close to full send yet either. The whole process was surprisingly easy. I literally just had to copy paste some commands from the user guide and it looks to be working. So I think we can make it right now. Should we start with RAID 0? Obviously. We have to start with RAID 0. RAID 0 is not really a great use case for this because RAID 0 has no parity data to calculate. It's just taking each bit, writing it to the next drive in the sequence and attempting to multiply your capacity and your speed. You get no extra resiliency whatsoever. In fact, it's worse because if any one drive fails, all the data is gone. Oh, hey, look at that. You can totally use this with Seidon SAS drives too. I don't know if you'd want to because I think the limit is 32 drives per controller thing. I don't know. I wonder if you can put multiple. I, it looks. Why? Yeah, good point. Why, why, why? I don't know. It's for NVMe. Yes, okay. So here, let's see. List NVMe drive. Let's see if they all show up. So we got 7.7. It's probably tibby bytes. Or what capacity are those drives? 7.68. Oh, so then that is terabytes. Cool. There's a bit of an, it's not interesting because it's very similar to like ZFS, but there's kind of a structure. So you start with your physical drives. You know, you got your NVMe drives. You can also connect NVMe over fiber drives, which is pretty cool and have the controller in this and your drives and some other JBOD. Pretty sick. We're not going to do that yet. There is still a limit of 32 drives. So it's not like you're going to connect 200, but 32. With that done, you can go ahead and make your drive group, which is kind of like a ZFS Zpool or just, it's like your array. You can pick your rate level at this stage. You can have multiple drive groups. I think you can have four. So you could have like, say you had 16 drives, you could have like four groups of four. Those would all be discrete pools. Unlike ZFS, they're not, it's not like having four VDEVs that you then combine into a pool. Yeah. It's like having four pools. Yeah. Let's go back a bit and actually create our physical drives. It says create. That's the command, but really it's like, take me, take me over. We're unbinding it from the operating system. And giving it to the G-Ray controller. Got it. There's a cool little feature here. You can go like dev slash NVMe zero to 11. Oh, that's cool. And it makes them all. You don't have to do it one by one. There you go. Made them all successfully. And then we can just check the list to see if they're all there. Cool. Yes. This is pretty solid documentation from what I've seen so far. Compared to the documentation for Microsoft storage spaces. Oh God. Compared to the documentation for anything Microsoft. All right. Create drive group. We're going to do RAID and then PD ID. So that's a physical disk. So it'll go zero to 11. It's doing stuff. See if we can see it now. G-Ray list drive underscore group. Ha ha. 92 terabytes. That's not bad. Just like that, hey? Yeah. That's fast. Now a little bit of an interesting tidbit here. We don't have a usable file system yet. This is just the array. It's not like ZFS where there's a file system built in. Instead, we actually have to make a virtual disk. So you can make a number of them. You could have like a 10 terabyte one. You could have like a 50 terabyte one. Sure. We're just going to make one that's the whole thing. This is just block level storage. Yeah. So we'll make big virtual disk that's the full size. And then we'll have to put a file system on it as well. So let's do that. Create virtual drive. Okay. So our drive group is zero. So we'll say zero. And then I'm not going to specify a size. And I think, yeah, that'll make a full size one. And there we go. We have our virtual drive. It says it's optimal and it's 92 terabytes. Okay. Now I got to make a file system on it. I already like made these commands so I can just copy past to them. Got the file system working. I think it was a little bit angry about how I had previously had a file system on these disks and then just deleted it, made a new one. Anyways, I deleted everything, rebooted it, created it again. Now it's happy. I also realized that because we now have two dims per channel, previously when I was just kind of tinkering with this, I just had 16 sticks in, which is eight per one per channel. Now we have twice that. Usually that means your memory speed is going to go down. Unfortunately on the gigabyte servers, you can force it to be full speed captain. We're doing 32 Q depth, one meg sequential read. And that's with 24 threads. So two per NVMe drive. That's usually pretty standard. Oh, I heard it spun. Oh, holy. It is straight up just immediately. Okay. It did go down a bit. It's still twice as fast as what we've seen with CPU raid though. It's only like, the CPU usage is only like, and it's barely touching the CPU. Three, four percent. That might even just be like FIO, but actually it says it's system stuff. So it's probably the system stuff. That's crazy. It looks like we've, you can hear the fans going though. Yeah. It knows it's doing something. It looks like we've leveled off around 40 Gibby bytes a second, which is, yeah, it's basically twice what you could get was that effects pretty much. That's insane. What's going to be more interesting is the rights. Cause in like traditional raid five, it's really CPU intensive to write. You'll get like a 10th, the performance of your read speed. So if that's still good, that will be the truth. Well this is raid zero anyway though. Yeah. So do we even care? Should we just switch to raid five? Let's switch to raid five. Okay. Okay. To be clear, there are potential disadvantages of going with the solution like this. One of the great things about ZFS is its resiliency. We have actually made significant progress. Big shout out Wendell from level one techs, by the way, on the data restoration project from our failed ZFS pools. And we're going to have an update for you guys soon. We're down from like 169 million file errors to like 6,000. So make sure you're subscribed so you don't miss that. And I cannot necessarily say the same thing about whatever this is doing. Well, the other thing is we're also locked into like their ecosystem. Now this is very like proprietary software. We were able to take these ZFS V-devs and pools and just like import them into TrueNAS. Yeah. Even the ones I just did Delta one and two, those pools are from like 2015. New hardware, new software, new version of ZFS. I imported those 2016 pools. It took like literally 30 minutes for it to import, which is a scary 30 minutes, but it just, it did it. Do you want to do raid five or raid 10? Raid five, raid 10 is lame. I mean, it might be lame, but it's fast. Well, no, it's not. Okay. I shouldn't say it's lame. There's a time and a place for raid 10. Let me walk you through. With raid 10, I get only six drives worth of capacity. That's it. The rest is all redundant, which is great. Cause all of these could fail and I'd still have all my data, but it's bad. Cause that's expensive with raid five. I get 11 drives worth of data, but I can only sustain one failure on these kinds of solid state enterprise class devices though. They usually all fail at the same time. Should be okay. Yeah. You're going to want to have a backup. You got a backup. That's what I mean. What's our backup within about 30 seconds or two minutes or something like that anyway. Yeah. Yeah. Yeah. We're on our raid five array now. I'm going to be doing basically the same test. Hey, wow me. 32, 32 cues, one Meg read sequential. World of Warcraft, my whole face. You see, it's only 18 gigabytes a second. That's not that fast. Wait, what? Really? 18 gigs a second? Oh, what's our CPU usage? 1.7%, 1.9%. Okay. Wow. But wait, now it's getting faster. Wait, what happened? Did you know that was going to happen? Yeah. Oh. Well, I watched it happen earlier. It does like two steps. Okay. So it's written, you know, like 30 gigs and then it starts going. And then there'll be kind of one more bump where it'll go like above 30 gigabytes a second. Wow. That's freaking crazy. With raid five. Still at 2.5% CPU usage. There we go. 35 gigs a second. We're at almost 3% CPU now. Remember this is a read test. So it'll be interesting to see what the write is like. Yeah, that's pretty quick. 35. Yeah, it's really fast. 35 gigabytes a second. Are we switching to this? We might switch to this. I want to try to put it in the Wannix server because then we can use the same SSDs. Right. Yeah. That would be even faster. But I'd have to do that on a weekend. That's Jake asking for overtime on camera. Yeah. No, I don't want to work on it. I have zero desire to do that. Okay. Let's try write now because that's really where we're going to see the difference. Wow. CPU usage is like 20%. That's pretty hefty. Yeah. And we're only doing five gigs a second. That's not actually that great. That's a lot less impressive. So CPU usage is actually going down while performance goes up. Yeah. It's more like 12% CPU right now. 11, 10. It's like it takes a second to like... What am I doing? Where am I putting stuff? Yeah. Like it needs a ramp up. He's over there. I'm coming back. Okay. Seems like it's leveled off around nine gigabytes a second with around nine to 10% CPU usage. So pretty good CPU usage. Still very strange. I mean, it's very acceptable in terms of performance. I mean, that's a Gibby, Gibby or Mibi bites. So it's probably closer to about 10 gigabytes a second. Should we try like a random test? It'd be interesting to see how many IOPS because that's another thing software RAID will struggle with. Yeah. Let's do it. Look at this guy. He's like goes wide stance here. Man spreading. Just trying to be a little more ergonomic here. This is gonna be a 4K random read. We're doing 48 threads, a little bit more and a Q depth of 64 this time. Okay. Let's see. Oh, that's CPU usage. That's CPU usage though. This is like the absolute most punishing test you can do. And we're pulling off six and a half million IOPS on a RAID five and actually 25 gigs a second at 4K block size. Holy. That's insane. Oh my God. So the theoretical performance of these drives would put us at around 12 million IOPS, like raw to each of them. That's insane. Pretty good. If we were on an Intel based system, we might actually be able to get a little bit more or with Intel drives, but yeah. Dang. That CPU usage is staying high. I can tell you just from the temperature of the back plate though, that GPU is at work. We can look at it actually. So it's at 70 degrees. And considering the kind of airflow going over it right now. The interesting thing is the GPU usage just stays at a hundred percent even if you're not using the array. Kind of weird. I wonder if that's like a... Look, the fan is spinning. 55%. It just has like, it's like you have like a little desk fan inside of like a hurricane. Yeah, or a wind tunnel just going past it. Oh, thank you for the cooling. Yeah. Let's try writes. Same specs, everything else. Let's just give it a sec to... No, no sec. No sex? No. I mean. I mean. That's a camera. And... Not the kind of operation where that's a lot slower. Yeah. It's writing though. That's way harder. So doing 1 million IOPS writing. So these drives are only rated for 85K random writes. Well, that's really... Wow. So that's actually almost perfect scaling. Yeah. That's fricking incredible. So if we do 85 times 12, it's almost perfect scaling. That's probably the most impressive test we've seen so far then. Yeah. That's crazy. What's cool about the writes still maxing out these drives though, is that because people are actively editing off of these drives, while you are dumping copious amounts of data onto them, this could make a huge difference to footage ingest. This is fricking crazy. We're still gonna run into a huge bottleneck that is Samba, but maybe once they have SMB direct on Linux, like RDMA support. We kinda have to deploy this. Yeah, I think so. Let's find out if we can. Oh yeah. Okay. So I got a- That's why the bench is here, right? I got a Threadripper bench here. We're gonna just put that card in it. I got us a little NVMe drive. We're kind of clashing brands here. This is like wearing Adidas and Nike in the same outfit. We got our liquid drive. This has got four NVMe drives on it, and then just like a little PLX switch. So we'll put that in there and then we can raid those four drives. Okay. It's even less sophisticated than I thought. Yeah. It's just- That's not on a normal card. There's no like- It's just this piece of metal. So then they just probably sourced a random PCI slot. It looks the same as the regular Nvidia one, cause like these slots are the same, but there's no cutouts. Okay, sure. I'm gonna go get a cooling fan, cause that looks awful. Which one? The one in there? Oh, he's fine. He's not fine, Jake. He'll be all right. He's a good guy. So let's see, describe license. Let's see if our license is still valid. License is still valid. Well, that's awesome. Interesting. Okay. Yeah. It sees our NVMe's. Interesting. Oh, these don't support 4K block size. Oh. Do we- I think this is all we really needed to know anyway. It boots, it runs. It probably works. Let's just boot into Windows and we'll get the last answer. Holy it's there Linus. What? Nvidia T1000 in device manager. It's just a freaking GPU. I just want to see if this BIOS version matches any of the BIOS versions in the tech power up database. Because if it does chances are, Oh, hello. Yeah. I'm pretty sure that just worked. G sync display connected. No way. Oh my God. Just works. It's a graphics card. Well, I mean, we knew it was a graphics card. It's a functional display outputting graphics card. Okay. I got to see this BIOS thing now. It's a PNY T1000. So they must be who the manufacturer- Yeah PNY is the exclusive manufacturer of Quadro, or excuse me, Nvidia RTX. Workstation. Workstation. It's exactly the same at every other metric. GPU accelerated raid. What world are we even living in? And it's just a regular ass GPU. And it's not even a crazy powerful one. Like this is a basically what? Like a 1650 or something like that. It's literally a 1650, same silicon basically. So what can we just run? Can we just run G raid on like freaking like a 6,000? So maybe it just completely doesn't care what GPU. Should we put a different GPU in that? Should I go get a GPU? There's gotta be some validation. I'm going to get a GPU. Okay. Well, I'll be back. Do we have an A anything? I'll just get like a Quadro or something. Or a T would be better. Touring kernel. Excuse me. Jesus. It took a little while to generate the rendering kernels, but it blenders. Oh, I think I found something. It's pretty fast too. Oh my God. Are we going to have G raid? Check that license. Me just- I want to know. So let's see. Let's first see if the server is okay. It's not running on the GPU. You see it would, it would show the running process. So let's see. It might not work. I don't think it's going to. Bummer. That would be so funny. I think the license is done per GPU. Oh, I wonder if it, as part of applying the license, it binds it to it. It's relatively unsophisticated. It makes you kind of wonder why they would even bother at that point, but- Money? No, no, no. I don't mean that like licensing is a waste of time. I just mean like it's relatively unsophisticated if you wanted to spoof that. Yeah, you can do it. Yeah. Huh. Oh, I'm disappointed. I wanted to like throw eight times the compute at it and see what it would do. Oh well. To be fair, the game is not launching. Watch G raid reach out to us after this and be like, yeah, we can hook you up with that. And there we go. Continue campaign. Sure. 5%. How do I play this? With a controller. This is a good game. You should totally play it though. It's really fun. Is there actually no way to play it with a keyboard? I don't know. I've never tried. It kind of looks like- Try like WSD or like shift. God damn it. All right, I'm glad I told Rocket League to keep downloading. This doesn't launch either. What the hell? Yeah, the T1000 is behaving kind of weird, but it can game. We launched Broforce. It's fine. And you know what else is fine? Our sponsor. Thanks to Telus for sponsoring today's video. We've done a ton of upgrades to my house over the past few months. And probably the most important one to me is our new Telus PureFiber X connection. Telus PureFiber X comes with Telus's Wi-Fi 6 access point and can get you download and upload speeds of up to two and a half gigabit per second. That is the fastest residential freaking internet speeds that you can get in Western Canada and is perfect for multiple devices and simultaneous streaming. With the new consoles out, it means that you can download a new 50 gigabyte game in less than three minutes or a five gigabyte 1080P movie in just 16 seconds. Assuming that the connection on the other side can keep up with you. You'll also get to enjoy an upload speed that's up to 25 times faster than competitors, which means that your streams will be crystal clear. I mean, you can be streaming in like fricking 8K at that point. So get unparalleled speed on Canada's fastest internet technology with Telus PureFiber X by going to telus.com slash PureFiber X. If you guys enjoyed this video, this little SSD from Liquid here is hardly their most potent. We built a server using five, I think, of their way bigger eight SSD honey badgers. It's called the Badger Den and it's freaking amazing. 100 gig to six. That's crazy.