 As many of you probably know, the last few years have been a heck of a ride for data enthusiasts, with high-speed storage becoming increasingly accessible. I mean, something like Samsung's 960 Pro SSD. It's tiny, it costs about a third of what I paid per gigabyte for my first large boot SSD, and it's over an order of a magnitude faster when it comes to real-world performance. But what some of you might not know is that networking has been keeping pace. So these bad boys right here are Mellanox ConnectX 4 cards, and even though they're already two generations old, they can reach transfer speeds of 100 gigabit, or about 12 gigabytes per second. That is fast enough to download Fortnite's install files, yes we went there, in literally one second. Holy sh**! So let's check them out. Music Before we get to the new crazy stuff, let's do a quick refresher on traditional networking hardware. So I've brought along a couple of examples here. These two cards run at gigabit and 10 gigabit speeds respectively. So this guy right here is about 10 times faster than this one. But other than that, they've got a lot in common. So they've both got Intel controllers on board, they both plug into a PCI Express 2.0 slot, and they are both Ethernet. So thanks to their use of inter-compatible communication standards and the ubiquitous RJ45 connector, they can talk to each other directly or through a network switch like this one, albeit only at the speed of the slowest link in the chain, be it on this 1 gig card or through this 1 gig network switch. And honestly, either of these, especially this one, should be more than enough for the average person for quite some time. But this isn't average person land, which sounds like the world's most boring amusement park. So we've decided to go totally overkill and take it to the next step to greater than 10 gigabit speeds, which brings us then back to our Kinect X4s. So the first difference, the ports. This beefy looking thing right here is what's called a QSFP Plus connector. And as you probably figured out on your own, you can't just plug a standard network cable into this port. And even if you could, well, I guess that brings us to the next difference. The fact that out of the box, these cards are designed to run not on Ethernet networks, but on InfiniBand networks. So even if you could plug it into your network switch, it wouldn't be able to communicate with it, without some configuration. And then finally, this one's actually pretty interesting. These cards, yes, my friends, these network cards use a full fat PCI Express Gen 3 times 16 connection. That is the same as your graphics card. And they actually need it, probably more so than your GPU, if all you're doing is gaming. So then, with that in mind, we will not be using just your average gaming machine for our testing. So on one side of our link, we've got Intel's flagship 18-core processor with a RAM page 6 Extreme motherboard, 128 gigs of RAM. And then in the other corner, we had to slum it a little with a 16-core 7960X and an ASUS X299 Deluxe, but then with the same amount of RAM and of course, it's RGB across the board. Now the reason that we're using the X299 platform with Core i9 processors, is that we need to make sure that we have enough PCI Express lanes coming directly off the CPU. So 44 should give us enough for 16 lanes for networking, 16 lanes for our quad NVMe storage devices, and then, you know, some leftovers for the graphics card. So then, for our NVMe storage, we scraped together four Samsung 960 Pros for our first one, and then four Corsair MP500s for the other, with both of them running with quad SSDs in RAID 0. Today's video is about how to go fast, not about how to put on your seatbelt. This is like hilarious to me, I never thought I would see the day when the GPU in a gaming rig is the lowest priority like, bitch tier PCI Express device sitting at the bottom of the board. Anyway, for our OS, we had to go with Windows Server 2016, because as much as we wanted to try out Windows 10 Pro for Workstation, which is supposed to support RDMA, the tech that allows for these super high-speed transfers, it just didn't seem to be working for us for some reason. Alright, now, at this point, before we go further, I want to give a big shout out to the guys over at Mellanox for hooking us up with these 100 gigabit cards, as well as a pair of 100 gigabit capable passive copper direct-attach cables. Wow, that just hit the bench. So if you want to try this out at home, these cards are actually available on eBay for like 200-300 bucks a pop, and then you'll pay about $60 for a 3-meter cable like this one. It is worth mentioning though that if you're planning on running anything further than 5 meters, you have to use an active fiber cable, which could cost upwards of $2,500 new. So... Damn. Alright, moving on. Configuration then is our last step. So while Jake does that behind me here, thank you, Jake, let's talk about some of the technology behind this networking magic. So these cards are designed for use with two different network fabrics, InfiniBand and Ethernet. And what makes InfiniBand special is that compared to even the sub-millisecond latency of a typical Ethernet network, InfiniBand networks can have less than 25% as much, making them suitable for use cases like over network storage access and combining the processing power of multiple servers, just like in a data center or supercomputer. And this is cool. When you configure InfiniBand correctly, it also forms what's called a lossless network, meaning that packet loss should basically never happen. For compatibility though, we're going to be using them in Ethernet mode alongside a technology called RDMA or Remote Direct Memory Access. When you put these together, the setup is called ROCE or RDMA over Converged Ethernet. Now, regular Ethernet implementations require a lot of hoop jumping in order to transfer data, as any information sent must be first moved through the transport protocol's driver, then through sockets before it can reach the application's memory, eating up CPU cycles and increasing latency in the process. However, with RDMA, the network adapters are able to access data directly from application memory, offloading much of the processing from your CPU onto the actual processor that sits right on your network adapter. So these are known as zero-copy transfers, and they allow for incredibly fast transfers that are no longer limited by CPU processing power. Pretty dang snazzy. So, are we ready to go? We should be. Each card has an IP and a 50 GB RAM disk, so we should be able to do some pretty quick Windows transfers. So, are we going directly from RAM disk to RAM disk right now? So, I think what we'll do, we got to see, make sure it's working first, right? That would be good. I mean, that's supposed to be your job, but... Fingers crossed. You never... The second you try to do a networking demo, like... So, we got our 40 GB text file. What? 36 GB, or 37 GB. Why? Why even? It's just a big file, okay? Oh, that's not bad. That's not 10 GB, or 100 GB. So, we've got just over 2 GB per second. So, we're reading from that system's NVMe array. Okay. And then we're dumping to the RAM disk on this system. So, let's try going the other way. So, now we can go RAM disk to RAM disk. Is that right? No, that's still the same thing, but just the other way around. Whoa! Just shy of 4 GB per second. I mean, let's put what just happened there in context. That is a 40... Well, just shy of 40 GB file. So, like... Okay, Doom. Doom on PC is like 60 GB. And it takes, you know, probably for your internet connection at home, like what? An hour to download and install? That just happened in real time. Okay. What's next? RAM disk to RAM disk? I gotta check. I think one of our RAM disks isn't working. I might have broken something. Okay, so check this out. We've got the RAM disks working, but for now, this is just another quick benchmark. So, when we aren't limited by the overhead of Windows File Explorer and Windows File Transfers, if we're just using a straight disk performance benchmark, I want to show you the kinds of numbers we're looking at. So, anyone who's familiar with Atto is going to already know that this is freaking nuts. At 2 KB sizes, we are already seeing speeds in excess of 100 MB per second, because we are actually reading and writing off of the four Corsair SSD array that's on the other machine. This is stupid. It's pretty cool, actually. It's really cool. And it's getting stupider as time goes on. We are already hitting 600 MB a second writes. That's weak, man. At 16 KB though. Just wait. That's the key. And it just keeps getting crazier. We're at 1 GB per second already. Three! Doubled! 3 GB a second writes. Oh, wow. Oh, man. We just cracked 4 GB a second. We just cracked 5 GB a second. And this is on a remote machine. This is not a local array. This is over the network. This is a slow test. Yeah, it's a really fast slow test. Yeah, it takes a while. A really slow fast test. It's in the billions of bytes. Billions of bytes. It's like not even readable anymore at that point. No, it's just like what is going on? How do I math? Oh, meanwhile. Are we there? Just cracked it. 10 GB a second. Okay. So what's our next test? To RAM disk, I guess. Sure. Let's do it. So what's interesting here is that we actually hit our peak speed earlier, but then we level off to the RAM disk. Yeah, but our writes are like way better. And if we look at CPU usage, it will show in here processor time was almost zero the whole time. Crazy. So like if you're using traditional Ethernet without RDMA, that would be like pretty high pegged up there. So why don't we do as fast a Windows transfer as we can now? We'll go straight to the RAM disk. Sure doesn't take long to move 40 GB files around with this. Even on like a relatively slow. Relatively slow. 2 GB a second. How will we ever manage? Painful. Yeah. So about the same. Yeah. So it's a Windows thing. Pretty much what's going on here is that we have reached because we've seen faster transfer speeds in Atto. So we have reached pretty much the limit of what Windows can handle for the time being. And I don't think it's going to be in Microsoft's list of high priorities anytime soon to figure out how people can use Windows Explorer to copy files that faster than about 4 GB a second. But that's okay because this was a lot of fun and hopefully you guys enjoyed it. If you want to see what this kind of tech gets used for in the real world, because this is just not what it's really for, check out our kind of unboxing of SFU's Cedar Data Center. We'll have that linked in the description because that's this kind of technology on a whole other level. With that said, I mean even for someone like us, actually it could be useful. Maybe we could try to rig up like a crazy way to improve the responsiveness of scrubbing in Adobe Premiere for the editors or something like this. 100 gigabit for all the editors? Yeah, 100 gigabit. Well, no, but we could have like a 100 gigabit trunk. Yeah. And then they could all go 10 gig. It's bonded 10 gigs. Oh, like... So, you know what? Tell you what. Let us know in the comments if you'd like to see a video on what we can figure out to do with these in an actual deployment. So, thanks again for watching guys. If you disliked this video, hit that button. But if you liked it, hit like, get subscribed, or maybe consider checking out where to buy the stuff we featured at the link below. Probably eBay links, I guess. While you're down there, you can check out our merch store, which has cool shirts. Like, I don't know how long this one's gonna last, but it's gonna be up there as of the time of shooting this pretty soon. I think it's funny. And also, we have a link to our community forum, which you should totally join. That shirt. I know, right?