 This is an SSD. No, no, seriously. This whole thing is a complete SSD, and it's designed for massive data centers like what you might find at the likes of Meta. So how come I've never heard of it? Well Pure Storage wonders that too, and they've sponsored this video for us to take a tour of their facilities here in sunny Mountain View, California and dig deeper into just how these things work. And maybe while they're at it, they can explain a little bit about how they can make the slowest flash used in SSDs today run twice as fast as last year's good stuff. Let's take a look. From the beginning, Pure Storage's secret sauce has been in their approach to software. While their solution is proprietary, with it, they're able to tightly control each individual NAND flash chip in an array potentially the size of a whole rack. But there's no way that I know of to drill down to that level of detail with any normal SSD. Like even enterprise-grade SAS and U.2 drives have controllers that make the drive show up as disks. Like you don't get direct access to the chips. So what's up with that? This is a direct flash module, or DFM for short. It's a strange-looking module. It's a lot larger than any SSD form factor I've seen before, and there's a lot of NAND flash on there. Up to 48 terabytes in the current generation, along with supercapacitors to make sure writes complete properly in the case of a sudden power loss. Pretty standard for data center SSD, yes, but while you'd expect loads of DRAM cache to make it fast, there's no DRAM on here. That is a terrible thing for performance under normal circumstances, just because of the way SSDs work. Whenever an individual die is busy performing some action, the data on it is completely inaccessible and you have to wait. DRAM caches collect the data and delay writing it until there's enough of it that it actually makes sense to spend the time actually writing it out. Because in order for an SSD to write that data, any blocks that aren't totally empty first have to be erased. As you add additional levels of storage to a cell, so from single to multi to triple to quad, the time it takes to actually do this increases exponentially. And if you remember from our review of Intel's early DRAMless QLC drives, you would know that it can be slower than a hard drive. This could be catastrophic for real-time applications if your DRAMless array decided that it was time for some spring cleaning. So have pure storage just gone completely mad? If so, well, there may be a method to their madness. Up to four of these DFMs can slot into a blade like this one, which itself is a self-contained server running a socketed Xeon processor. There's the DRAM. Yes, this Xeon CPU is acting as a glorified SSD controller. Now it does other array management stuff too, but to talk to the storage, it actually runs a very low-level interface bootstrapped from Linux that talks to a custom controller on each DFM. These controllers do one very simple task, provide an interface to all the flash on the module. It doesn't really do anything that you'd expect from a typical SSD controller. All of that is handled by the CPU. While it does use the MVME protocol and a U.2 connection, it won't show up as a disk in any other system simply because it doesn't have the brain. This arrangement lets them directly control when, where, and how new data gets stored. It's kind of like Apple Silicon in a way, except they can handle multiple vendors of flash in the same array. More importantly, there's no pretending to be a hard drive. So while block sizes can vary between vendors, you're not forced to write out a whole 4 kilobytes each time you need to write a couple of bytes. So because the blade knows what the whole array looks like, data that's changed very frequently can be grouped together to prevent a situation known as tomb stoning, where deleted but not erased data can stick around because it's uneconomical to evict it, which means that that cell is effectively never touched again. That's partly why over-provision space, which is where a percentage of the raw capacity is reserved for system use, is important for SSD performance, a workaround that Pure Storage doesn't really need. In a nutshell, each of these blades is basically an SSD with extra steps. Because of that, Pure Storage is able to control things like when and where to start where leveling, which is incredibly important to QLC flash because it's typically only rated for less than a thousand program erase cycles. So obviously there's a lot of incentive to never have to cycle a cell if you don't have to. To help with that, data is compressed wherever possible. There's another component to this too, because you can only ever read or write flash memory, not both. The more you can avoid writing to frequently read cells or overwriting frequently written data to those cells, the less often you're stuck waiting for the SSD to come around and actually spit out the data you need. And that's especially important with QLC because of how long it takes to erase and rewrite data on that type of flash. Remember, at this kind of scale, you're talking databases that are massive enough and accessed often enough that even a delay of tens of milliseconds could be a major problem. And of course, they've thought about that too. What if I told you that in the time it takes for a write cycle to end, the inaccessible bits can be rebuilt from parity data? Yeah, that's not something you can do with a traditional SSD. The controllers that pretend that they're still spinning disks for the PC's sake quickly become the bottleneck. Let's meet a complete system and see it at work. This is a FlashBlade S chassis, Pure Storage's newest product. Each of these can hold 10 blades with 4 DFMs each, which at up to 48 terabytes per DFM is a staggering 1.9 petabytes of storage in a single 5U chassis. Remember the lengths we had to go to in order to get a petabyte of hard drives? And then how many boxes a petabyte of flash took up? This is next level. There are four power supplies on this thing that are all modular, each with the capability of drawing 2.4 kilowatts from the wall. And while they don't sell them right now, Multi-Chassis Configs with up to 10 of these things in a rack are in the pipeline. A fully loaded FlashBlade S has enough redundancy that it can lose a whole blade and a DFM and another blade and be totally fine. The blades connect to each other via the same backplane that the blades slot into, called Fabric I.O. modules or PHYOMs for short. Each FlashBlade S has two PHYOMs, which automates managing the blades and chassis units with a total of 8 100 gigabit per second lengths, only four of which are currently used. They say that when they're ready to enable the others, they'll be able to do it for free via a software update, something they're able to do thanks to their Evergreen model. There's a lot to this, but basically they don't charge for new software features, and their modular approach to hardware means that you can continuously upgrade to the point where one of their Alpha stage clients from 10 years ago has continuously upgraded their array with no downtime to this very day. It's like the array of Theseus, none of it is still the same except for the data. And even that might be different now. But enough talk. Let's see them roar in the data center. This is a rack full of the previous generation versions of what we were just looking at a little while ago. These are all connected together with a fabric like this is I think all fiber, and they're all connected via these back planes here. So this top one here is connected to all of the cables. This bottom one is mostly there's some tape here as well. And all of these are communicating with each other in a maximum layout. So this whole rack is acting as basically a single SSD. As you can see, all of these power connectors are modular. The actual power cables themselves have tabs on them. So you can't pull them out. So yeah, this is all of their last gen stuff. This over here is a rack full of their new gen stuff. This is Class plate S. What we're looking at here is more or less a setup of a bunch of standalone units. Rather than having everything built together into one giant array, each one of these is a smaller array. Compared to spinning disks, this is actually pretty good power density. Like for the amount of storage you're actually getting here, it's significant savings. But it's something like 1.3 watts per terabyte or something like that. This is one of the first modules that they actually built for this type of array. It actually has an FPGA on it. It takes all of its information from the SD card here. And that is a mini USB port. That tells you how long ago this was. But this was NVMe before pretty much anybody was using NVMe. It uses a U.2 connector here to connect directly to the backplane. Basically, they had to create a PCI Express backplane for this because actual controllers, HPAs, didn't really exist at the time for this kind of storage. And I was just handed one of these. This is a trameless version. It's been taken out of the tray of the DFM that's currently being used. We can see that there are super capacitors on here in order to make sure that in case, say for example, one of these were unceremoniously janked out of the server, any data that was already being written will be ripped. So it won't be partially written. It won't get corrupted in that way. Also, on the back, we can see here that we have extra NAND flash chips. This is a 48 terabyte module. 48. That is a massive amount of storage. No hard drive could possibly even come close to this within the next... I don't know how long it would take because currently at the rate of growth for hard drives, it would probably be another 10 years or so, assuming hard drives are still relevant at that point in time, which is something that beer storage is trying to actually avoid. They feel that flash storage is inherently superior to spinning disks. And quite frankly, from what I've seen so far, it definitely kind of looks like it. You get better density, better power draw, and you get a much, much lower level ability to just manage the data on the drive than you would get from a traditional type of hard drive. Now, don't be asking yourself, who could possibly leave this much storage? Well, the answer, Al, there's rails and stuff here, so this is great. So the answer is anybody who's into deep learning. So this is an NVIDIA DGX. This is one of the things that beer storage has actually partnered with NVIDIA for. Meta recently partnered with beer storage and I think also NVIDIA for their AI deep learning platform. So the DGX is powering the brains, whereas beer storage is powering all of the data sets that they need to crunch through. So you're never going to get a GPU with petabytes of video memory, at least not in the next, I don't know, several decades at least. So what we're looking at here is an array that's fast enough to keep up with those deep learning workloads on those GPUs to keep them fed so they can actually be doing their job more often. Otherwise, what you would need is a massive amount of memory for the system to cache that kind of thing, which is just impractical. So here we have NAND Flash doing that exact job. It is very responsive and very dense. So for those reasons, basically, if you have a huge deep learning workload, this is pretty much the premier solution right now, as far as I can tell. Both NVIDIA and beer storage do you think so? What we're looking at here is an example of an NVIDIA DGX A100 in action. Right now, this is actually computing deep learning data, and it's connected via 100 gigabit per second links through these switches here to the beer storage arrays. So there's the previous generation FlashBlades, and I believe there's also a FlashBlade S there as well, which is the current generation. I think they're doing performance testing on those right now to see which is faster and how to basically optimize for these workloads. It's pretty amazing that we're basically at a point where SSDs aren't fast enough, and yet the SSDs that we're placing them with are technically slower because they're using QLC memory instead of TLC, which is just mind boggling. It's supposed to be an order of magnitude less efficient, and yet they're making it work. Now you have to think to yourself, these are full computers in these blades and server chassis that are basically just being used as SSD arrays. What happens after the SSD array is retired, like when you no longer need it or when you upgrade to a new one? Is the whole chassis just kind of thrown out? Well, no. What beer storage is doing is they have a whole bunch of completely empty chassis here running virtual machines and other workloads that don't require that kind of storage. So they're repurposing those Xeon processors that would otherwise just be, I don't know, like if you retire the SSD and put it on your shelf, what does that make it? So in this case, these moved on even if the Flash has died or it's been upgraded because it was too small in capacity. In fact, behind you there are a whole bunch of processors and RAM and other stuff that's just sitting there waiting to be tested or reused. Now you're like wondering how you even manage to talk to these things. Well, the file system that they use, I say file system very loosely. It's actually more like a database that you can actually then create what they call authorities. On top of it, 128 of them split across the entire array. The larger the array is, the faster they go. All of those authorities can be used to do things like create object stores like Amazon S3. And from there, you can also create SMB, so Samba, Windows file sharing or NFS for Linux file sharing support. So pretty much it doesn't look any different to the end user. You can use it however you feel like you need to use your data. So you can pull stuff directly down from Amazon for your cloud services deployments, or you can just use it as network storage if that's what you really want. And what's really cool, they probably wouldn't let me do it, but if you take out one of the DFMs and rearrange it or something, within 10 minutes it picks right back up without having to do anything special. It just brought it back in and it's like nothing happened. So you can say, for example, if you've got a blade that's misbehaving or you want to upgrade it, you can completely migrate over without having to change anything about your configuration. Your users basically won't know what happened because nothing will have happened. It's kind of magic. It's really, really warm back here, but these switches we were looking at earlier, all of these chassis plugged in, this is one of them. Not only is this a network switch, but it's also basically the same type of thing you'd find in a biome in the back of a FlashBlade S. So it's got an x86 processor in here that handles all of the communications. In fact, when you plug in multiple FlashBlade chassis, this thing takes over and actually orchestrates the entire lot of them. So they're no longer doing their own individual arrays. This thing takes over automatically. Also, each one of these ports can do 40 or 100 gigabits per second depending on the FlashBlade chassis. The older models can only do 40, whereas the newer model, FlashBlade S, that can do 100. So it's a massive amount of data that will flow between this thing and the rest of the rack. Now that you've seen the tech, let's talk about who these guys even are. Pure Storage was founded in stealth mode back in 2009 and debuted in 2011 as one of the first companies to introduce all Flash infrastructure solutions in the industry. In the early days, they used consumer-grade SSDs, which, if you look back on the state of SSDs back then, was pretty ballsy. But the solution was always software-driven, and they quickly began developing their own Flash modules, which started shipping in 2015. Fast forward to today, and they partnered with companies like Cisco and Nvidia with clients across the globe. So big thanks to Pure Storage for sponsoring this video and letting us show off their gear. You can learn more and maybe deploy one of these for yourself if you're a straight baller or if you're an IT manager at the links below. Thanks for watching, guys. Maybe go check out one of the Intel Design Center tours that Linus did a little while ago. Those are really, really next level in terms of how behind the scenes we're seeing. Intel basically never lets anybody see that kind of stuff, and I'm glad that we got to see this kind of stuff here.