 You ever spend a lot of money on something only to have it arrive broken or otherwise defective? It's a pit. Even worse is opening up the box for your replacement only to find out that they didn't actually fix the problem. And I think that's what just happened to me with over $10,000 worth of Intel's 4TB P4500 SSDs. You guys remember these? We tried to roll them in a new server deployment over a year ago, only to conclude that they were fundamentally flawed and could never work in it. Intel graciously agreed to swap them all out for us. But here's the thing, Intel. I thought it was understood that I didn't want the exact same broken thing back. And yet, here we are. Telling you about our sponsor, SmartDeploy. SmartDeploy enables IT admins to manage PCs from the cloud. You can push windows, apps, and security patches to any device anywhere without leaving your desk. Get your exclusive software for free worth over $800 at smartdeploy.com slash Linus. Let's get started. Let's get started. Let's get started. Let's get started. Let's get started. Getting Intel to agree to replace these took a very long time and wasn't straightforward. As far as I can tell, their initial plan was to do nothing about my problem until I pulled strings with my media contacts. In fairness to Intel, these are secondhand drives I bought on eBay. But also in fairness to me, they are still within their warranty period. So my request isn't totally unreasonable. And besides, while Intel has never publicly acknowledged any design flaw with the P4500, I've heard of other customers even getting cash refunds as compensation for their trouble. And if you've ever dealt with a computer hardware returns department, you'll know that cash refunds ain't the kind of thing that happens when it's the customer that screwed it up. So let's take a closer look at what they sent. I don't actually know what to say here. As far as I can tell, everything about these is identical to the drives that I sent back to Intel. Model number, capacity, yeah. It's, they're the same thing. Look at this. These actually have marks on them that would seem to indicate that they've been installed in drive sleds before. I'm not actually mad about that though. It's pretty typical to get a refurbished product back when you send in an RMA. And it's not like they still make this drive or anything. Just not exactly confidence inspiring. And come to think of it, do I have any way of knowing that they actually sent back different drives and didn't just box back up the same drives and send them back to me? You dealt with the RMA for this, right? Did they ask any questions about the issues we were having? They just processed the RMA and sent us back those. Do we even know these are new drives? I have no clue. These came from Latin America. Oh, right, of course. We sent them a whole list of serial numbers before we shipped the drives. So, oh yeah, 720 versus 807. These would have been manufactured way later, assuming they didn't tell serial numbers are sequential. Okay, so they are different drives at least, but after double checking with Wendell from LevelOneTags, who helped us diagnose the problem in the first place, he seems to be under the impression that nothing about these new units should change the problems that we had before. And they were big problems. Individually, the drives behaved fine, but as soon as you built them into a larger array of some sort and then hit it with any kind of sustained load, even just copying a file would cause individual drives to randomly fritz out or drop out for a few seconds at a time and then reappear, absolutely tanking the performance of the whole array. And we tried everything, Windows, Linux, ZFS, dropping down to PCI Express Gen 2 speeds in the BIOS. Wendell eventually narrowed it down to the way this drive issues interrupts to the CPU. If you don't know, an interrupt is basically a way for a device to tell your computer's processor, hey, I've got something for you, come pay attention to me. Now, since this is a storage device, chances are that the thing it's got is some data. But small problem, if the CPU were to respond to that interrupt too quickly, which is what we saw in Linux, there's a chance that the data won't be in the drives buffer yet. Missing data, uh-oh. So the Linux kernel would keep retrying the drive to see if the data turns up. And lo and behold, it would, totally uncorrupted, which is good, but with the aforementioned performance problems, which are bad. Now, Wendell's hacky workaround was to tell the operating system to pull the drives, essentially having the CPU checking constantly. Got anything for me? Got anything for me yet? How about now? Do you got anything for me? He's some kind of genius and it did work, but that approach comes with a bit of a performance penalty as well. Think of it kind of like the postal service. It's way more efficient for you to schedule a pickup with FedEx whenever you wanna send a package. Instead of just having a courier show up every hour of every day on the off chance that you have something to ship. So poor CPUs checking in with 24 of these drives constantly. Not great. And obviously most drives don't have this problem. It's just that these ones do, and it wasn't just one of them. It was the entire batch of all 26 drives. So why would they send us back the same bloody thing that wasn't working? And if that's not what happened, then what on earth did they change about these? The plot thickens. This AMD Epic Roam server has been sitting completely untouched since the last time that we tried to roll it out and gave up ultimately replacing it with a Dell loaded up with liquid honey badgers. But just because we found another solution doesn't mean we couldn't use more capacity and we wouldn't like to freaking use it. It's 96 raw terabytes of NVMe storage, each with their own dedicated PCI express links. AMD Epic is a beast. I wanna freaking use the thing. So let's fire it up, shall we? Let's see if we can figure out what they changed. Obviously something they could have changed would be the firmware on the drives. But conveniently, because I haven't touched this thing, I've got the exact same version of Solid State Drive Toolbox from Intel that I had before. And I can check if it allows any kind of firmware update. Here we go. Nope, exactly the same firmware. Even though they're running the same firmware as the ones I shipped out, maybe there's been an update. We can check that. No. Okay, no, no firmware change then. I have mostly given up that this is going to do anything, but let's at least give it a chance. We can use this custom view and administrative events to see the drive dropping out or not. You'll get a little yellow warning. A fatal hardware error has occurred in memory. We'll have to deal with that later. Performance is as expected for a Gen 3 drive. That's somewhat hopeful. That's promising. Nine runs, zero dropouts. Of course, as I mentioned before, we saw the biggest problems with multiple drives running at the same time. I mean, you can kind of imagine that's how something like this makes it past QC on AMD, Gigabyte, other server manufacturers as well, and Intel's side. Everything works fine in isolation, but you open up all these, load them up with drives, and slam the PCI Express controller with enough high speed storage that you're getting dangerously close to RAM-like speeds, and well, the wings start to burn up a little, don't they? Let's try four drives next. Okay. I think I'm probably gonna have to reboot. Theoretically, they're hot-swappable, but like, no dropouts so far. I'm feeling good. Hey, you guys want a free tech tip? You notice that in a mirror, your read speeds scale individually with drives, but you only get the write speed of half of your number of drives? That's because you have to write two mirrored copies across the entire array, so you're effectively splitting your writes in half, but with reads, you can actually take advantage of all the drives in parallel. Neat, huh? Okay. Big test time. That's promising. We've got eight drives running in parity mode in storage spaces, which I prefer because it gives way more capacity than mirror. Now, you'll see a couple of these are not blinking, but that's more likely just due to the fact that in parity mode, it writes in a very different way compared to mirror. These write speeds suck for how many drives are in there, but from our experience, it seems to be a weird interaction between Crystal Disk Mark and storage spaces because we've seen better real-world performance than what it measures with. This defies all logic. I don't understand how it's working. I mean, to be clear, I'm happy. Also, we haven't actually loaded this thing up yet. Should we try 16? Oh yeah. No, wait. Okay, disk 10 has been surprise removed. Wait, what? Hold on a second. I got one of those before. I haven't removed any. I just thought it was something to do with while I was putting drives in, but I didn't take any drives out, did I? There it is. Store in VME, reset to device, blah, blah. RAID port 15 was issued. This is exactly the problem we had before. I need a drink. Lttstore.com. Prove it's water in here. Or prove it's not water. Go buy a water bottle. They're insulated. They're really nice. Let's go back for a second to the questions I set out to answer today. Number one, what did they change? Okay, well, we've got that one. Nothing. Question number two, what was up with this replacement process when they've indicated, at least to other customers, that they're aware of this problem and they know evidently that they can't, they have no way of fixing it. Because here's the thing. Any other Intel drive, at least according to Wendel, would have worked fine. The slightly newer 4501 doesn't have this problem. The 4511, which is fundamentally the same drive, but in their ruler form factor, same thing. Why send me stacks of broken drives when you know how much, like, it's not free, you know? The cost would be the same to send me something that works. Unless they do work. One of the most basic troubleshooting steps when you suspect a piece of hardware is defective is to take it and put it in another system to isolate your variables. Unfortunately, testing just one of them at a time wouldn't have told me anything because they worked fine and NVMe servers that can take 16 or 24 drives don't exactly grow on trees. So it wasn't an option for me. Fortunately, now that old one has been replaced, I can actually take these new drives, chuck them in here and see if we see the same behavior. All right. Locked and loaded, all 24 drives. Let's see if it works. Performance is not as good. Even at its best, this server has a fraction of the PCI Express connectivity and its Gen 3 compared to AMD Epic. But if the drives don't drop out and we get that consistency, that's way more important because we're only accessing this for video editing over a network. You don't actually need 20 and 100 gigabytes a second. That's just like genital measuring at that point. Wait, there's a disk. Oh, disk 26 has been surprise removed. Okay, yep, yep, that's fine. That actually did happen. So that means the whole thing ran, no errors, works perfectly. Slowly, but perfectly. So then now Intel's behavior looks like actually more generous than stupid. So a tech, probably without understanding the whole backstory, saw these perfectly functional drives in the RMA pool and went, I don't know, the customer's always right and replaced them, not realizing that the problem was because of this edge case error that only shows up on some platforms under some workloads. So thanks, Intel. It's a specific incompatibility between this drive and second gen AMD Epic CPUs. Dell has already fixed the timing behavior through a firmware update on at least some of their boards, maybe one or multiples of their customers put in a big order for new servers and then had a bunch of these they didn't wanna replace. And curiously, AMD's Daytona test platform from Quanta doesn't have the problem either. So maybe just Gigabyte couldn't be arsed to fix this, but then in fairness to them, it's just this one outdated drive from Intel. And therefore it probably doesn't affect any of their actual customers. Like remember guys, this right here is an engineering sample unit, meaning I didn't pay for it and it's still technically the property of Gigabyte. So it's not really anyone's fault, but that doesn't change that I spent over $10,000 on drives and my server doesn't work and I want it to work. So what I've decided to do is just eat the performance penalty and keep these new drives in my old Super Micro Box, even though the limited number of PCI Express lanes means they are bottlenecked to hell and back. I mean, while we waited on a solution to this, Liquid stepped up and set us up with their Honey Badger Den, which is faster anyway and has been rock solid. So thanks Liquid, thanks Intel, I think. And now this will be the capacity SSD server. There you go. And this will be my segue to our sponsor, Drop. Check out their Enter Keyboard. It's made with enthusiasts in mind, making it easy to swap out keycaps and even key switches. It's got an aluminum top plate and plastic bottom plate that feels great and white LEDs for visibility in dark conditions. The PBD keycaps are double shot, so they shine through, and it weighs 964 grams. So you could basically kill a man with it. Not that I would recommend that. It's available in three colors with your choice of mechanical switches, and you can buy yours today at the link in the video description. If you guys are looking for another server video to watch, you can check out the original Deployment Nightmare with, sorry, not this, with this one, or you can check out, we actually did a few videos on the Honey Badger Den. That thing is so fricking fast, like 100 gigabytes a second raw. Dangerously close to memory speeds.