 Well hello YouTube. We have a bit of a problem today on this section of the moving but we've been moved in for months moving vlog. Something is going on with the server and we haven't been able to work on a project for more than like 30 minutes at a time before the server crashes and our computers all freeze and then we're all sad because we can't work on them. Most of the s*** we do is on the server. Well I'm doing I'm actually doing some writing right now because I can't do that. See I'd love to be working on my next videos. Anything that doesn't need the server we can do now. I know. That's not much. But it always seems to go down. Yeah at the worst possible. When there's nothing else to do. But you know YOLO you only Liao once. Let's see let's see the stress that Linus is going through. I'm only getting about 25 minutes per second 30 minutes per second. It's probably gonna crash like while you're doing this. How do you fix servers? How do you fix it? Tail Linus. He will fix it. That's your go-to? Yeah. Whether you're running a small business or a one billion dollar or even multi-billion dollar enterprise, Rackspace has your dedicated storage environments covered. Check out the link in the video description to learn more. Alright so here's the situation. Wannix server over the course of the last couple of days has been spontaneously going offline and turning. And it's not just a one-time thing. It's a one-time thing. So you're going to have to do a lot of work to get it to work. So you're going to have to do a lot of work over the course of the last couple of days has been spontaneously going offline and turning off. In spite of my best efforts to turn it back on and complete the backup that I'm in the middle of trying to do to our new un-raid vault. While I was standing in front of it I observed one of my raid controllers giving up the ghost. So as a reminder this server is running three raid fives striped together in windows of a total of 24 SSDs. So if one of the raid fives drops out entirely all the data is gone. Only about 10% of the backup that I was just in the middle of remaking is saved. So it's time to investigate. Alright so here's Wannix server. So far troubleshooting steps that I've tried. You never want to rebuild something like this if there's a possibility you're going to have to go to a data recovery service because that can make it much much worse. So in terms of troubleshooting steps I tried transplanting it into another case so that I could use a different SATA backplane. That didn't work. I tried a different power supply, a better power supply. That didn't work. None of that fixed it and I even went as far as to put in another LSI raid card that I had and try to import the array. But while it did detect my drives, all of them as un-configured good and a foreign array, it was not able to import it. So there is some good news there though. While I'm getting firmware errors kicked out by this card right here when I try to boot, all the drives are detected by another card. So hopefully it didn't write too much garbled data to the array. Okay so this is basically what I've been doing for the last 14 hours is sending emails, contacting different data recovery services. There's some that are local, there's some that are not local, but I'm not ruling those ones out. In fact, right now the most likely solution looks like werecoverdata.com. They've got some custom tools that they've created that will allow them to SSH into the system and actually potentially recover the missing raid five and then the other two raid fives and assemble them all together and export them as one gigantic pile of data to one of our servers, all without me actually having to send the drives away, which would be pretty ideal. Now obviously an approach like that would not work in the case of physical damage to the drives, but because it is only a raid controller issue, they're saying hey there's a shot at this so let's give it a try. So I downloaded their remote recovery client thing here. So right now I'm waiting for their custom Linux based tool to load on a bootable external SSD so that they can SSH into the server. I'm not having a good couple days. Right now I can't get the utility to detect my USB drive. It's called tech support. Oh bloody hell. I just figured it out. This is Linus calling by the way. I just realized I have to run the WRD disk image as administrator. You guys should put that in the documentation. You guys should put that in the documentation. It never fails. This is why it's a good idea to call tech support because the second you call them you'll solve the problem on your own. Right so in theory we now have a bootable drive. Okay so step one is getting this unnecessary extraneous server out of the way. I'm grabbing a UPS to use. Now in a perfect world I would have a motherboard or an HBA card, a host bus adapter card that could allow me to plug in all 24 of the drives for them to do their data recovery magic on. But what we're going to have to settle for is having eight of them plugged into the motherboard at a time as each RAID 5 is rebuilt. RAID cards do all kinds of funny nonsense and they don't give other software direct access to the drives at a bit level in some cases. So we need to remove all these drives and plug them in directly to the motherboard. Okay now it should boot to that in theory. Let's just make sure all the drives are detected in the BIOS and everything. Oh that's a problem. I need to pull out the other RAID controllers as well otherwise they're going to be looking for their drives and I don't have them powered up right now. Okay. Oh actually that doesn't look very good. PCIe port errors. What? So let's make sure all of our drives are even showing up. Oh that is extremely worrying. Oh that is extremely worrying. Oh that's not good. That means potentially three of the drives in a RAID 5 are dead which means this data is not coming back. None of it. Not this particular RAID. No none of it. None of the. But I thought we have three. None of it will be coming back if we lost three drives from one of the RAID 5s. So let's try moving things around. I've never tested the SATA ports on this board. It's possible three of them are dead however extraordinarily unlikely that would be. Five six. Okay we've got six now. Okay that gives some hope. I'm just moving SATA ports around. I was just going to say right now I'm even having difficulty getting this system to post so I'm at a whole new level of concern. Concern. Yeah I'll let you know. I'll send an email. Okay so I'm at the point now where I don't want to put the drives on this system. Let's go with this one. I didn't like all those PCI Express errors that the bootable USB was kicking out. I don't like that some of these drives aren't showing up. I'm starting to wonder if that's what corrupted the LSI card. Royal pain in the ass. There's a thousand things I can think of that I would rather be doing right now. Okay I got seven right now. Really? Yeah. Wow maybe it was bad PCIe. Line of speaking. Well I can tell you what's happening so far is I put it on the system that I was running the RAID card and the drives in and that baby started spitting out PCIe errors all over the place. So I got everything pulled out of that system and I have plugged all the drives now into a different test bench and I'm trying to see because I wasn't even getting all the drives showing up. Two three four five six seven eight. Okay hey you guys are on the phone for the magic moment where all eight drives are detected by a new motherboard. So I'm beginning to think there may be a motherboard issue that caused the firmware on that LSI controller to corrupt in the first place. So I'm actually pressing enter and I am booting up to your USB and it is loading correctly on this. I think that board is dead. Um okay well that's I don't know if that makes it worse. Probably does doesn't it? You can tell me I can take it. That says waiting up to 60 more seconds for network configuration. Good timing with the follow-up call. If you called one minute earlier I've been cursing you for dialing my number so I have to answer my phone because I was in the middle of thinking only five of my drives were being detected. You guys know what you're doing right? I had a local shop that I was going to physically physically take it to and they basically they were like they kind of freaked out when I said I was having someone use their remote tool. They were like well unless these guys really know what they're doing that's pretty dangerous and I was like okay well I'll check with them I'll make sure they know what they're doing. So you guys know what you're doing? So basically they're explaining that they're going to virtually mount the volumes so we're not doing any writing to my disks. So even in the event that we cannot recover it by this means that still leaves other options open to us. Okay they're in. Okay well that's that's very wonderful to hear but I'm trying not to get too excited right now. They sound pretty confident. This is like the ultimate roller coaster right now. So to put this in the appropriate context for the viewers right now Wannock server has several in progress projects stored on it that do not have a backup anywhere else including a fully shot channel super fun that involved a thousand dollars in equipment rental including lots of footage from Linus Tech Tips videos. Like it is going to be a huge huge devastating loss for us to lose this. I mean it's got a lot of our templates for editing videos for scripting templates for employee reviews like all kinds of stuff like this is our main working server. No the off-site backup server has not ever been built yet. The vault was the backup server. If I ever have to like act in a in a video where I have to look for Lauren I'm going to think back to this moment. Okay so I do need to reboot it once in order to add that eight terabyte drive. Oh anyway right. Okay thank you very much gentlemen I hope to speak to you with good news very soon. Ah ah ah 12 SATA connectors ah ah ah. Okay so let's make sure everything is showing up. One two three four five six seven eight of those our boot device and our eight terabyte recovery media. We are ready to rock. Okay so within about 10 minutes they contacted me back they said okay yeah we're ready to have all the drives in there so that means we are going to have to do some janky ass right now. So basically what you're looking at here is these drives are going to be powered off this power supply with a jumper. These cards and all these drives are going to be powered off of this rig. So now we can power all those drives without powering up this motherboard or any of that nonsense. Okay so we have a grand total of 25 drives connected to this test bench. 16 of which are running in bays of this enclosure and eight of which are spread out here. So there we go there's one of my LSI controllers there's its eight drives there's my other one whoa where are your virtual drives. I think I remember a troubleshooting step where I tried swapping all the drives to different bays where I was trying to see if it was the back plane or the cables. Wow panic moment there but it's okay. So now we should see two virtual drives here on the screen. That's one set show me the second set. Thought we lost another raid five there okay it's good feeling good. Reality TV ain't got nothing on IT work. Show me all the devices two raid adapters one eight terabyte drive one two three four five six seven eight Kingston drives. Boom okay I will let the data recovery specialists know. All right so all that's left now then I think I got in touch with them and I asked if we can put a 10 gigabit network card in so we can accelerate the offloading of all of our files. So all we're waiting for now is for them to reboot the machine and set me up with a network share that I can access and fingers crossed just start pulling off all the footage and well everything else. All right so check this out here come on here so I've got the instructions for how to access our data so I go into the tritium share and there it is a bunch of ulas a bunch of stuff we've got what looks like the file structure. Let's test it how about one that I that we haven't finished releasing yet like something from the uh the melee battler here let's just let's watch it hmm okay XA footage for sure. Uh oh. Okay here we can copy one of these let's copy one of these clips to my local machine and see if that helps. No oh boy. Let's have a look I wonder if everything's even here. Like I don't even see the linus tech tips folder one and two are missing. These full applications is empty that should be full of things. Okay well I guess we better get in touch with them. So uh thanks for watching uh subscribe follow all that good stuff and see you next time. I just got an email back from them. Please stand by addressing your emails with the engineer. It's possible the raid definitions were changed from one reboot to the next because of drive letter assignments in the linux. It's a 26 it's 26 drive limit we have I think we have more than one. We have more than one. We have more than one. We have more than one. We have more than one for the 26 drive-limit we have I think we have more than 26 devices in there. I got another email back. Linus we recreated the block devices. Please go ahead and check the videos and content that you were looking at again. I'm not enjoying this there's something there. I'll let them know the chairs empty again. This is just like the wildest rollercoaster ever. This is just like the wildest roller coaster ever. I don't even, yeah, I don't even have anything else to say actually. It's almost funny. I hope to some of you it's funny. Here's the email I got 17 minutes ago. I was stuck on a call. We got everything remounted, so you can now go ahead and view the data once more. This is the moment. Do we not have a two? I don't think we ever had a two. Oh, okay. Okay, so let's go to that channel super fun video that we were looking at. Yes, yes, Swagway jousting. Okay, and it was the delivery? I'm getting like tingling. Okay, okay, you gotta check the source files though because that was what we know was corrupted. Okay, okay, okay, okay. So untranscoded footage XA. They did it. It's back. Oh my God. It's back. Here's another file, it's back. Do we have everything? In theory, yes. Really? Holy. Oh. I think it's finally over. That is fantastic. Hey guys, guess what happened with the recovery? What? We got everything back. Oh, thank goodness. Like it's just been such a stressful few weeks and that was just such a huge, huge thing. All of our pitches and proposals and stuff are on that server. Okay. Wannock's back. Bully? Really? Everything is back as far as I can tell. I have played back video files. They are not corrupted. Wannock is officially back. Swagway jousting footage for the melee battles should be fine and we'll have a better backup scheme in the future, I promise. That is all. We're making a safe video shot. We should physically print all of our videos, every frame of every video on that paper. Did you hear the good news? That Wannock's back? Wannock's back. Yay for hiring other people to do our jobs. I know, I don't think it's back, but it is. It's back. What are you doing? I'm just letting everyone know that Taren, it's back. The server's back. Okay. So it's not always that our sponsors tie so perfectly into our content, but today it happened. Today's sponsor is Rackspace, the top tier managed cloud computing company. They pride themselves on best in class service across all platforms. They've got over 300,000 customers in 120 countries with 10 worldwide data centers. They've got Red Hat, Cisco, Microsoft, VMware certifications. And all of that amounts to, whether you're running a small business or a billion dollar enterprise, whatever your needs are in terms of capacity storage or flash-based high performance storage, they can take care of it without you going gray or bald because you did something wrong and screwed up a backup and went and lost a raid array or whatever else. They've got dedicated storage to meet your performance, security, network capacity, and compliance needs. Everything from direct attached storage. So that gives you the flexibility and scalability of just a simple, okay, let's attach some drives to this thing. It's redundant, cool. Off you go to the races to SAN. So that's high availability and reliability, fully redundant for business. To even NAS with support for demanding workloads like virtualization, file sharing, and rich media. In fact, their NAS stuff can scale up to 20 petabytes of capacity. They've got public cloud and private cloud options as well. And you can get your own server, a scalable private cloud in the data center, in your data center, in their data center. And it's all supported by Rackspace and VMware. They call their support fanatical because these people are available 24 hours a day, seven days a week, 365 days a year. And they've got industry leading service level agreements, both managed and intensive. So if you need dedicated storage, they got you covered. Go to rackspace.com slash LTT to learn more and not go through what I did. I just, yeah, I don't really have anything to say. So thanks for watching guys. If you disliked this video, then hit the dislike button, I guess, but come on. This is about as real as it gets around here. If you liked the video, hit that like button, get subscribed, maybe even consider checking out our Amazon affiliate instructions for how to use it whenever you buy stuff there. Instructions are up there, buying a cool shirt like this one, or even giving us a month of contribution through our forum, which gives you a little contributor badge. Now that you're done doing all that stuff, hey, maybe you wanna check out that channel super fun video that we just discovered is back. And that'll definitely be worth your while. We joust with each other on Swagways. It's pretty awesome. So, see you next time.