 Behind me right now is the biggest supercomputer in the country. It'll be serving researchers across Canada, studying the human genome and bioinformatics, particle physics, materials research, even humanities research. It's called CEDAR. It cost the federal government through the Canadian Foundation for Innovation over $16 million, and we get to be the first to unbox this beast. Savage jerky is created without the use of nitrates or preservatives. Use offer code LTT to save 10% at the link in the video description. So CEDAR is a big data machine. It takes up a quarter of the 5,000 square foot data center it occupies, meaning actually that there's room for it to grow. But right now it has 27,000 Intel Xeon processing cores, 190 terabytes of RAM, 64 petabytes of storage, 584 GPUs and a total power draw of 560,000 watts. Though with that said, its efficiency is a shocking 1.07 on the PUE scale, where one would be perfect and a typical data center would be one and a half to two. We'll get into how they did that a little bit later though. So our tour starts right here. Behind me are what they call the high availability racks. So everything back there has dual power supplies for redundancy, with a battery backup for that and a diesel generator backing up that. Everything back here is mission critical. Things like networking, login servers and management servers are all here. And this is also where you'll find the bulk of CEDAR's storage. Let's get in for a closer look at CEDAR's connection to the outside world. This networking appliance from Huawei has a street price of around a million dollars. Wow! And right here, this is where it gets really bananas. These guys are CEDAR's dual 100 gigabit connections through Vancouver. And then as if that wasn't enough, these orange ones here are dual 40 gigabit connections through nearby Surrey, just in case somebody puts a backhoe through one of these other fiber lines and they would have otherwise lost their internet connectivity. I mean, that's their backup! Backup! But Ethernet is not really the way you want to connect high performance computing nodes. This right here is the true networking heart of CEDAR. These are 48 port Omnipath switches and they're configured in what's called an island topology. So the island is in almost all cases 32 compute nodes. Each of those compute nodes is connected to 32 ports on one of these switches in its rack. Then the remaining 16 ports come back to here. That means that every island gets a dedicated line to each of the core switches, giving you failover and massive bandwidth. Each one of these fiber links right here is capable of 100 gigabit per second. So even though between islands we are, let's say, bottlenecked by our 16 connections, that's only half the total theoretical speed within an island, we're still talking over 100 gigabytes per second. So it's not really an issue. Okay, now let's move on to SFU and Compute Canada's version of Petabyte Project. Spoiler alert! Theirs is better in every conceivable way. So in the five cabinets behind me, we've got CEDAR's 50 petabyte IBM tape library system. They have a 40 gigabit link to the rest of the supercomputer and each of the 5,000 10 terabyte magnetic tapes inside can be grabbed out of storage, moved with like a robotic arm into a reader, and the data can then be accessed when needed. And this is done automatically. Cool, right? Okay, yeah, but due to the slowness of that swapping process, this is still what we would consider to be cold or archival storage. Next up here is general purpose storage land, where any data that's being used for any current research project would be housed. So here they're using off-the-shelf 5U racks, each of which contains, let's see if we can crack one open here, a total of two kind of trays here and 84, 8 terabytes, let's have a look here, enterprise capacity SAS drives from Seagate. But there's actually more to this system than meets the eye. Every four of these storage nodes requires two nodes of what they're calling object storage servers. These act as a high-speed cache with their SAS 10,000 RPM drives, as well as as kind of like a traffic cop for everything behind it. So every single read or write to these hard drives actually goes through these nodes. So right now, general storage land is 10 petabytes, but in the near to mid future, it will be expanding to 20. Twenty! Now that DIY approach to storage is great for scaling up at a low cost, but when it comes to performance, they went for this data direct network storage appliance because it has got the real goods. Now, in the rack next to this brain, you'll find a mere four petabytes of actual storage due to its higher cost, but thanks to its proprietary hardware, custom software, and solid state burst buffers, this thing can handle up to 40 gigabytes per second of sustained throughput, making it perfect for data intensive applications that rely on humongous data sets. Now let's get into compute. There are about half a dozen different types of compute nodes all connected to the same high-speed OmniPath network backbone that are optimized for different types of research. We'll begin with the base compute node. There are a whopping 576 of these. Each of these is a computer. So there's actually four in a single 2U shell, each of which contains two Xeon E5 2683 16-core processors, 128 gigs of RAM, and about a terabyte of RAID 0 SSD storage for scratch. So each rack here contains two islands. So that's a total of 64 compute nodes, giving us a whopping 2,048 compute units per rack. So these nodes are the basic workhorse of Cedar, handling everything from Monte Carlo simulations for material science to simulating dynamic processes in nature with a high degree of randomness, like snowfall or rainfall. They would also be used in any highly parallelized workload, because if you need, you know, 10,000 CPU cores for one job, there aren't enough cores in any other class of server to handle that kind of load. Moving right on up, we've got the big memory nodes. There are 48 of these, and half of them are just like the basic nodes, except with 512 gigs of RAM, while the other half of them, these puppies, have one and a half terabytes of system memory. These ones take up twice as much rack space, though. Each of these one use is a single dual socket system, because you know what? There just wasn't enough gosh darn room for all 24, 64 gig memory modules that are required for that much RAM. First world problem? Yes. These guys are really special. These are the aptly named 3 terabyte nodes. There are only a handful of them, but these are quad socket machines with Xeon 4809 V4s. Four of them. But wait a tick! Those are only eight core processors. These don't even have more processing cores than those little tiny ones that take up half of you. What's the deal here? Well, it turns out that some bioinformatics workloads, like genome sequencing, don't actually scale very well with more processors. They just need massive amounts of memory to hold the data sets that they need to work on. So while the team here probably isn't super stoked on using up four U's just so they can stuff more memory into the system, until Intel Optane reaches a higher level of maturity, this is the only choice they have. Now finally, we're getting to my favorite nodes, the most expensive nodes. These are the GPU nodes. And while they're actually quite similar to the base nodes with respect to their CPU and RAM configurations, what's got the researchers in the fields of molecular dynamics, AI and machine learning all amped up about these are the quad NVIDIA Tesla P100 graphics cards that they have crammed into each one. I mean, seriously, with 1500 watts of power being consumed by each one of these, it is an engineering marvel that they've crammed enough power and cooling to make this whole thing work. So actually, now that you think about it, how exactly did they do that? So the keen-eyed among you might have already caught a couple of hints earlier in this video, but the secret lies in the rear doors on the server racks. Look how thick this is. Yes, my friends, this entire door is a gigantic heat exchanger. So their servers don't actually have water blocks. That would be more expensive. What they're doing is they've just got the fronts of the racks all sealed up, so there's no back draft pressure, and they've got normal air-cooled servers that pass the air from the front, where they just draw in room temperature air in here, and it comes out hot, like 30 plus degrees, and push it through this heat exchanger, where it is actually cool to my skin. That's how efficient these are. And that cooling system is massively expandable too. You can actually see above me, I am standing, where we've got a blue and green cooling pipe connected to a whole bunch of quick-release fittings, ready to add more racks right here. But to see what they actually do with the heat, we're actually going to have to go upstairs, where we'll find the final and perhaps the coolest stop in our tour here. This is the mechanical room, where the pumps and these freaking pipes take all the water from downstairs and dump it into three cooling towers outside the building. Now, right now, the weather is favorable to cooling. The ambient temperature is quite low, so it's just operating as gigantic radiators. But get this, when the conditions become less favorable in the summer, they kick things into high gear with an automated system that sprays water onto the fins of the radiators in the cooling towers. And if you watched our bomb cooling video, which you can check out right here, you'll be familiar with this concept already, but this is called evaporative cooling. And by these means, even in ambient temperatures up to 30 degrees Celsius, they can achieve the 17 degree coolant levels that they need to, without employing the massive chiller unit that they have over on the other side of the room. Squarespace is the way to build a website, whether it's for your small business or for your, you know, local freaking book club. It doesn't matter. If you want a web presence affordably and quickly, Squarespace gets it done for you. It starts at just 12 bucks a month and you get a free domain. If you buy Squarespace for the year, you just pick one of their templates and boom, you upload some pictures, you fill in some text, it's all cloud-based and your website will, simple as that, look great on any device. Every website comes with a free online store and their cover pages feature allows you to set up a beautiful one-page online presence in just minutes. So start a trial with no credit card required and start building your website today. Then, when you decide to sign up for Squarespace, don't forget, head over to squarespace.com forward slash LTT and use offer code LTT to get 10% off your first purchase. So a massive thank you to SFU and Compute Canada for allowing us to run amok in their data center. Thanks to you guys for watching. If you disliked this video, you know what to do. But if you liked it, hit that like button, get subscribed, maybe consider checking out where to buy the stuff we featured at the link in the video description. Also down there, you'll find a link to our merch store which has cool shirts like this one in our community forum, which you should totally join.