 If you ever find yourself in Pittsburgh and want to catch a glimpse of the future, Carnegie Mellon University is probably a good place to start. To get inside the robotics lab, you'll need to walk past Tank. He's a robo-ceptionist and, well, actually Tank is really important to this story, but he isn't what I want to show you right now. This is. Looking at this large geodesic dome covered in wires, you can't help but feel like you're staring into a time machine, and in a way you kind of are. But rather than traveling through time, this panoptic studio is capturing it. Using an array of cameras, researchers here are able to capture and then replicate any action into a 3D model. From a dance to a cello performance, it can perfectly reconstruct these moments in time. And while that's pretty impressive by itself, it's just the starting point in a quest to decipher an age-old code. So basically you can see that this is a kind of geodesic dome. Looks like a sphere. We specifically designed this architecture so that we can put cameras as uniform as possible. And you can see also that we specially modularized the system so that we only designed one panel and we can duplicate the panels. One panel is composed of 24 small cameras and one high-resolution cameras and they are saved in the local hard drives and in the end all the local data are they are transmitted to the NAS. And so that sounds like a ton of data, like how much data is in one minute of capture? So one minute of data is about 600 gigabytes. That's huge data. That's a big file. In fact, to this point, the Panoptix Studio has captured over one petabyte of data. That's about a million gigabytes. And the latest is, well, me. Ready? Can you see me? Yeah? So while I'm dancing, those 500 cameras are capturing every movement I make. And after a couple of weeks of processing, it looks something like this. This point cloud view is not only showing a bunch of tracking points, it's tracing the paths they could take while I'm moving around. All right, glad I stretched before that. Now, motion capture by itself isn't anything new. Similar technology has been used to bring digital characters to life in films for years now. Motion capture technology has gotten so good that an actor like Andy Serkis can really inject life into a fully digital character. But to do so requires a full motion capture suit with tracking markers placed all over your body, even painted on your face. Now, this works well enough for trying to bring a dragon to life, but it's hard to feel natural in that kind of equipment. The Panoptix Studio, in contrast, allows for similar tracking, but without any of the physical markers. You can just freely move your body in a way that feels natural and unrestricted, which is a necessity for them to capture what they are really looking for. A really good quote from Edward Sapir, who is a linguist. He says, there's an elaborate and secret code that is known to no one, but understood by all. And he's referring to all the gestures that we use to communicate. And that's kind of the code that we're trying to sort of crack or understand. Our goal, essentially, is to endow machines with the capacity to understand these social interactions. So that's where Tank comes back into play. You remember Tank, the roboceptionist? He can see me via camera and react to my presence, but he can't understand more than what I type to him. So when I start to get frustrated that he isn't helping me find the Panoptix Studio, he can't read that frustration or react in a way that helps ease my tension, which kind of makes him bad at his job. Machines have a very poor understanding of what all those things are. So, for example, if I'm not paying attention to you or if I'm looking at you and paying attention, those are the kinds of things that machines just don't respond to right now. And that's kind of a big deal. Studies have suggested that when we communicate with each other, more than half of the message we're sending is based on the way we move. Our posture, facial expressions, and gestures can at times convey more than our words alone. Here, let me show you. So now you can see me and you can read what my body is telling you, even if my words are saying something different. Maybe from my posture you can tell I'm in a cramped little sound booth, or you can tell I'm excited about this story, that I'm nervous about reading from a script, that I'm hopelessly improvising into a camera to try and give you a sense of how expressive body language can be. When we communicate with each other, we use many interesting hand gestures, which is really important to communicate with each other. But it's very hard to understand why we are doing this motion when we are doing this motion. Right. I don't know why I'm sitting like this. I'm not trying to send you... I mean, I guess I'm sending you a signal, but I don't know what it means. Exactly. Usually we are using our gestures, but it's very hard to define why we are using it. We are not doing this consciously, right? This is nonverbal gestures. I mean, I'm just like trying to sit in a way that looks good on camera. I like your socks, by the way. Thank you. I'm trying to show off my socks. That's why I'm sitting like this. And that's the really weird thing about body language. It's something we all do unconsciously and understand innately, but if I were to stop and try to tell you why I'm holding my shoulders a certain way or moving my hands, I wouldn't understand it to be able to explain it. So if we can't explain it to each other, how can we possibly teach it to machines? Well, it turns out we can teach them the same way we've been teaching a lot of other artificial intelligence systems. Machine learning, specifically a technique called deep learning, has produced a number of huge breakthroughs in the last few years. A very simplistic view of it is you feed machines a lot of data about a certain subject, like say driving a car. Feed it enough data about how we drive and it will be able to start understanding the system at play and react in a similar fashion to the way we do. Most recently, Google's DeepMind watched a few years worth of TV shows to understand how to read human lips. Now it can do that better than most experts. The Panoptix Studio hopes that given enough visual data showing how humans interact, its deep learning system will be able to tease out the structure of that elaborate code, making it possible for computers to read and write our body language, possibly understanding a facet of our communication better than we do ourselves. So if all this sounds familiar, you have probably been watching the new HBO series Westworld. It's about a park filled with robots that can so vividly replicate human behavior it becomes impossible to distinguish who's a real human and who's not. So Westworld is science fiction, sure, but do a quick YouTube search for lifelike robots and you'll find we've actually made some incredible progress in the last few decades. And some stuff to fuel your nightmares. By the time you're grown up, I'll be as smart as a real person and we will be like brothers. I do not know if you will remember me then, but I'll never forget you. Someday I'll come and find you and we'll be good friends. But the benefits of these social robots could be immense. Look at Milo. He's a robot that's being used to help kids with autism. He can help these children identify social cues and emotional reactions that they can sometimes struggle to express and interpret themselves. And robots will also help us care for the elderly, so less Westworld and more Franken the Robot. Beyond helping accomplish physical tasks, future robots could become vital emotional supporters as well. Imagine instead of a roboceptionist, Tank could help someone suffering from Alzheimer's socialize on a more regular basis, identifying their mood and reacting appropriately. There's many components that need to happen before this works. I mean AI in particular needs to get better, right? But I think that there will be a big advantage in terms of how we will treat machines once they start respecting at least, for example, whether we are paying attention to them or whether we are open to an interaction and other sort of social signals that we use in our daily conversations. But if all this talk of lifelike robots makes you nervous about the coming of some sky net that'll wipe out humanity, consider this. Teaching robots this social behavior might be the best way for us to foster a better understanding between humans and their creations. So one way of putting it is that we use machines and even robots as tools currently, but what we will achieve when they interact with us as almost equals is we will find collaborators. Also we won't start using machines as tools but rather as agents that we can collaborate with. So I think in those terms that that will be very important. So if we want to integrate machines and it's eventually going to happen, they should at least respond to us in a way that we appreciate. This research is already being used by a company you probably know well, Facebook. The social networking giant and its Oculus division are working with researchers from the Panoptix studio to find ways to make avatars in VR more accurate and social. This could make communication with a friend more intimate or it could give Siri a body that can walk and talk just like a real person.