 So I think we should bring out our guest. He's a brilliant guy. He's a principal researcher at Microsoft. He's making some frankly bizarre stuff. And I'm looking forward to this conversation. Please welcome Andy Wilson from Microsoft. How are you? Thanks so much. No pressure. No pressure. You have to one up Battleship. Can you do that? I'll try. Kind of low pressure. So Andy, these guys don't know you. Maybe they know you a little bit. We did a video months ago, like a year ago or something, that I thought was really cool. Visited. I came to visit your home away from home, your lab. Which you actually do live in. You do live there, right? Yeah, pretty much. Tell me what you do. How did you start doing this and what do you do at Microsoft? And give me a little of your backstory. So I'm a researcher at Microsoft. Microsoft has a big research group of about 850 PhD level folks doing research on all sorts of things. I'm mostly interested in natural user interaction and different ways that we can exploit sensing technologies and display technologies in completely novel ways and remake the way that we relate with computers. So a lot of the things you saw there are prototypes where we believe very firmly in building things and playing with them and just sort of finding our way through all these new things just by experimenting and trying as many things as we can. And your work is, I mean, from what I saw and some of the stuff we're going to show today, your work is really about, I mean, we're very used now to these kind of, we have a screen or we have something in front of us, we have something in our hand and we interact with that. We're kind of like going into that interface. Your stuff is like taking that interface out into the world. I mean, a lot of what you do is, it's like projecting onto the world basically and making the world the interface. Where does that instinct come from? I mean, what is the end game for you with that? So that's right. So I like to call it beyond the surface. So I worked on the first version of Surface, the one that used video cameras, the large 30-inch tabletop. And so for me, it's always been about what can we do with sensing technologies in interesting ways? And for me, the work that we've done with depth cameras, for example, you know, lifts all of the interaction we're getting off of touchscreens out into the real world. And that's where we really live. We don't live in the flat plane of a display. We live in the real world. Let's bring the sensing and display into our real world. I mean, you headed up the Surface project when it was a large table. So I did the very first prototype and a couple of us, a couple of guys and I did the first efforts around Surface and building the first prototype. It was a lot of fun, really. Was it painful to watch Microsoft introduce a tiny tablet called the Surface? Did that make you feel... No, actually, I was very flattered that they decided to reuse our name. I thought that was a great choice. But now you can't call the Surface the Surface anymore. That's right. And there's that, right? And that hurts, doesn't it? A little bit. You can share. There's no judgment here in this room, okay? So you also incorporate a lot of Kinect sort of like technology into what you're doing. Is Kinect the first... I mean, that's, you know, there aren't many consumer products that are actually utilizing the physical around you to be... Is that the first step? Do you see that as just like the version 0.1 of this? Absolutely. It's the opening move in a great renaissance in natural user interfaces. You know, people have been doing this research for a long time. I was a graduate student at the MIT Media Lab in the 90s. And we were doing a lot of this hand tracking kinds of things and interacting with things on screen that now you can go out and buy. It's really nice to be able to see that move from the lab. Gosh, it's taken a little while. But now it's like, now that it's commodity, at least for certain kinds of applications, let's think about other applications where this would actually be useful. Is this the kind of thing that you would find in a conference room at a workplace? That gets me excited when I hear that. Would we find this in a conference room? Probably. Yeah, I'm working on it. Is there anything more exciting than something in a conference room? I mean, I know that's up here. That's like you're in... I know you're in bright orange on the conference room tech, but like, can we go to red? What about for the home? What about for the kitchen, for instance? Yeah, kitchen is a nice one because you don't want to get your hands all dirty on the touchpad, right? So maybe you put that on your tabletop in your kitchen counter and you actually lay out the recipe and the things that you need. And we've had a few demos over the years at Microsoft where it would be really fun for kids. You set out the recipe and the ingredients for the recipe and kids get that. And yeah, sure. Can I pitch an idea to you for the kitchen? Let's hear it. So I'm imagining something like these tentacles hanging down over your kitchen work surface. And then you use some kind of motion tracking. The tentacles are all full of different spices. And use the motion tracking. You put a chicken or something underneath and then you just kind of minority report the spices all over it. Excellent. Do you think that's something you could work on? Sure, absolutely. We actually have... Please make that happen. I just... for me? We actually have some video stuff that you couldn't bring with you. This thing is called a Beematron. Can you explain? Is this the Beematron? That's the Beematron. And so what that... The Beematron is a pan tilt motion platform kind of like what you see at rock concerts with lights. Except we replaced the light with a projector and a depth camera. And so what we're doing is projecting the car and the car responds to the real world just as if it were a real model radio controlled car. And so by using the pan tilt movement we can actually get all the way through the room. We can project a graphic into this person's hand and move that graphic from one hand to the other. Here's another shot of a ramp. This guy's a terrible driver. Yeah, absolutely. So the idea really is to bring the interaction out into your entire world. So with one camera, one projector in this motion platform we can track these people. There's we can project graphics onto this person and react appropriately. She can actually pull graphics from the wall and send them from one place to the next. Like a spice. Yeah, sure. For instance. You could track the... Get the curry. Yeah, exactly. Paprika, boom, tentacle down onto the chicken. Why not? And then you have another one. You have another one that we have video of which is called the Holoflector. Can you tell me a little bit about this? This is like... This freak, this is insane. So this is not done in post, right? This is actually a video shot of a mirror. And I'm standing in front of the mirror and then I'm interacting with this graphics. And it's a little bit like the Pepper's Ghost illusion you might have seen in a Disney theme park. It's a little different, but we can superimpose graphics onto it. And so it's really augmented reality but in a very large form factor. And we're incorporating some concepts from a phone. Here's maybe I'm doing a little holographic phone call. This is actually a connect recording. Is that you? That's you talking and holding yourself? That's me. This is my little homage to Princess Leia. And you know... That's a little weird. Yeah. But you brought something today. I did indeed. And we should take a look at it. Let's check this out. What is it called? What did you bring? This is the wearable multi-touch projector. Okay, I'm gonna come up with a better name for that. Wearable multi-touch projector. Let's check this out. Yeah. You want some? Do you need help? This is handsome. This is nice. Now, what did you say this is called again? We call this the wearable multi-touch projector. We got to come up. But why don't you just call it the Predator? I mean, it's kind of, I don't know. Yeah. A little bit. Looks a little bit violent. So this is kind of like a Kinect here. It is basically like the Kinect. It uses the same technology. It's a short throw version of Kinect. So it actually works in the sort of arm reach volume, just the area in front of me. And you think in the future, everybody will have like a Lenovo laptop hanging off of their side? Is this the future? I feel like... So we build prototypes, right? And I'm just starting the projector. There's also laser projector. Is it a Pico projector? Pico projector, indeed. Okay, great. And so this is actually our second prototype. The first one we built actually disconnected to a desktop computer. First one had a desktop that was hanging off of you. That's right. It was like a full gaming tower. You had like a little assistant to carry next to you. This is our second one. And of course, if we were to consider productizing this, I mean, we would not have this. But this is a really nice setup because we can actually write like really sloppy code and get that running on here and do all the experiments we want to do. So you admit your code is sloppy. Absolutely. That's fine. There's nothing wrong with that. All right, so show us what this thing does and then tell me why it does it. Right. So the idea is to bring some of the interaction into the world. So we can actually co-opt surfaces in the world. And I'm just going to lean over here and start this. And it'll take a moment to start, hopefully. Is this now what version of Windows is this? This is Windows 7, actually. Uh-huh. So you haven't upgraded yet, huh? Not yet. Not yet. Interesting. Okay, so we're using the depth camera to actually- Ignoring that completely. Yeah, exactly. All right. We'll right past that. So here I am. So I'm going to touch the wall here. And you notice that I get a bit of a menu. Okay, it's a little bit like our Windows Phone interface. This is the new Windows Phone 8 I've been hearing about. Yeah, that's what you're talking about. So and you notice that as I move back and forth, whoops. There, it just disappeared. That's all right. We can start that again. And as I move back and forth, it actually stays the same size. We're actually using the depth camera to rectify or make it appear as if the graphics are the same size or actually- The real goal would be to stabilize it so it feels like there's actually paint on the wall. So what I'm going to do is show you a couple things. There's a phone icon. We can actually touch this. And there's this ubiquitous slide to unlock. Oh, ubiquitous. That is a patent case right there. All right, there we go. Apple, are you watching? Because you're going down. You're done. Why even bother? I just find to take whatever money is in your bank account right now and send it back to Apple. So we have a dial pad here. So we can do the kind of expected thing. You notice there's this sort of red ball here. And that's showing me where it thinks my finger is. Is this an actual phone number? No, well, gosh, I hope not. Keep going. Yes, OK. Let's call someone. And I'm just dialing a random number. Please don't call that number. Is this actually capable of dialing? And it is not dialing. It is not dialing. So that's just something we've been playing around with. But it gives you an idea of the sensing technology that we have. So we can do a few other things. Oh, by the way, it's multi-touch. What if my finger got in here? What would happen if this guy came in? It might work. If you come in here a little bit. It is working. Yeah. How do I- can I erase this? Yeah, absolutely. This is horrible what you've drawn. Yeah, it totally is. Let me see if I can do something. Yeah. So it works a little bit better if you come in from the side. And it's a bit flat. Oh, there we go. Yeah. And that has to do with the way that the algorithm works that we run. That's going to draw a face, but that's obliterated. Right. Now, you notice that I'm actually stopping each of the little demos by putting my hand over the camera. It's kind of a funny thing to do. But it's a way for us to reset the system. I'm going to show you a couple more things. OK. Great. So we have- here's your map. And again, as I move back and forth. Apple might want to also take note of this. Right. And so it's multi-touch as well. And so I can scroll around. And there we go. So that's the map. Nothing surprising there. And then let me show you- Can I see the Hoover Dam on that? I just want to see if it's intact. So there's a couple of other things that we can do. One is the notion of being able to invoke an application just by drawing the space in the world that you want to see it. And that's not- Be the space that you want to see is what you're saying. Yeah. So tell me what the practical- what you envision a practical application for this is. Is that a tip calculator? Yeah, exactly. Something like that. So actually it's- Current to converter? That's great. Yeah. So give me a- so I'm walking around. It's in the future when these are just commonplace. Yeah. Well, give me a scenario where this is becoming- I need this. So a good one would be where we just met and we're maybe- we met in the hallway and we're- I want to show you something over here and I want to show you it in the large. I don't want to like huddle around the tiny little phone. Right. I can actually throw this up on the wall and imagine it's bigger and brighter and all these things. Projector technology is going to get better, of course. Then we can interact with this as if it were a real large multi-touch display. Right. So the guy's like, I just met this nerd with this thing on his- And now he wants to show me something. Exactly. That's right. But of course everybody will have them. Well, imagine it's a tiny thing. Maybe it actually fits in a button on your shirt. The entire thing in a button on your shirt. Yeah. You think we can get there? I think so. Really? In just a matter of time. Yeah. Interesting. All of these things that I'm showing you here are really small when you sort of take them in the limit. The Pico projector is just a- there's not even a lens on the Pico projector. Right. Can you show the audience? Turn around here. I just want you guys to see- you're seeing the back. But do you see this whole setup here? I mean, the idea that this could get down to the size of a button seems frankly like a pipe dream. Are you on any dope right now? But look at the size of the first cell phones. Right. And look at what they're at now. That's true. They were originally like this tall. They'd have wheels on them. I mean, if you took- And a thing with a wire that came out. If you took a modern cell phone and showed it to somebody from 30 years ago, they'd be like, what is this alien technology you're showing me? Right. It does look alien. Well, this certainly does. Yeah. Right. Let me show you one more thing. Okay. And so, let me set this. And then I just want to give you a little bit of an idea of the processing that we're actually doing. So this is just a display that shows the fingertip tracking. And you notice there's- it's multi-touch. And then as I reach in, it goes from hover state to a touch state. And it's- we're doing two at once. And then finally, I'm going to back out a little more time. And real quick, I want to show you the nature of the processing that we're doing. And this is basically just showing you the depth stream that we're getting from the Connect camera. Oh, wow. Right. Can I get in here? Yeah, you can get in there. Right. So you can get an idea of some of the processing that we're doing. Do you want to- I don't know what this is. Exactly. Right. But that's cool though, right? It shows you- Check this out. Yeah, there we go. Oh, did I turn it off? No, it's gone. Yeah. So I got too far away. It's too- I was too funky for it, which is a problem. A lot of people feel I'm too funky. And then- And the sheets. And the other thing that we've been studying is the idea that we can actually project onto anything, not just a whiteboard or a wall or something like that. And then we can actually track this notepad, for example, and actually use that as a display. This is- you're tripping me out now. And so we can actually do slide to unlock. Oh, yeah, the ubiquitous slide to unlock. And then we can even do it on your back. So it's a turnaround. On my back? Yeah. So for example- I don't like where this is headed. Do I need to take my pants off for this demo? Go on. We can get this to work. So yeah, there we go. So- Oh, I like this. You're a little bit lower. I have a little bit of a back pain in my- Yeah, right there. It's perfect. There we go. Can you press really hard? Oh, yeah. Slide to unlock. Okay. Ooh. So literally, we can use any surface available. Is slide to unlock supposed to make you sexually aroused? Is that a thing? All right. That's very cool. That's very cool. Just turn around here. Let me just- Andy Wilson, everyone. Andy Wilson and his- And we never did come up with a name for this. I'm going to call it the sex machine. Is that cool?