Join Steve Wright for an in-depth discussion in this video Roto masking examples, part of Nuke Essential Training (2014).
The roto node's sophistication gives it great power and flexibility to solve a number of masking problems. However, this sophistication also creates confusion as to how to set up for specific situations. In this video, we'll see how to set up the roto node for a variety of common production situations. We'll scoot over here, to zoom in on this back drop node, to look at the first issue, and that is the size of rotos. What controls the size of the roto node. Now this image is a 512 by 373.
If I switch the viewer to the alpha channel. And then switch over to the roto mode. You can see that the roto node is also 512 by 373. That's because it's the inherited size from whatever image it's connected to. But what if it's not connected to an image? I'll disconnect it here, and suddenly the roto node takes on the size of the project settings. Now, this is often not a problem because Origin is right down here, 0, 0. So, the Marcy picture actually comes up here and fits like this.
So, even though its larger than it's needed, it often doesn't hurt anything. However, if you want to set the roto to a particular size, you can use a reformat node like this. Set it to any size you want, I'm using HD. And then hook the roto node up to the reformat node, and it will become that resolution. Let's get over here to take a look at how to mask a basic operation, like a grade node. Starting with the roto node. I'll open it up here. We have a basic mask in the alpha channel by default, which is fine for what we want to do.
If we look at the grayed node here. Switch the viewer to the alpha channel. I'm going to hide the spline. We'll open up this grayed node. And whenever you hook something up to the mass input of a node, it automatically spawns it right here. So now I can turn up the gain. I'm going to turn overlays off to get rid of the spline. And now that grayed operation is being masked off, by that roto shape. Let's look at a similar work flow, where the mask is in the alpha channel. Again, starting with this image. I'll turn my overlays back on, clear the property bin.
This roto node here also has it drawn into the alpha channel. So keep in mind, the original image is a three channel image, and it's not using the alpha channel. So here, I have added the roto to the alpha channel. And then in this grayed node I'll have to tell it to go look in the alpha channel, it doesn't know it's there. So I'll show you that by turning up the gain and the whole image has gone up, okay. See the whole image is being effected. But now I'll say mask this off by the contents of your alpha channel, and there we go.
Now only the masked off region gets the brightness. When you're keying a green screen or a blue screen, you're almost always going to have to garbage matte it. Now there's two ways to do that. You can garbage matte the original green screen to turn it into a nice pure green, or you can garbage matte the resulting key. Let's look at the first situation where we're going to set the green screen out here to be a nice beautiful and uniform green. I've drawn this roto here, draws a nice shape around the character, and of course it's putting the mask in the alpha channel as per usual.
So, how are we going to use that, to flood fill out here a nice uniform green. Well, we'll select the bezier and set the output, instead of alpha, to RGB. Now it's painting into the RGB layer, where the green screen it. Next, we'll go to the shape tab. We'll say invert. Well, now I've flip filled out here with a solid color, but it's the wrong color. So how are we going to fix that? Well, we'll open up our Eye Dropper node, come in here, and just sample the green screen, and there we have it.
We've flood filled the entire green screen with a nice uniform green that the keyer is going to just love. In other situations, you've already pulled the key, and you're going to want to garbage matte the resulting key. So here's the same picture. Here we've done a primat key. We've got all the stuff out here that we want to get rid of, with a garbage matte. So, we'll take a look at this roto node here. And again, by default, it's flood filled the alpha channel with white. So here's how we deal with that. We'll select the node, go over to the shape tab, and invert it so now it's filling the outside with white.
And then we set the white to black. There we go. Another situation, you want to mask off a four channel image. So I have this jet. Obviously, a nice four-channel image. And I've got it composited over this background. So what I want to do is erase the nose of the jet. So, here's the workflow. We'll start with our jet and the four-channel image. We'll switch to the rotor node, and you can see that my rotor shape is drawing a white mask on top of the jet. Let's open up that roto node here so we can play with it. So to erase the jet, I need to turn the RGB and the alpha channels to black.
So we'll set the output to RGBA, all four channels. Select our Bezier. Go over to the shape tab and set the color to black. Now, the RGB channel and the alpha channel, are faded off at the head, and if we look at our composite. I'll clear the property bin. We can see the front of our jet is nicely faded off. Here's another work flow for masking a merge operation. I have a nice four channel image here, and I've got it composite over the background here. I'll switch to the alpha channel and what I want to do, is I want to mask it off to only composite within the white mask.
So, looking at the merge node again, I'll hook up the merge mask input to the roller shape, and now it only is allowed to composite inside the mask. We'll switch to the RGB channels, and turn off the overlays. Another case, let's say you have this four-channel image. And you need to a mask in this alpha channel. Well, if you just do that, then your alpha is going to be added to the existing alpha. So what we really want to do is replace the old alpha with our new one.
Easily done. Open it up. Select Rectangle, and click on Replace. And that button there means replace the current alpha/ channel with whatever you've drawn in the Roto node. Last, how to isolate an element. We'll switch to this Read node here. Clear the Property bin, and look at the RGB. What I want to do is isolate this graphic and composite it over another background. So, how do I do that? In this roto node, as usual, it's drawn the mask in the alpha channel.
So, what I need to do, is perform a pre-multiple. Well, the roto node has that built in. So I can just go to pre-multiply, and set it for RGB. Now I have a four-channel pre-multiplied image, that's ready to composite. As we've seen, the roto node can solve a great many types of masking problems, but it's difficult to remember all of the possible colinations and permeatations. Use this video as a handy reference guide to how to set up the roto node for your real world production problems.