Render layers are of supreme importance as they give the artist the power to fine-tune the image in every conceivable way post-render. Rendering separate passes as image sequences allows for the re-rendering of specific portions if an error or visual glitch occurs rather than a brute force re-render of the entire sequence.
OK. So you've got your scene set up, fully animated, shaders and textures in place, carefully lit, cameras in place etc etc...what next?
In our Channel Box / Layer Editor on the right side of the screen, under the Render tab, we've only got one render layer. This layer should be called masterLayer and includes everything in the scene. This is the layer we've been rendering when we shoot out a test render. Since tweaking EVERYTHING involved in even a simple scene still won't result in a perfect render, it seems prudent to split the elements we'll need into separate layers which we'll later combine in a compositing package such as Nuke or (if you must) After Effects.
We'll start with an Occlusion layer, creating a self created shadow based on proximity rather than light. To create a new render layer simple click the Create new empty layer button, do NOT use the button to the right of it...it's glitchy beyond belief.
Name your new render layer accordingly by double clicking on the text. Notice how when we have our new layer selected, nothing shows up in the viewport.
Select the masterLayer and, in the outliner, select all your geometry. We ONLY want geo in our occlusion layer, no lights at all. Once you've got all your geometry selected, right click on your occludeLayer and click on Add Selected Objects.
Right click once again on your occlusionLayer and click on Attributes. Oddly enough I often encounter strange glitches with this function and sometimes have to click the Attributes button several times before the Attributes for that given render layer actually pop up. Mileage may vary.
On the appropriate tab for your Render Layer in the Attribute Editor, hit the Presets button and select Occlusion.
You'll see a new, specialized surface shader pop up and all your geo will turn black.
Upon rendering, you'll get a result similar to this. This is far too dark, we'll have to adjust this new surface shader as we should be seeing darkness only occupying areas of extreme fold rather than spanning entire surfaces.
With the new surface shader selected, in the Attribute Editor, click on the tab for mib_amb_occlusion1.
Setting the Max Distance to 1 usually fixes this gradient problem for me, but feel free to play with the parameters. This looks much better now, on to the next render layer!
Using the same steps as before, create a new render layer, rename, and assign only the geometry to it. We're going to make a Diffuse layer. The Diffuse preset isn't terribly helpful here, so we're going to need to dive into our Hypershade and manually adjust each shader to give us the flat color we're looking for.
Select a shader and, in the Attribute Editor, set a Layer Override for the attribute Ambient Color by right clicking on that attribute's text and selecting the appropriate option.
The text will light up orange, meaning that anything you do to this specific attribute will ONLY change it for this render layer, if you switch to another render layer this attribute will snap back to the way it was before.
Crank the Ambient Color up to pure white. Repeat this process on every single shader in the scene, obviously do NOT do this for your occlusion surface shader or maya's default shaders.
Now, upon rendering, we should be seeing only the color information with nothing extra.
In my scene I have two different light sources, so I'll want to make a render layer for each one. I'll add the geometry as usual to each layer, but this time I'll include the relevant light in the render layer as well.
Just as in the Diffuse layer, we'll need to be making overrides for each shader in these new layers. The first order of business is to place a layer override on the Color attribute. Next change the color to pure white, this may not be possible if you have a texture plugged into this attribute. To get around this simply right click on Color and select Break Connection. Since we just put a layer override on this attribute, it won't affect the texture on other layers, just this one. Then you should be able to change the color to pure white.
Once every shader has been altered in this fashion, you should have a pass that contains mostly just light and shadow information. I only plan to use the shadows coming from my blue light source, so I've turned off shadows on my orange light. Here's my blue light and shadow pass.
Here is my orange light pass, notice the lack of shadows in this pass. Never render something you don't need.
I've only covered the most basic of passes here but if you need additional passes, there are many tutorials in existence that should help you if I am not present to assist (note for random internet visitors, this was directed at the class I'm teaching...sorry I can't personally guide you through other layers random internet friends).
Other passes include but are not limited to:
Specular Pass (I've got mine piped into my light/shadow passes here, but separating them gives even more control).
Matte Pass or Object Buffer Pass(using flat, primary colors to allow you to select a specific section of screen in order to modify said section in isolation).
Depth Pass or Luminance Pass(for adding depth of field blur and other effects).
Once I've had a chance to composite my passes, you'll see a noticable difference in quality from the original beautyLayer render. For comparison's sake, here is the beauty render:
...and here is the final comp of all four of our layers. The additional control of using various layers rather than the default beauty layer allows the artist to produce a much more interesting and refined image. Given critique from an art director, we wouldn't need to necessarily rerender anything, we could go back into the comp and tweak the settings on each layer.
