I’ve been experimenting with Maya 2009’s Mental Ray Render Passes feature. It’s intended to easily allow separation of different elements of the render into separate images. An easy example would be separating the diffuse light from the shadow. In practice it seems to be way more finicky ( or broken) than it ought to be. After many trials I’ve developed a workflow via After Effects that successfully composites all my passes so that they just about match the default composite render.
Step 1. Understanding The Problem
There seems to be a bug with the color clamping in most Maya image output file formats when using render passes. Even with most of the file types that support higher bit depths there seems to be a tendency to clamp down to an 8 bit range, and also do it incorrectly (for compositing needs). The shadow pass is the first area where I noticed this. Instead of a “normal” shadow pass that only involves black and white values, this Maya Mental Ray shadow pass now includes hues from the scene and is also luminance inverted.
The correct and normal method to composite this type of shadow pass would be to combine or merge it on top of the diffuse color pass using a “difference” apply mode. This almost works, but for some reason (improper clamping) it causes inverted colors to appear in the deeper shadows.
This clamping issue causes inconsistency between the different color layers that need to fit together just so in your compositor. This bug also affects other render pass types. Direct Irradiance pass doesn’t composite properly with diffuse material color pass. depth maps do not create proper values over the entire scene area, instead only the front part of the scene. A whole mess of problems emerge. So now you are stuck using a 32 bit file format to get those render passes to work right. Well that’s fine.. but then on top of that it turns out that only one of the possible image file format works properly, and the rest clamp down to 8 bit even though they may be theoretically compatible with higher bit depths. EXR is our savior. For some reason there seems to be a bug with many other file types such as targa, tiff, even iff, do not work right, even if the frame buffer is set to 32 bit. Until Autodesk fixes this issue, we need to work around it.
Step 2. So Now What??
Here are the steps to output 32 bit EXR sequences out of Maya.
2. Maya-> Render Settings-> Quality tab-> Data Type-> RGBA [float] 4×32 Bit
Now you should end up with a sequence of EXR files, either a bunch of different passes in separate files or a ton of channels all crammed into the same file. Here’s how to deal with them inside of After Effects.
1. After Effects->Import->select your EXR sequence passes
2. Lay your passes onto the timeline. If you’ve made one EXR with all your passes inside of it, then make a duplicate layer on the timeline for each pass type you made.
3. Select each layer and apply Effect-> 3D Channel-> EXtractoR
This plugin is now included with After Effect CS4 . If you have an earlier version then you can still download it for free from this site. Also, I find that version 1.3 is much more stable than version 1.2 (that shipped with CS4), so you may want to update in any case.
4. Click the EXtratoR Channel Info Box in the Effect Controls tab. Here you will need to pick the R, G, and B layers that are appropriate to the pass you are trying to display in each layer. If you have separate EXR pass files then there will probably only be three choices in each, but if you have a composite EXR with all your passes then you may have a very long list. You can see how I chose the RGB shadow pass channels for my shadow layer in the image above.
5. Select the appropriate apply modes for all your layers. Shadow should be on Difference, Specular should be on Add, Direct Irradiance should be on Multiply.
At this point you should still be seeing a messed up looking composite that does not match your beauty pass- but we’re almost there! We need to set the entire project to a 32 bit color space.
6. After Effects-> File -> Project Settings -> Color Settings -> Depth -> 32 bits per channel (float)
Now at last you should have a 3D composite that matches your beauty pass!