Step 1. Understanding The Problem

There seems to be a bug with the color clamping in most Maya image output file formats when using render passes. Even with most of the file types that support higher bit depths there seems to be a tendency to clamp down to an 8 bit range, and also do it incorrectly (for compositing needs). The shadow pass is the first area where I noticed this. Instead of a “normal” shadow pass that only involves black and white values, this Maya Mental Ray shadow pass now includes hues from the scene and is also luminance inverted.

This is the shadow pass image as it comes out of Maya.

The correct and normal method to composite this type of shadow pass would be to combine or merge it on top of the diffuse color pass using a “difference” apply mode. This almost works, but for some reason (improper clamping) it causes inverted colors to appear in the deeper shadows.

This is how the shadow composite should not look. Notice the incorrect green hues in the deeper shadow areas.

This is the Beauty Pass. This is what the shadows are supposed to look like.

This clamping issue causes inconsistency between the different color layers that need to fit together just so in your compositor. This bug also affects other render pass types. Direct Irradiance pass doesn’t composite properly with diffuse material color pass. depth maps do not create proper values over the entire scene area, instead only the front part of the scene. A whole mess of problems emerge. So now you are stuck using a 32 bit file format to get those render passes to work right. Well that’s fine.. but then on top of that it turns out that only one of the possible image file format works properly, and the rest clamp down to 8 bit even though they may be theoretically compatible with higher bit depths. EXR is our savior. For some reason there seems to be a bug with many other file types such as targa, tiff, even iff, do not work right, even if the frame buffer is set to 32 bit. Until Autodesk fixes this issue, we need to work around it.

Step 2. So Now What??

Here are the steps to output 32 bit EXR sequences out of Maya.

1. Maya-> Render Settings-> Common tab-> Image Format-> OpenEXR

2. Maya-> Render Settings-> Quality tab-> Data Type-> RGBA [float] 4×32 Bit

Now you should end up with a sequence of EXR files, either a bunch of different passes in separate files or a ton of channels all crammed into the same file. Here’s how to deal with them inside of After Effects.

1. After Effects->Import->select your EXR sequence passes

2. Lay your passes onto the timeline. If you’ve made one EXR with all your passes inside of it, then make a duplicate layer on the timeline for each pass type you made.

3. Select each layer and apply Effect-> 3D Channel-> EXtractoR

This plugin is now included with After Effect CS4 . If you have an earlier version then you can still download it for free from this site. Also, I find that version 1.3 is much more stable than version 1.2 (that shipped with CS4), so you may want to update in any case.

4. Click the EXtratoR Channel Info Box in the Effect Controls tab. Here you will need to pick the R, G, and B layers that are appropriate to the pass you are trying to display in each layer. If you have separate EXR pass files then there will probably only be three choices in each, but if you have a composite EXR with all your passes then you may have a very long list. You can see how I chose the RGB shadow pass channels for my shadow layer in the image above.

5. Select the appropriate apply modes for all your layers. Shadow should be on Difference, Specular should be on Add, Direct Irradiance should be on Multiply.

At this point you should still be seeing a messed up looking composite that does not match your beauty pass- but we’re almost there! We need to set the entire project to a 32 bit color space.

6. After Effects-> File -> Project Settings -> Color Settings -> Depth -> 32 bits per channel (float)

Now at last you should have a 3D composite that matches your beauty pass!

Click on any of the images to view the videos!

Click on the image to view the video!