This had me crying long hours into the night for a rather extended period. Then I found the solution in the amazing blog of David Johnson in a post about surface shader transparency. And life has been better since.
Ive looked all over the web looking for a way to render occlusion and luminance for transparent objects properly and the only ray of hope seemed using some more mr nodes (something like fg occ or something ) and doing some more connections to get a transparent occ. As if life with mr wasnt complicated enough! VERY STRANGE then that I didnt find this simlpe and obvious solution (once you start using it) anywhere else!!
Anyways, here goes. For transparent occlusion – you create a maya layered shader. Put a surface shader as its only layer. Disconnect the transparency connection that happens automatocally once you plug the surface shader. Now connect your transparency file to the transparency channel of the alpha shader [ if your image has inbuilt alpha like a tiff or png, then just plug in the same image or plug in a black n white transparency map and check alpha is luminance]. Finally, on the surafce ahder you will have a compositing flag. Set it to layered texture instead of layered shader. So this is your occlusion shader with transparency. Of course, you need to connect the mib amb occ node to the actual surfce shader. Now apply this layered shader to your transparent objects instead of applying the surface shader directly and presto, ull see occluion respecting transparency! Maybe it sounds a bit more complicated than it is coz of all the text im using but try it, its the most straightforward thing in the world.
The principle behind getting transparent depth pass is exactly the same. Create a new render layer..go to attributes and select luminance depth preset. now remove material override from the layer and reapply the newly created surface shader ( this is done so you can assign a different shader to the transparent objects). Now do exactly as above. Plug in the surface shader into a layreed shader, map appropriate transparency and apply this layered shader to the transparent objects ! bingo! perfect transparency with luminance pass :) (to know more about setting up the luminance pass, read my previous post)
If you want to know the reason behind this, read on.So this is the part that I learnt from David’s blog. Lets take the case of transparency with luminance. Why cant we plug in transparency into the surface shader directly? Well, because a surface shader’s colour channel behaves pretty much like ncandacence. So it first calculates the transparency an dthen calculates the surface colour and blends it on top of the transparency. so if you have a full black surface colour, your transparency will behav corectly, if you have a fully white surface colour, the transparency will be completely gone and if you have anything in between, some of the transparency will be gone. By plugging the surface shader to a layered shader, you pass the colour info from your surface shader into the latter and since here, colour behaves as colour (not incandencence), transparency is calculated after the colour information is calculated and hence you get correct results. Whew!
Hope this helps some hapless soul like me :)