ENGLISH l ESPAÑOL

What the Stencil Buffer is? 

Just like the Color buffer and the Z-Buffer, this contains extra information commonly storing integer values and it's used to limit the "rendering area" of some kind of objects on screen and say to the camera the what are the conditions to paint those objects.

The Stencil buffer has many advanced uses in vfx, but for this article we're going to teach you a simple example about how to use it with shades in Unity.

REQUIREMENTS

• It works with every current render pipeline, in my case i will use URP. 
• Basic knowledge of shader programming with HLSL.
• Basic knowledge with Unity.
• In my case i am using Unity 2020.3 but this should work with any other version with no problem.

EJERCICIO

We're going to develop a mask using shaders that will allow us to paint on screen objects that are just in the correct position in relation to our mask and camera.  

SYNTAX

For this we have to use two different shaders, one for the "mask object" (the object that will work as a 3d mask) and the object that it will only be painted just when it is inside of the mask silhouette. 

I'm going to create two different "Unlit Shaders" for this example, one for the mask and another for the masked object. 

We're going to write the next code in the "SubShader" Area and before the "Pass" area, remember that these are two different shaders. 

• ref: An Integer that is assigned to the shader and the buffer will read, this will allow the buffer to compare the "ref" number with other shaders to know how to behave. This integers could be from 0 to 255. 

• comp: The comparison operation that the shader have to do to know if the object have to be rendered or not. If the comparison is a success, the "pass" will be executed and hence the "fragment output".  

• pass: This section indicates what to do with the ref number in relation to the stencil buffer, in this case, we're going to use "replace", this will write the ref number in our stencil buffer. 

-'equal' It refers that the pixels of the masked object only will be painted on the screen when the ref value is the same in the stencil buffer as in the shader. 

-'always' It refers that the pixels of the masked object will always be painted on screen under these conditions. 

Next, we're going to create a plane and a sphere as testing objects, let's put the sphere in front of our plane.

Let's create two materials, each one with our new shaders that we just wrote; Assign the mask material to the plane (this will limit the rendering area of the sphere), and we're going to assign the other material to the sphere.

Now we can see how our plane is acting like a mask limiting the rendering area of our sphere. 

This happens because the plane is substituting the value inside our stencil buffer to 1 (just in that area), and in our sphere shader we're telling to it that the sphere only will execute the "pass" of its shader (and with this the fragment output) only if the value is the same as the stencil buffer value (in this case is 1). 

NOTE: It's possible that at the moment we assign the materials to the sphere or the plane these flickers or just disappear, to solve this you just need to add +1 to the render queue of the sphere's material, this is to tell to the z-buffer that this object have to be painted in front of the plane (or first), otherwise, these two are competing all the time for being painting in the same space.  

Additional features

In many occasions we don't want that our mask has any type of color, to do this is very simple, we just have to add one code line "ColorMask 0" in the subshader area, this will stop the shader from write any kind of RGBA channel, so, our mask will have the same function but without any color.

But, what happens if we want that our mask goes in front of the masked object? Right now our mask only works if it's behind of the sphere, this is cause both objects are "Opaque" type, and if the mask would be in front of our sphere, our buffer could not see the sphere object and therefore would not register it, I mean, there would be no way to compare both ref values of our shaders. 

To solve this, again, we just need to add a code line in our mask shader in the "SubShader" area, we're going to write "ZWrite Off" to enable the object's transparency, now the camera can see both objects no matter which one is in front or back. 

We are going to change the "RenderType" to "Transparent". 

Now our mask works no matter if it's in front or in the back of our sphere.

Now you can use the shader masks to create great VFX or just add them to your game mechanics. I made this card to test this kind of effects.

DOWNLOAD

If you are interested to directly download the shaders and a scene sample you can do it here: 

DOWNLOAD SHADER

You are free to use them for any project you want when you got it!  

For any kind of doubt, just let me know in the comments or send me a direct message here. If you want to make a donation for the site you can do it in my Ko-fi account, this will help me a lot to continue creating content for you!