Just like the Color buffer and the Z-Buffer, this contains extra information commonly storing integer values and it's used to limit the "rendering area" of some kind of objects on screen and say to the camera the what are the conditions to paint those objects.
The Stencil buffer has many advanced uses in vfx, but for this article we're going to teach you a simple example about how to use it with shades in Unity.
- It works with every current render pipeline, in my case i will use URP.
- Basic knowledge of shader programming with HLSL.
- Basic knowledge with Unity.
- In my case i am using Unity 2020.3 but this should work with any other version with no problem.
For this we have to use two different shaders, one for the "mask object" (the object that will work as a 3d mask) and the object that it will only be painted just when it is inside of the mask silhouette.
I'm going to create two different "Unlit Shaders" for this example, one for the mask and another for the masked object.
We're going to write the next code in the "SubShader" Area and before the "Pass" area, remember that these are two different shaders.
- ref: An Integer that is assigned to the shader and the buffer will read, this will allow the buffer to compare the "ref" number with other shaders to know how to behave. This integers could be from 0 to 255.
- comp: The comparison operation that the shader have to do to know if the object have to be rendered or not. If the comparison is a success, the "pass" will be executed and hence the "fragment output".
- pass: This section indicates what to do with the ref number in relation to the stencil buffer, in this case, we're going to use "replace", this will write the ref number in our stencil buffer.
-'equal' It refers that the pixels of the masked object only will be painted on the screen when the ref value is the same in the stencil buffer as in the shader.
-'always' It refers that the pixels of the masked object will always be painted on screen under these conditions.
Next, we're going to create a plane and a sphere as testing objects, let's put the sphere in front of our plane.
Let's create two materials, each one with our new shaders that we just wrote; Assign the mask material to the plane (this will limit the rendering area of the sphere), and we're going to assign the other material to the sphere.
Now we can see how our plane is acting like a mask limiting the rendering area of our sphere.
This happens because the plane is substituting the value inside our stencil buffer to 1 (just in that area), and in our sphere shader we're telling to it that the sphere only will execute the "pass" of its shader (and with this the fragment output) only if the value is the same as the stencil buffer value (in this case is 1).
NOTE: It's possible that at the moment we assign the materials to the sphere or the plane these flickers or just disappear, to solve this you just need to add +1 to the render queue of the sphere's material, this is to tell to the z-buffer that this object have to be painted in front of the plane (or first), otherwise, these two are competing all the time for being painting in the same space.
In many occasions we don't want that our mask has any type of color, to do this is very simple, we just have to add one code line "ColorMask 0" in the subshader area, this will stop the shader from write any kind of RGBA channel, so, our mask will have the same function but without any color.