A stencil buffer is an extra data buffer, in addition to the color buffer and Z-buffer, found on modern graphics hardware. The buffer is per pixel and works on integer values, usually with a depth of one byte per pixel. The Z-buffer and stencil buffer often share the same area in the RAM of the graphics hardware.

In the simplest case, the stencil buffer is used to limit the area of rendering (stenciling). More advanced usage of the stencil buffer makes use of the strong connection between the Z-buffer and the stencil buffer in the rendering pipeline. For example, stencil values can be automatically increased/decreased for every pixel that fails or passes the depth test.

The simple combination of depth test and stencil modifiers make a vast number of effects possible (such as stencil shadow volumes, Two-Sided Stencil,^[1] compositing, decaling, dissolves, fades, swipes, silhouettes, outline drawing, or highlighting of intersections between complex primitives) though they often require several rendering passes and, therefore, can put a heavy load on the graphics hardware.

The most typical application is still to add shadows to 3D applications. It is also used for planar reflections.

Other rendering techniques, such as portal rendering, use the stencil buffer in other ways; for example, it can be used to find the area of the screen obscured by a portal and re-render those pixels correctly.

The stencil buffer and its modifiers can be accessed in computer graphics by using APIs like OpenGL, Direct3D, Vulkan or Metal.

The stencil buffer typically shares the same memory space as the Z-buffer, and typically the ratio is 24 bits for Z-buffer + 8 bits for stencil buffer or, in the past, 15 bits for Z-buffer + 1 bit for stencil buffer. Another variant is 4 + 24, where 28 of the 32 bits are used and 4 ignored. Stencil and Z-buffers are part of the frame buffer, coupled to the color buffer. The first chip available to a wider market was 3Dlabs' Permedia II, which supported a one-bit stencil buffer.

The bits allocated to the stencil buffer can be used to represent numerical values in the range [0, 2ⁿ-1], and also as a Boolean matrix (n is the number of allocated bits), each of which may be used to control the particular part of the scene. Any combination of these two ways of using the available memory is also possible.

Stencil test or stenciling is among the operations on the pixels/fragments (Per-pixel operations), located after the alpha test, and before the depth test. The stencil test ensures undesired pixels do not reach the depth test. This saves processing time for the scene. Similarly, the alpha test can prevent corresponding pixels to reach the stencil test.

The test itself is carried out over the stencil buffer to some value in it, or altered or used it, and carried out through the so-called stencil function and stencil operations. The stencil function is a function by which the stencil value of a certain pixel is compared to a given reference value. If this comparison is logically true, the stencil test passes. Otherwise not.

In doing so, the possible reaction caused by the result of comparing three different state-depth and stencil buffer:

• Stencil test is not passed
• Stencil test is passed but not the depth test
• Both tests are passed (or stencil test is passed, and the depth is not enabled)

For each of these cases can be set different operations over the examined pixel. In the OpenGL stencil functions, the reference value and mask, respectively, define the function glStencilFunc. In Direct3D each of these components is adjusted individually using methods SetRenderState devices currently in control. This method expects two parameters, the first of which is a condition that is set and the other its value. In the order that was used above, these conditions are called D3DRS_STENCILFUNC, D3DRS_STENCILREF, and D3DRS_STENCILMASK.

Stencil operations in OpenGL adjust glStencilOp function that expects three values. In Direct3D, again, each state sets a specific method SetRenderState. The three states that can be assigned to surgery are called D3DRS_STENCILFAIL, D3DRENDERSTATE_STENCILZFAIL, and D3DRENDERSTATE_STENCILPASS.

Due to the lack of precision in the Z-buffer, coplanar polygons that are short-range, or overlapping, can be portrayed as a single plane with a multitude of irregular cross-sections. These sections can vary depending on the camera position and other parameters and are rapidly changing. This is called Z-fighting. There exist multiple solutions to this issue:

- Bring the far plane closer to restrict the scene's depth, thus increasing the accuracy of the Z-buffer, or reducing the distance at which objects are visible in the scene.

- Increase the number of bits allocated to the Z-buffer, which is possible at the expense of memory for the stencil buffer.

- Move polygons farther apart from one another, which restricts the possibilities for the artist to create an elaborate scene.

All of these approaches to the problem can only reduce the likelihood that the polygons will experience Z-fighting, and do not guarantee a definitive solution in the general case.

A solution that includes the stencil buffer is based on the knowledge of which polygon should be in front of the others. The silhouette of the front polygon is drawn into the stencil buffer. After that, the rest of the scene can be rendered only where the silhouette is negative, and so will not clash with the front polygon.

Shadow volume is a technique used in 3D computer graphics to add shadows to a rendered scene. They were first proposed by Frank Crow in 1977^[2] as the geometry describing the 3D shape of the region occluded from a light source. A shadow volume divides the virtual world in two: areas that are in shadow and areas that are not.

The stencil buffer implementation of shadow volumes is generally considered among the most practical general-purpose real-time shadowing techniques for use on modern 3D graphics hardware. It has been popularised by the video game Doom 3, and a particular variation of the technique used in this game has become known as Carmack's Reverse.

Reflection of a scene is drawn as the scene itself transformed and reflected relative to the "mirror" plane, which requires multiple render passes and using of stencil buffer to restrict areas where the current render pass works:

1. Draw the scene excluding mirror areas – for each mirror lock the Z-buffer and color buffer
   1. Render visible part of the mirror
   2. Depth test is set up so that each pixel is passed to enter the maximum value and always passes
2. for each mirror:
   1. Depth test is set so that it passes only if the distance of a pixel is less than the current (default behavior)
   2. The matrix transformation is changed to reflect the scene relative to the mirror plane
   3. Unlock the Z-buffer and color buffer
   4. Draw the scene, but only the part of it that lies between the mirror plane and the camera. In other words, a mirror plane is also a clipping plane
   5. Again locks color buffer, depth test is set so that it always passes, reset stencil for the next mirror.

While drawing a plane of shadows, there are two dominant problems: The first concerns the problem of deep struggle in case the flat geometry is not awarded on the part covered with the shadow of shadows and outside. See the section that relates to this. Another problem relates to the extent of the shadows outside the area where the plane there.

Another problem, which may or may not appear, depending on the technique, the design of more polygons in one part of the shadow, resulting in darker and lighter parts of the same shade. All three problems can be solved geometrically, but because of the possibility that hardware acceleration is directly used, it is a far more elegant implementation using the stencil buffer: 1. Enable lights and the lights 2. Draw a scene without any polygon that should be projected shadows 3. Draw all polygons which should be projected shadows, but without lights. In doing so, the stencil buffer, the pixel of each polygon to be assigned to a specific value for the ground to which they belong. The distance between these values should be at least two, because for each plane to be used two values for two states: in the shadows and bright. 4. Disable any global illumination (to ensure that the next steps will affect only individual selected light) For each plane: For each light: 1. Edit a stencil buffer and only the pixels that carry a specific value for the selected level. Increase the value of all the pixels that are projected objects between the date of a given level and bright. 2. Allow only selected light for him to draw level at which part of her specific value was not changed.

Stencil buffer implementation of spatial drawing shadows is any shadow of a geometric body that its volume includes part of the scene that is in it. If any part of the scene belongs to this volume, light is not illuminated given, otherwise it is. This problem is compounded by the increase in the number of lights but does not address the number of areas on which the shadows fall. There are several solutions to the problem, but we followed the following algorithm: 1. Draw a scene without light 2. Lock the Z-buffer and color buffer, so that the two can not make changes For each light 1. Using in-depth information about the scene (Z-buffer) to fill the stencil buffer only on parts of the scene where volume shadow does not exist or are not visible from the existing buildings. 2. Unlock buffer for color, and adjust the function of the Z-buffer to allow amendments only where the depth value equal to an existing 3. Draw the scene illuminated only by this light, but only for part of the scene passing the stencil test

Each of these passages implies that a clean stencil buffer can be used.

As for the shadows, this technique can be used to illuminate parts of space that are under strong light. For example, the brightness of the spotlight in a dark room with a large presence of dust in the air could be seen illuminating the appropriate volume of space.

A further example is the so-called soft shadow, in which the transition between the illuminated and shadowed part of the scene is out of focus. Specifically, one way to achieve this effect stencil buffer is to multiply the volume of the shadow, and that as the copies, respectively are scaled according to a geometric series with a low magnification, e.g.,. 1.04. The Center of scaling can be the center of gravity of the polygon that represents the top volume. This in itself will give a series of composite shadows that give the desired effect.

Another implementation includes the field of visualization during the modeling technique solids Constructive Solid Geometry (CSG), wherein stencil buffer, together with the Z-buffer, can successfully solve the problems of the Boolean operations of the SOLiD .

glEnable(GL_STENCIL_TEST); // by default not enabled
glStencilMask(stencilMask); // allow writing to stencil buffer, by default (0xFF) no mask.
glClearStencil(clearStencilValue); // clear stencil value, by default = 0
glStencilFunc(func, ref, mask); // by default GL_ALWAYS, 0, 0xFF, always pass stencil test
glStencilOp(fail,zfail,zpass); // by default GL_KEEP, GL_KEEP, GL_KEEP, do not change stencil buffer
glClear(GL_STENCIL_BUFFER_BIT); // clear stencil buffer, fill with (clearStencilValue & stencilMask)

Test: ( ref & mask ) func (stencilValue & mask)

Depending on the three possible conditions of the stencil function/depth function.

1. Stencil Test Function fails:

   If say func is GL_NEVER, the stencil test will always fail. 
   Neither Color/Z-buffers are modified. The stencil buffer is modified as per glStencilOp fail.
   If say glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP) then GL_REPLACE takes place and 
   stencilValue = (ref & stencilMask) // will become ref

2. Stencil Test Function passes/Depth Test Function fails:

  If say func is GL_ALWAYS, the stencil test will always pass, but depth test may fail.
  Neither Color/Z-buffer are modified. The stencil buffer is modified as per glStencilOp zfail.
  If say glStencilOp(GL_KEEP, GL_INCR, GL_KEEP) then GL_INCR takes place and
   stencilValue = (stencilValue+1) // will become 1

3. Stencil Function passes/Depth Function passes:

  If say func is GL_ALWAYS, the stencil test will always pass. If the depth test also passes.
  Both Color/Z-buffer are modified. The stencil buffer is modified as per glStencilOp zpass.
  If say, glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP) then Stencil values are not changed, only Color and Z-buffers are modified.

Typically Stencil buffer is initialized by setting Z-buffer and color buffer masks to false. and then setting appropriate ref value to stencil buffer by failing the stencil test every time.

  // disable color and Z-buffers
  glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  glDepthMask(GL_FALSE);

  glStencilFunc(GL_NEVER, 1, 0xFF); // never pass stencil test
  glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);  // replace stencil buffer values to ref=1
  glStencilMask(0xFF); // stencil buffer free to write
  glClear(GL_STENCIL_BUFFER_BIT);  // first clear stencil buffer by writing default stencil value (0) to all of stencil buffer.
  draw_stencil_shape(); // at stencil shape pixel locations in stencil buffer replace stencil buffer values to ref = 1

Now use the initialized stencil buffer and stencil test to write only in the locations where the stencil value is 1:

  // enable color and Z-buffers.
  glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
  glDepthMask(GL_TRUE);

  // no more modifying of stencil buffer on stencil and depth pass.
  glStencilMask(0x00);
  // can also be achieved by glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

  // stencil test: only pass stencil test at stencilValue == 1 (Assuming depth test would pass.) 
  // and write actual content to depth and color buffer only at stencil shape locations.
  glStencilFunc(GL_EQUAL, 1, 0xFF);
 
  draw_actual_content();

• Z-buffering (depth buffer)
• Shadow volume (a stencil buffer technique creating shadows)