Introduction
This page describes how to mask part of an Augmented Reality (AR) object and show the real-world camera's image instead.
For example, here AR window is put on a real-world wall.
The scene in the window extends far beyond the actual window. But the scene is only visible inside the frame of the window because the rest is hidden with an AR mask.
A simpler example is to have the AR Mask function as a virtual floor which will hide any virtual objects that are under it. Here the chair is moved through this AR Mask floor:
You can think of it that after the virtual model is rendered, an AR Mask is rendered at the height of the floor. Parts of the AR objects that are hidden by the AR Mask will be replaced by the real-world background.
| 1: Virtual model |
2: Red AR Mask |
3: Real-world background |
NOTE: There are various ways to achieve this effect, this page describes the simplest methods. More complex situations may require more complex shaders, and methods.
Aximmetry SE - Scenes Rendered by Aximmetry
Aximmetry's Camera modules have a special render output called Mask. AR objects can be masked using this Mask render output, where the masks are models that have special shaders that write into the Mask render output.
AR Mask
Imported models can be used as Masks, but in this example, we are using Aximmetry's built-in rectangle body module as a model. This rectangle body will function as a mask for the real-world floor.
To align the rectangle with the floor, after adding the Rectangle module, set its Plane pin to XZ (1). It is also recommended to increase its dimensions to cover the whole floor by changing its X Start - Y End pins to a high value. The Scene Node module can be used to position the rectangle to where is your real-world floor. If you are using shadow or planar reflection mirror planes then you should position the rectangle slightly under them with the Scene Node.
Once the rectangle is correctly placed, connect the [Common_Studio]:Shaders\VCamU_Billboard_Mask.xshad shader to the Rectangle module. This is a special shader that also writes into the Mask render output.
Next, the rendered mask needs to mask out the rendered image. For this, you need to enter the Render General Linked Compound.
Inside the Render General compound, you use the Mask output pin of the Camera module. First, you invert this mask output with the Inverter module that you add. And then use it as a mask with the Masker module that you add.
Above, you only mask the solid render output of the Camera, in some cases, you also want to mask the Transp out, the Highlight Map, and any other rendered image by the camera.
The Camera will render a black background in the mask by default, this can cause artifacts at the borders of the mask. To prevent this, set the Mask Bg Color pin of the Camera module to fully transparent.
Now, if you move an AR object under the floor level through the AR Mask, it should be clipped by the real-world floor.
Debugging Mask
When editing your Mask, it is hard to see where are its boundaries, even when looking at the rendered Mask output.
To visualize the Mask while editing your scene, the following logic can be added. This logic blends the Mask's image over the final output.
The Solid Color module's color pin can change the color of the Mask.
You can select a color that is easily noticeable in your scene, for example red.
NOTE: This debugging image will not contain transparent images, for example, mirror reflections.
Complex Geometry for Mask
In the previous paragraph, the chair was clipped directly by the ground. This is not always the intended way to use AR Mask, for example, you may want to create a virtual window or door effect. Such geometries are easy to create and animate in the Flow Editor.
You can achieve the above circular mask by using the Disc Section body module instead of the Rectangle module. When setting a positive value for the Radii Inner pin of the Disc Section module, a hole will appear in the geometry. Make sure, you set the Plane pin to XZ (1) and give the Radii Outer pin a big enough value to cover all your intended virtual floor.
Since there is nothing under the circle, you see the real-world image. But you can easily add a box or any other geometries that will function as the hole's interior and will hide the real-world image.
In this case, a Box body module is added. Where the texture is facing inside cause the Flip pin is turned on.
Since in the above image the Box was set to 3x3x3 in dimensions, the connected Scene Node's height is set to a bit more than minus 3, so it wouldn't occlude with the mask, reflection plane, and shadow plane.
NOTE: If you want to achieve the same mask but with a rectangular shape, then all you need to do is to set the Circum Segments pin of the Disc Section module to 4. If you want an oval shape, then you can adjust the scales in the Scene Node if you set the Transformation pin input mode to Pos-Rot-ScalXYZ.
To animate the opening and closing of the circular hole and the height of AR objects, you can use the Sequence Editor and Sequence modules.
Further Considerations
Alternatively, it is possible to just use a cutout shader like [Common]:Shaders\Special\Cut.xshad. The cutout shader will render the alpha into the rendered image, this is visible when looking at the Out output pin of the Camera module. Since this method already renders everything with the alpha, there is no need to add additional logic with masks in the Render_General compound. However, this can lead to complications when using post-process effects.
NOTE: All the above SE methods are useable with non-AR camera compounds. However, camera compounds with billboards will also render the billboard's mask into the Mask output pin of the Camera module and this can lead to complications.
Aximmetry DE - Scenes Rendered by Unreal
This paragraph describes how to make an AR mask in Unreal. Initially, you must establish an AR scene within Unreal and Aximmetry by adhering to the guidelines found in this documentation:
There are two main methods for generating AR masks in Unreal:
- Geometry: This method involves using an object's geometry as a mask. However, it is important to note that this method does not fully support transparent materials.
- Material: This technique requires configuring the logic within objects' material to delineate an area where pixels should remain invisible. While this method is compatible with transparent materials, its primary drawback lies in the complexity of adjusting settings for irregular boundaries as geometries cannot function as masks in this method. It is relatively straightforward to apply for fixed boundaries (for instance, a uniformly elevated floor). Furthermore, it necessitates modifying every material potentially influenced by the mask.
Geometry Method
We will add a virtual window into the starter content scene of Unreal by making a wall to mask out the AR objects. But of course, you can do this with any other AR objects or scenes.
First, you create a new material:
Open the material and then set that material's Blend Mode to AlphaHoldout and set Shading Model to Unlit:
And set Translucency Pass to Before DOF:
Save the material and then apply the material around the window on meshes.
In the starter content scene, the window was already surrounded by a wall, so applying the material on the wall was enough to achieve to desired effect:
After cooking the project and running it in Aximmetry, you should see the camera's image around the window:
Complex Geometry and Texture for Mask
In most cases, you don't need complex geometry to make simple masks like the above wall in the previous paragraph. You can put together some of the basic shapes of Unreal to achieve the same effect without using any modeling software. For example, you can make a wall around a window by putting together 4 planes or cubes.
You can also use textures to define the Mask on geometry. For this, you need to connect a texture to the Opacity of the material:
You can define a gradient transition with the texture:
NOTE: By using the Get Aximmetry Video blueprint node, you can even have a video from Aximmetry as an animated mask texture in Unreal.
Transparent Objects
The Alphaholdout blend mode is unable to clip transparent objects. Instead, it transitions abruptly between appearing in front of them or behind them, depending on the actor's placement.
There are several methods to determine the order in which transparent objects are rendered.
For instance, by setting an actor's Translucency Sort Priority to -1, and assuming no other transparent actors have adjusted their Translucency Sort Priority, this configuration should position the Alphaholdout Mask behind every transparent object within the scene:
Translucency Sort Priority will not function with materials whose Translucency Pass is not set to Before DOF, which is the same setting used by the Alpha Holdout material.
For additional information about handling transparency in Unreal, please refer to the documentation from Unreal.
Material Method
In instances where your project includes transparent objects that might clip the mask's plane, there is an alternative method available. This method involves incorporating logic into the materials of these objects, ensuring they do not display pixels located within a specified position range.
For instance, consider a scenario with a flat floor where the goal is to hide any pixels beneath this surface. To achieve this, one can utilize the pixel's world height position value, opting not to display any pixels situated below the floor's elevation. This process can be accomplished by accessing the pixel's position through the World Position node within the material. Subsequently, you can apply the following logic to adjust the opacity to zero for any pixels whose World Position height falls below a predetermined value of floor height:
NOTE: The above logic produces a hard cut. However, with slightly more complex logic, it is possible to implement soft cuts or various other transition effects.
Sky
Usually in AR scenes, the whole background is the real-world camera's image and there is no need to show any artificial sky. This is why by default, the sky is not shown in Unreal AR cameras.
However, in very rare cases, it is desired to show some background sky, like in the above example with the window into a different world.
In this case, SM_SkySphere mesh can be used with sky materials.
The SM_SkySphere mesh can only be found under the Engine folder in the Content Browser.
NOTE: You will also need to set the AximmetryAR tag on this Actor object.
NOTE: HDRI sky maps can also be used as a material. And Unreal's HDRI Backdrop plugin will also work with AR mask.
