Introduction
This page describes how to mask part of an AR object and show the real-world camera's image instead.
For example, here AR window is put on a real-world wall.
The scene in the window extends far beyond the actual window. But the scene is only visible inside the frame of the window because the rest is hidden with an AR mask.
A simpler example is to have the AR Mask function as a virtual floor which will hide any virtual objects that are under it. Here the chair is moved through this AR Mask floor:
You can think of it that after the virtual model is rendered, an AR Mask is rendered at the height of the floor. And parts of the AR objects which are hidden by the AR Mask will be replaced by the real-world background.
| 1: Virtual model |
2: Red AR Mask |
3: Real-world background |
There are various ways to achieve this effect, this page describes the simplest methods. More complex situations may require more complex shaders, and methods.
Aximmetry SE in scenes rendered by Aximmetry
Aximmetry's Camera modules have a special render output called Mask. AR objects can be masked using this Mask render output, where the masks are models that have special shaders which write into the Mask render output.
AR Mask
Imported models can be used as Masks, but in this example, we are using Aximmetry's built-in rectangle body module as a model. This rectangle body will function as a mask for the real-world floor.
To align the rectangle with the floor, after adding the Rectangle module, set its Plane pin to XZ (1). It is also recommended to increase its dimensions to cover the whole floor by changing its X Start - Y End pins to a high value. The Scene Node module can be used to position the rectangle to where is your real-world floor. If you are using shadow or planar reflection mirror planes then you should position the rectangle slightly under them with the Scene Node.
Once the rectangle is correctly placed, connect the [Common_Studio]:Shaders\VCamU_Billboard_Mask.xshad shader to the Rectangle module. This is a special shader that also writes into the Mask render output.
Next, the rendered mask needs to mask out the rendered image. For this, you need to enter the Render General Linked Compound.
Inside the Render General compound, you use the Mask output pin of the Camera module. First, you invert this mask output with the Inverter module that you add. And then use it as a mask with the Masker module that you add.
Above, you only mask the solid render output of the Camera, in some cases, you also want to mask the Transp out, the Highlight Map, and any other rendered image by the camera.
The Camera will render a black background in the mask by default, this can cause artifacts at the borders of the mask. To prevent this, set the Mask Bg Color pin of the Camera module to fully transparent.
Now, if you move an AR object under the floor level through the AR Mask, it should be clipped by the real-world floor.
Debugging Mask
When editing your Mask, it is hard to see where are its boundaries, even when looking at the rendered Mask output.
To visualize the Mask while editing your scene, the following logic can be added. This logic blends the Mask's image over the final output.
The Solid Color module's color pin can change the color of the Mask. You can select a color that is easily noticeable in your scene, for example red.
This debugging image will not contain transparent images, for example, mirror reflections.
Complex Geometry for Mask
In the previous paragraph, the chair was clipped directly by the ground. This is not always the intended way to use AR Mask, for example, you may want to create a virtual window or door effect. Such geometries are easy to create and animate in the Flow Editor.
You can achieve the above circular mask by using the Disc Section body module instead of the Rectangle module. When setting a positive value for the Radii Inner pin of the Disc Section module, a hole will appear in the geometry. Make sure, you set the Plane pin to XZ (1) and give the Radii Outer pin a big enough value to cover all your intended virtual floor.
Since there is nothing under the circle, you see the real-world image. But you can easily add a box or any other geometries that will function as the hole's interior and will hide the real-world image.
In this case, a Box body module is added. Where the texture is facing inside cause the Flip pin is turned on.
Since in the above image the Box was set to 3x3x3 in dimensions, the connected Scene Node's height is set to a bit more than minus 3, so it wouldn't occlude with the mask, reflection plane, and shadow plane.
If you want to achieve the same mask but with a rectangular shape, then all you need to do is to set the Circum Segments pin of the Disc Section module to 4. If you want an oval shape, then you can adjust the scales in the Scene Node if you set the Transformation pin input mode to Pos-Rot-ScalXYZ.
To animate the opening and closing of the circular hole and the height of AR objects, you can use the Sequence Editor and Sequence modules.
Further Considerations
Alternatively, it is possible to just use a cutout shader like [Common]:Shaders\Special\Cut.xshad. The cutout shader will render the alpha into the rendered image, this is visible when looking at the Out output pin of the Camera module. Since this method already renders everything with the alpha, there is no need to add additional logic with masks in the Render_General compound. But this can lead to complications when using post-process effects.
Note all the above SE methods are useable with non-AR camera compounds. However, camera compounds with billboards will also render the billboard's mask into the Mask output pin of the Camera module and this can lead to complications.
Aximmetry DE in scenes rendered by Unreal
This paragraph describes how to make an AR mask in Unreal. First, you will need to set up an AR scene in Unreal and Aximmetry following this documentation: AR workflow in Aximmetry DE
We will add a virtual window into the starter content scene of Unreal by making a wall to mask out the AR objects. But of course, you can do this with any other AR objects or scenes
First, you create a new material.
Open the material and then set that material's Blend Mode to AlphaHoldout and set Shading Model to Unlit.
Save the material and then apply the material around the window on meshes.
In the starter content scene, the window was already surrounded by a wall, so applying the material on the wall was enough to achieve to desired effect.
After cooking the project and running it in Aximmetry, you should see the camera's image around the window.
In most cases, you don't need complex geometry to make simple masks like the above wall.
You can put together some of the basic shapes of Unreal to achieve the same effect without using any modeling software.
For example, you can make a wall around a window by putting together 4 planes or cubes.
Sky
Usually in AR scenes, the whole background is the real-world camera's image and there is no need to show any artificial sky. This is why by default, the sky is not shown in Unreal AR cameras.
However, in very rare cases, it is desired to show some background sky, like in the above example with the window into a different world.
In this case, SM_SkySphere mesh can be used with sky materials.
You will also need to set the AximmetryAR tag on this Actor object.
HDRi sky maps can also be used as a material. And Unreal's HDRI Backdrop plugin will also work with AR mask.
