Introduction
This document details the process of rendering a scene using either Unreal Engine or Aximmetry and overlaying an Augmented Reality (AR) scene or elements from the other engine.
Aximmetry is capable of rendering scenes or AR elements just like Unreal. For guidance on creating scenes within Aximmetry SE, please refer to the Creating Content in Aximmetry SE documentation. For creating AR elements in Aximmetry SE, continue with the Aximmetry SE Scene Setup (AR) documentation after that.
NOTE: Combined rendering can also be achieved using multiple machines, serving as an effective strategy to distribute and reduce overall performance load. For those interested in implementing such a setup, the Combine Different Productions in Separate Machines documentation could be regarded as complementary material to guide you through the process.
Unreal Engine Scene with Aximmetry AR Elements
Imagine you have created a studio scene in Unreal Engine and wish to enhance it with 3D elements rendered by the Aximmetry engine.
In Unreal, add an Aximmetry camera— for example, a Green camera (virtual, 1-3 billboards) — and set it up in Aximmetry according to the documentation.
Next, connect your Aximmetry-rendered elements to the camera compound's AR Overlay pin. For instance, use one of the [Common_Studio]:Compounds\Render\Render_General compounds as the rendering camera. These compounds automatically receive the camera data from Camera compounds (VirtualCam_Unreal_3-Cam).
Then connect the root Scene Node of the Aximmetry-rendered elements to the Render_General:
The Aximmetry-rendered elements should appear over the Unreal scene and move synchronously when the camera moves, provided they are precisely positioned to the Unreal scene. To convert positions from Unreal to Aximmetry's coordinate system, use the [Common_Studio]:Compounds\Tools\Unreal_Transformation.xcomp compound.
Billboard Position in Green Cameras
To adjust the Billboard layering order:
- in Virtual Camera compounds, navigate to the CAMERAS control board and select the CAMERA & RENDER SETUP panel.
- In Tracked Camera compounds, navigate to the INPUTS control board and select the SCENE panel.
To display the Billboard in front of the Aximmetry elements turn AR Overlay Behind On:
NOTE: Aximmetry renders the Box with the various colors in the above image.
To display the Billboard behind the Aximmetry elements turn AR Overlay Behind Off:
IMPORTANT: Turning Allow Virtuals On causes Aximmetry elements to consistently appear on top of the billboard, regardless of the AR Overlay Behind setting.
Talent Position in LED Wall Cameras
When the AR Overlay To LED is turned on in the STUDIO panel, then the AR-rendered graphics are displayed on the LED wall, placing the AR elements behind the talent. Conversely, when this setting is deactivated, the AR content is displayed in front of the talents. This occurs because the AR content is blended over the camera's live feed rather than being integrated with the imagery displayed on the LED wall:
Complex Examples
Aximmetry packages include a variety of compounds and examples that demonstrate the 3D functionalities of Aximmetry. These compounds can be useful for virtual production.
You can conveniently modify these compounds and integrate them as AR elements. For instance, let's utilize the [Tutorials]:Extrusion\Text_Extrusion_Curved.xcomp compound.
You can drag and drop this compound into the Flow Editor, then unlink it:
Next, you will need to ungroup the compound:
Following this, you should connect its root Scene Node to the Render_General compound:
NOTE: The compound's position can be altered from the root Scene Node.
After completing these steps, you should be able to see it overlaid on the Unreal content. However, in the case of the Text_Extrusion_Curved compound, you will probably notice the black floor (reflection) obscuring the Unreal content.
This issue can be fixed by setting the Mirror_Norm shader module to be transparent. You can achieve this by changing its Transparency to Add, and setting its Color to black.
However, the reflection will continue to utilize the old camera instead of the Render_General camera. To fix this, rewire the Mirror Camera's Bound To pin to the Render_General's Bindings pin:
Afterward, you can change for example the text displayed in the Text_Curved compound:
By following the steps mentioned above, you should obtain a final image similar to the one below, where the sphere, the 3D text, and their reflections are rendered by Aximmetry:
NOTE: In the above image, the billboard does not have a reflection; only elements rendered by Aximmetry do. To set up the reflection for the billboard, please follow the Unreal reflection guidelines.
NOTE: The reflection's shader (Mirror_Norm) employs a normal map to reduce the smoothness of the reflection. It might be desirable to reset the Normal Map pin of the shader or implement a similar approach for reflections in Unreal.
Aximmetry Scene with Unreal AR Elements
Imagine you have crafted a studio scene using Aximmetry and you want to complement it with 3D elements rendered by the Unreal Engine. This will walk you through the process of integrating Augmented Reality (AR) elements from Unreal into your Aximmetry scene.
To begin, you will need to create an Unreal AR project. Set up the Unreal AR camera according to the Aximmetry DE Scene Setup (AR) documentation.
LED Wall Cameras
When dealing with LED Wall cameras, the setup mirrors that are outlined in the AR in Green and LED Wall Projects of the Unreal documentation. This is because both Aximmetry and Unreal projects use the same LED Wall compounds in Aximmetry.
NOTE: To modify the position of AR elements relative to the scene rendered by Aximmetry, you should expose the transformation of the root object of Unreal Engine elements. To create a root object in Unreal Engine, group all elements under a single Actor within Unreal Engine's Outliner panel.
Green Cameras
There is no out-of-the-box solution for integrating Unreal AR graphics with Green cameras in Aximmetry. However, constructing a logic for putting the Unreal AR graphics in front of the Billboard and the scene is relatively easy.
To achieve this, the [Common_Studio]:Camera\ARCam_Unreal\Elements\ARU_Overlay.xcomp and the [Common]:Compounds\Utilities\Blend_Linear.xcomp compounds need to process the Unreal AR video and overlay it onto the Aximmetry-rendered video. We strongly recommend enabling the Linear pin in the Blend_Linear compound; its function is detailed in the AR Linear Blend part of the Unreal AR camera documentation.
Next, you need to put together the Control Data for the Unreal AR Camera. The Record Data collection pin contains most necessary data; you only need to add the correct FOV to it in the Cam FOV collection Key by using a FOV Calculator and Set Collection Scalar module:
Upon completion, you should get a combined visual representation akin to the one shown below, in this case featuring a Metahuman woman rendered by Unreal:
The Unreal-rendered elements should appear over the Aximmetry scene and move synchronously when the camera moves, provided they are precisely positioned to the Aximmetry scene.
