Search
Start typing to search...

How to install and work with the Unreal Engine based DE edition

Author:

Introduction

This document describes what is Aximmetry Dual Engine (Aximmetry DE), how to install it, how to get started with the included example projects and how to create your own projects.

Aximmetry DE is a combination of Aximmetry and our customized version of the Unreal Engine: Unreal Engine for Aximmetry. Aximmetry DE lets you create your virtual studios in the Unreal Editor and use them in Aximmetry to benefit both from the world building and rendering features of Unreal and the advanced virtual studio capabilities of Aximmetry.

To get started, read through this document, install Aximmetry DE, take a look at the examples and start experimenting with your own projects.

How to install Aximmetry DE

  • Run the Aximmetry DE installer
    • In the "Select Projects Folder" step, you can select which content packages should be downloaded and installed.
    • Install the "Common Library" package.
    • Install the “Tutorials and Examples” package. This contains some example projects.
    • The "Inventory" and "Studio: Demo Sets" packages are not required but we recommend you install these too.
    • The rest of this document will refer to the folder selected in this step as Projects.
  • The Aximmetry DE installer will download and launch the Unreal Engine for Aximmetry installer
    • We suggest you install Unreal Engine for Aximmetry to the same folder as Aximmetry (which is the default) but you can install it anywhere you want.

How to load the example projects in Aximmetry

We included 2 example projects to help you get started:

Basic ([Tutorials]:Unreal\Basic.xcomp) shows a minimal setup: a very simple Unreal scene with a virtual camera and a single billboard controlled by Aximmetry. A billboard can be described as the keyed camera image of a talent displayed on a plane in 3D space.

Complex ([Tutorials]:Unreal\Complex.xcomp) shows some more advanced features:

  • transparent objects in front of the billboard
  • high quality reflections
  • 2 billboards with some advanced settings (one of them is affected by lights in the scene, the other is not)
  • virtual camera movement
  • controlling the position of a light from Aximmetry

To use the example project:

  1. Open the Unreal project in Unreal Editor for Aximmetry. The Unreal project file can be found at [Tutorials]:Unreal\Basic\Basic.uproject.
  2. Click File / Cook Content for Windows. This converts the project into a format that can be loaded by Aximmetry.
  3. After cooking is complete, the Unreal Editor is not needed anymore. You can close it if you want.
  4. Open the Aximmetry compound ([Tutorials]:Unreal\Basic.xcomp) in Aximmetry.
  5. After the project is loaded you should see the rendered scene in your output window.

The same steps apply to the Complex project.

How to create your own projects

The basic workflow is the following (each step will be explained in detail in the next section).

Create a new Unreal project. Some project settings have to be modified as detailed below. Add an Aximmetry Virtual Camera to the scene. This is a custom Unreal package that facilitates connection between Aximmetry and Unreal, provides virtual camera functionality and renders the billboards. Edit the scene, build your virtual environment. After your scene is ready, you have to cook it, so it can be used by Aximmetry. Open Aximmetry, create a new compound and the drag the Unreal project file into it. After connecting the modules as detailed below you should have a working Aximmetry DE project.

In Unreal:

  • Create a new Unreal project. We recommend a blank blueprint project with no starter content. You can also use an existing project, in this case we recommend choosing the Open a Copy option when the project conversion window pops up.
  • Aximmetry Virtual Camera has to be able to write to a custom stencil. To enable this:
    • Go to Edit / Project Settings / Engine / Rendering / Postprocessing
    • Set Custom Depth-Stencil Pass to Enabled with Stencil
  • Enable clip planes:
    • Go to Edit / Engine / Rendering / Lighting
    • Enable Support global clip plane for Planar Reflections
  • By default, the player pawn will show up in the scene as a gray sphere. This can be resolved in 2 ways:
    • A) Set the player model to an invisible one:
      • Go to Edit / Project Settings / Project / Maps & Modes / Default Modes
      • Create a new game mode by pressing the + button (Give it a name and press OK)
      • The new game mode should be selected as the Default GameMode
      • Open the Selected GameMode section
      • Change the Default Pawn Class to None
    • B) Move the PlayerStart outside of the scene. In this case the gray sphere will spawned in an invisible area. (Its position is indifferent for Aximmetry.)
  • Set up the startup maps:
    • Save the map (File / Save Current)
    • Go to Edit / Project Settings / Project / Maps & Modes / Default Maps
    • Set both Editor Startup Map and Game Default Map to your map

Note: If you rename or move the map asset you have to clear and select the startup maps again because Unreal is not updating the project settings correctly.

  • Copy the Projects\Common_Studio\Unreal_Assets\Aximmetry_VirtualCam_3-Cam directory into the Content directory of your Unreal project. The Aximmetry_VirtualCam_3-Cam directory must be directly under the Content directory, don't place inside any subdirectory or outside the Content directory.
  • In the Content Browser go to Content / Aximmetry_VirtualCam_3-Cam / Blueprints. Drag the Aximmetry_Camera blueprint into the scene (the position and orientation doesn't matter, these will be set by Aximmetry).
  • Edit the scene, build your virtual environment
  • If you place objects in front of the billboard, you have to take some steps to avoid visual artifacts. The reason is that Unreal uses Temporal Anti-Aliasing (TAA) to make the edges of objects look smooth. The problem with TAA is that it makes the billboard look very blurry, so we disable it where the billboard is. By default, TAA will also be disabled for objects that are in front of the billboard. The lack of TAA might make the edge of these object look jagged. If this happens you can turn TAA back on for these objects. How you do this depends on whether the object is opaque or translucent. These steps have to be done individually for each object that occludes the billboard.
    • For opaque objects:
      • Select the object
      • In the Detail window, go to the Rendering section
      • Enable Render CustomDepth Pass
      • Set CustomDepth Stencil Value to 0
    • For translucent objects you have to do some additional steps:
      • Go to the material editor by double clicking the translucent material in the Content Browser
      • In the Details window, enable Translucency / Allow Custom Depth Writes
      • Set Material / Opacity Mask Clip Value to 0 or some other small value This is the alpha threshold above which TAA will be enabled.

Note: For translucent objects, enabling TAA can make the billboard behind it look blurry. This is not a problem if the object is not too transparent or the affected area is not too big. Otherwise you have to make a tradeoff between the blurriness of the billboard and the quality of the edge or avoid these situations entirely.

  • Save the project
  • Cook the project by clicking File / Cook Content for Windows
  • After cooking is complete, the Unreal Editor is not needed anymore. You can close it if you want.

In Aximmetry:

  • Create a new compound
  • Drag in the .uproject file from your Unreal project root directory. This will create an Unreal Project module node.
  • Select the Unreal Project module and set the Connection parameter to Cooked
  • Connect the Out pin to the output of the compound. Now you should see an image rendered by Unreal appear in the output window.
  • By default, the camera is at the origin, so it's probably intersecting the geometry of the environment. You can use either a Camera Mover module or a Virtual Camera module to move the camera. We recommend you use the virtual camera, because it automatically handles the billboards as well. To set up the camera movement:
    • Drag the virtual camera compound ([Common_Studio]:Compounds\VirtualCam_Unreal\VirtualCam_Unreal_A-B_Preview_3Cam.xcomp) into the Flow graph.
    • Connect the following pins (see screenshot below):
      • Unreal Project module: Out → VirtualCam: Rendered
      • VirtualCam: Cam Transform → Unreal Project module: Cam Transform
      • VirtualCam: Cam FOV → Unreal Project module: Cam FOV
      • VirtualCam: Preview → Compound output

howtode image1.jpg

    • Now you can move the camera around by holding down the mouse buttons and dragging the mouse in the preview window.
  • If you want to use a billboard:
    • Add an input. This can be a video (for example [Common]:Videos\Green\Green_BladMan.mp4) or a camera input.
    • Connect the following pins (see screenshot below):
      • Source output (e.g. Video Player: Out) → VirtualCam: Test Input 1
      • VirtualCam: B1 Active → Unreal Project module: B1 Active
      • VirtualCam: B1 Transform → Unreal Project module: B1 Transform
      • VirtualCam: B1 Texture → Unreal Project module: B1 Texture
      • VirtualCam: B1 AO Transform → Unreal Project module: B1 AO Transform
      • VirtualCam: B1 AO Texture → Unreal Project module: B1 AO Texture

howtode image3.jpg

    • Go to the Billboards window and make sure the cropping, keying etc. is set up correctly.
    • You can add a second billboard the same way.

Connection with Aximmetry

Passing data from Aximmetry to Unreal

You can pass arbitrary data from Aximmetry to Unreal. This can be used to move objects, set light intensity etc.

To pass a value, place a Get Aximmetry TYPE node in a blueprint. TYPE can be any of the supported types: Color, Integer, Logical, Scalar, Text, Transformation, Trigger, Vector and Video. The node will show up in Aximmetry as a pin on the Unreal Project module after you update the pin list. The pin list can be updated by clicking the red chain link button on the Unreal Project module.

The pin name and default value is determined by the Name and Default Value parameters of the Get Aximmetry node. The node also has an Order Index parameter which determines the pin order in Aximmetry. The pins are ordered in increasing Order Index order, but the value 0 is an exception, pins with this index will go to the end of the list. Both Name, Default Value and Order Index must be static (i.e. you cannot use the outputs of other nodes as their input).

The output of these nodes is updated per frame.

The Aximmetry_Camera blueprint uses this mechanism to get values from Aximmetry, so you can find many usage examples in that blueprint.

Most nodes are straightforward to use but there are some trickier ones.

Get Aximmetry Trigger

  • Create the following nodes: Get Aximmetry Trigger, Event BeginPlay, Bind Event to Trigger, Custom Event. (If you can't find the Bind Event to Trigger node, turn off Context Sensitive search in the right click menu.)
  • Make the following connections (see screenshot below):
    • Event BeginPlay: Exec (white triangle) → Bind Event to Trigger: Exec
    • Get Aximmetry Trigger: Return Value → Bind Event to Trigger: Target
    • Custom Event (red square) → Bind Event to Trigger: Event

howtode image2.jpg

  • Use the Exec output of the Custom Event node to execute the desired action.

Get Aximmetry Video

  • The following example applies a video image received from Aximmetry onto a plane in Unreal.
  • Create a base material (This material will serve as a base for the dynamic material we will apply to the plane. For more information on dynamic materials refer to the Unreal Engine documentation.
    • Create a new material (Content Browser / Add New / Material)
    • Open this material in the material editor by double clicking it in the content browser.
    • Add a Texture / TextureSampleParameter2D parameter from the right-click menu.The name of this parameter can be anything you want, but it must exactly match the “Texture Parameter Name” input of the “Get Aximmetry Video” blueprint node we will add later.
    • Connect the “RGB” output of the texture parameter to the “Base Color” output of the material
    • You can customize this material further to achieve the effect you want but this topic is outside the scope of this document
    • Save the material

howtode image5.jpg

  • Create a reference to your target object (e.g. a TV screen) by dragging it into the blueprint from the World Outliner (Actor Reference)
  • Add the following nodes to a blueprint:
    • Event BeginPlay
    • Create Dynamic Material Instance
    • Set Material
    • Event Tick
    • Get Aximmetry Video
    • Set Texture Parameter Value
  • Connect the following pins (see screenshot below):
    • Event BeginPlay: Exec (out) → Create Dynamic Material Instance: Exec (in)
    • Create Dynamic Material Instance: Exec (out) → Set Material: Exec (in)
    • Create Dynamic Material Instance: Return Value → Set Material: Material
    • Actor Reference (out) → Set Material: Target (This will automatically add a Get Static Mesh Component node.)
    • Event Tick: Exec (out) → Get Aximmetry Video: Exec (in)
    • Get Aximmetry Video: Exec (out) → Set Texture Parameter Value: Exec (in)
    • Get Aximmetry Video: Return Value → Set Texture Parameter Value: Value
    • Create Dynamic Material Instance: Return Value → Set Texture Parameter Value: Targe
  • Set up the node inputs:
    • Create Dynamic Material Instance
      • Parent: Select the parent material you created in the first step
    • Get Aximmetry Video
      • Name: This will be the name of the video input pin in Aximmetry
    • Set Texture Parameter Value:
      • Parameter Name: This must exactly match the name of the TextureSampleParameter2D parameter of the parent material you created in the first step.

howtode image4.jpg

Get Aximmetry Transformation

This node will show up as a transformation input pin on the Unreal Project module in Aximmetry. You can connect any transformation you want. However if you want to can control this transformation from Aximmetry in Edit mode, you can use a Scene Node:

  • Insert a Scene Node in your Aximmetry compound
  • Connect its World Transf output to the Unreal Project module (it is hidden by default - click the button in the upper right corner of the node to reveal it)
  • Enable Edit mode by choosing an output in the Edit / Edit Scene On menu (make sure the Preview output of the virtual cam module is connected to this output)
  • Select the Scene Node in the compound
  • You can move / rotate / scale the transform by using the handles in the output viewport

Passing data from Unreal to Aximmetry

This feature is currently limited to a single video output. It will be expanded in the future to cover all data types and arbitrary number of outputs.

The Set Aximmetry Video blueprint node can be used to send a texture to Aximmetry.

NOTE: Currently only one Set Aximmetry Video node is supported. If you place multiple, the system picks an arbitrary one and ignores the rest.

If there are no Set Aximmetry Video nodes in the scene, the default output of the Unreal Project module is the scene as it would be rendered to the screen by Unreal. It works like a hidden Set Aximmetry Video node with the name “out”. This default output has 3 possible formats that can be configured in Edit / Project Settings / Rendering / Default Settings / Frame Buffer Pixel Format.

Aximmetry Virtual Camera details

Setting up a virtual camera and billboard rendering in Unreal is a very complex process, so we built the Aximmetry_VirtualCam_3-Cam package to do most of the work.

It provides the following features:

  • A virtual camera: position, orientation, field of view (FOV) controlled by Aximmetry
  • Billboard rendering
    • Maximum number of billboards: 3
    • Optional shadow
    • Optionally affected by lights in the scene
    • Optionally affected by the tonemapper in Unreal
    • Enhanced image quality and sharpness
  • Based on Cine Camera: you can add post process effects etc.

The default settings should look pretty good for most use cases but there are some parameters you can tweak if it doesn't meet your needs:

  • Lit: Whether the lights in the Unreal scene affect the billboard. Disabled by default.
  • Cast Shadow: Whether the billboard casts a shadow. Enabled by default.
  • TAA Threshold: An alpha value above which TAA is disabled. We suggest setting it to 0.9. The valid range is 0-1 (inclusive).
  • Inverse Tonemap: The tonemapper in Unreal can distort the colors of the billboard significantly. This option is used to counter the effect of the tonemapper on the billboard. Enabled by default. NOTE: Using this feature with heavily processed billboard textures may lead to unexpected results (even if the texture looks fine in Aximmetry). For example: increasing the brightness significantly with an Adjuster module. This is because this feature expects the pixel values to be in a specific range – values outside this range are clipped and the texture starts to lose detail.

Gamma correction for video inputs

The Get Aximmetry Video blueprint node has a pin called S RGB that controls whether the received video is treated as sRGB or linear.

sRGB should be enabled for textures that contain color information (camera footage etc. that will be rendered into the scene) and it should be disabled for textures used for numerical calculations in shaders (normal maps, masks etc.) NOTE: This option only affects 8-bit textures. Unreal treats all non-8-bit and floating point textures as linear.

The Set Aximmetry Video itself doesn’t do any gamma correction. Its output depends on the settings and pixel format of the texture or render target you connect to it. For more information on this, please refer to the Unreal Engine documentation.

The default output is always treated as sRGB regardless of pixel format.

Interactive editing

Aximmetry can connect to Unreal in two different ways, each one is useful in a different scenario:

  • Embedded mode: Aximmetry directly loads the content generated by Unreal, so running a separate Unreal Editor is not needed. This mode should be used in production because it provides the highest performance. This is what you used so far.
  • Live Link mode: Aximmetry connects to a standalone Unreal Editor instance. This is useful while you are building your scene in Unreal, because it lets you interactively edit and test the scene, without having to cook it after every change.

In Embedded mode, Aximmetry uses the cooked version of the project which is generated when you press File / Cook Content for Windows in Unreal. This means that if you edit the scene in Unreal, you have to cook the project again. You can leave Aximmetry running with the Unreal Project module in Embedded mode - it will pick up the changes when cooking is finished and reload the project automatically. If you added / removed / changed any Aximmetry nodes in the Unreal blueprint as described in section 4.1, after cooking the project, Aximmetry will detect the change and a red chain link button will show up in the upper right corner of the module. Click on this button to update the pin list.

In Live Link mode, Aximmetry connects to the running Unreal Editor when you press the Play button. This is when the Unreal Project module will pick up the changes in the pin list as described above. Click on the red chain link button to update the pin list.

By default, the Unreal Editor runs in a reduced performance mode when it is not the active application which causes a significant decrease in framerate in Live Link mode. To disable this behavior, untick the "Use Less CPU when in Background" option in Edit / Editor Preferences / Performance / Editor Performance. If you run the scene in a separate editor window (as described below), it will still run in a reduced performance mode when it is minimized, so do not minimize this window.

To use Live Link:

  1. Open the project in Unreal and press Play
  2. Drag the .uproject file into an Aximmetry compound and set the Connection pin to Live Async
  3. Update the pin list by clicking the red chain link button
  4. Add a virtual camera (or a camera mover) and connect the pins as described above
  5. Now you should see the same image both in Aximmetry and in the Unreal Editor

You can stop and start both the Unreal scene and the Aximmetry compound or even the Unreal Editor and Aximmetry itself, the connection will be re-established automatically.

By default, the game viewport in Unreal is quite small. This results in a low resolution preview in Aximmetry.

To get a high resolution preview:

  1. Open the dropdown on the right of the Play button
  2. Select New Editor Window (PIE)
  3. Maximize the window

To automatically open a high resolution window:

  1. Go to Editor Preferences / Level Editor / Play / Game Viewport Settings
  2. Set New Window Size to your display resolution

Other considerations

Setting up reflections in Unreal Engine

There are 3 types of reflection in Unreal Engine:

1. Screen Space Reflections:

By default, the project uses screen space reflections if ray tracing in general is not enabled.
This is a very cheap effect in terms of performance however it produces very low quality results with severe rendering artifacts. We recommend using planar or ray tracing reflections instead.
There are some limitations when using screen space reflections:

  • Reflection textures provided by Aximmetry are ignored (The Mirror Offset, Mirror Feet Blur and Feet Blur Offset settings have no effect.)
  • If ray tracing is enabled: ray tracing ambient occlusion might cause a rendering error in the reflection if ray tracing shadows are enabled (the camera is in Ray Traced Shadows And Rasterized Reflections rendering mode). In this case, ray tracing ambient occlusion should be disabled:
    • Add a Post Process Volume. Set the following settings in the Details panel:
    • Check Post Process Volume Settings / Infinite Extent (Unbound)
    • Unheck Post Process Volume Settings / Rendering Features / Ray Tracing Ambient Occlusion / Enabled

2. Planar Reflections:

Planar reflections provide high quality results with a moderate performance impact. In general, we recommend using this type of reflection.
How to set up planar reflections is described in the Unreal Engine documentation.

3. Ray Tracing Reflections:

By default, the project uses ray tracing reflections if ray tracing in general is enabled.
Ray tracing reflections provide very high quality reflections but the performance impact is also high.
How to set up ray tracing reflections is described in the Unreal Engine documentation.

Note: If ray tracing is enabled in the project the camera has to be configred accordingly. See Ray Tracing in Unreal Engine.
If ray tracing is not enabled, there is no need for additional camera configuration.

Depth of field (DOF)

The billboard is not affected by the depth of field effect due to limitations in the Unreal Engine.

Translucent objects - like the billboard - are rendered after the DOF blur is applied to the scene, so these objects are not blurred. There is an option to render translucent objects before the DOF is applied. This blurs the billboard, however it leads to severe rendering artifacts (especially at the edges) because the depth information written by the billboard is not accurate enough for a good quality DOF calculation. For this reason, applying DOF to the billboard is not supported.

Ray Tracing in Unreal Engine

The Unreal Engine supports ray tracing. It consists of multiple features that can be turned on or off individually. See more information.

Most of these features can be used with Aximmetry DE but there are some limitations due to the experimental nature of ray tracing in the Unreal Engine.

Please be aware that all ray tracing features have a high performance cost, even on modern hardware. We suggest that you disable all ray tracing features then enable the ones that benefit your scene the most. How to configure these settings is described here.

The camera supports most ray tracing features but there are some caveats due to limitations in the Unreal Engine:

  • Ray Tracing Ambient Occlusion and Ray Tracing Global Illumination is fully supported.
  • Ray Tracing Reflections and Ray Tracing Shadows are supported, but only one of them can be enabled at a time.
  • Ray Tracing Translucency is not supported.

Currently the camera can operate in 3 rendering modes:

  1. Rasterized Shadows And Rasterized Reflections
  2. Ray Traced Shadows And Rasterized Reflections
  3. Rasterized Shadows And Ray Traced Reflections

The mode can be selected in the Details panel of the camera using the Aximmetry / Render Mode drop down menu. The scene must be set up according to the selected mode to avoid rendering errors.

Rasterized Shadows And Rasterized Reflections

This is the default mode. The camera works the same way as before the introduction of ray tracing features. We recommend using this mode because it can produce high quality results with good performance.

Setup (only required if Ray Tracing is enabled in the project):

  • Disable ray traced shadows for all the lights in the scene by unchecking Details / Light / Cast Ray Tracing Shadows
  • Add a Post Process Volume. Set the following settings in the Details panel:
    • Check Post Process Volume Settings / Infinite Extent (Unbound)
    • Check Rendering Features / Reflections / Type and set it to Screen Space (Note: Despite the name, Planar Reflections still work and should be used instead of Screen Space Reflections)
  • Select Rasterized Shadows And Rasterized Reflections in the Details panel of the camera using the Aximmetry / Render Mode drop down menu.

Ray Traced Shadows And Rasterized Reflections

Setup:

  • Enable Ray Tracing in the project as described here
  • Enable ray traced shadows for all the lights in the scene by checking Details / Light / Cast Ray Tracing Shadows. (This should be enabled by default.)
  • Add a Post Process Volume. Set the following settings in the Details panel:
    • Check Post Process Volume Settings / Infinite Extent (Unbound)
    • Check Rendering Features / Reflections / Type and set it to Screen Space (Note: Despite the name, Planar Reflections still work and should be used instead of Screen Space Reflections)
  • Select Ray Traced Shadows And Rasterized Reflections in the Details panel of the camera using the Aximmetry / Render Mode drop down menu.

Rasterized Shadows And Ray Traced Reflections

Setup:

  • Enable Ray Tracing in the project as described here
  • Disable ray traced shadows for all the lights in the scene by unchecking Details / Light / Cast Ray Tracing Shadows
  • Add a Post Process Volume. Set the following settings in the Details panel:
    • Check Post Process Volume Settings / Infinite Extent (Unbound)
    • Check Rendering Features / Ray Tracing Reflections / Include Translucent Objects and check the box next to it
  • Optional: Outdoor scenes where the talent's reflection can be seen in front of the sky require some additional setup:
    • Select the Sky Sphere
    • Select the SkySphereMesh component in the Details panel
    • Check Rendering / Ray Tracing / Visible in Ray Tracing
  • Select Rasterized Shadows And Ray Traced Reflections in the Details panel of the camera using the Aximmetry / Render Mode drop down menu.

Limitations

Currently some features cannot be fully supported due to limitations in the Unreal Engine:

  • Ray traced reflections and ray traced shadows cannot be used at the same time.
  • Setting the Mirror Offset of the billboard in Aximmetry to non-zero values produce rendering errors around the talent in certain cases when using ray traced reflections.
  • Brightness of the reflection of unlit billboards can be different from the actual billboard when using ray traced reflections (depending on the scene exposure).
  • Lighting does not affect the reflection of lit billboards when using ray traced reflections.
  • Rasterized shadows are not visible in ray traced reflections.
  • Ray traced shadows are not visible in rasterized reflections.
  • Ray traced translucency is not supported. (Do not set Details / Rendering Features / Translucency / Type to Ray Tracing on the Post Process Volume.)
Article content

Close
1/10