Search
Start typing to search...

UE5: Using Tracked Cameras and Augmented Reality with an Unreal Scene

Author:

Introduction

This document describes how to set up camera tracking and augmented reality in Aximmetry Dual Engine (Aximmetry DE). The material assumes a basic knowledge of Aximmetry DE and how to work with tracked cameras – If you are not familiar with these, please read How to Install and Work with the Unreal Engine-Based DE edition and the Tracked Camera Workflow first.

Camera Tracking

Setting up a project with a tracked camera is basically the same process as setting it up with a virtual camera – with a few minor differences which will be described here. How to set up a project with a virtual camera is detailed in the How to Install and Work with the Unreal Engine-Based DE edition.

In Unreal Editor for Aximmetry

  • Create a new Aximmetry Blank Unreal project or open an existing Unreal project.
  • Add a tracked camera. Aximmetry / Add Camera / Tracked camera (1 - 3 billboards)

  • If you want to disable Motion Blur:
    • Go to Edit / Project Settings / Engine / Rendering / Default Settings
    • Untick Motion Blur

In Aximmetry DE

  • Drag a tracked camera compound into the flow graph from [Common_Studio]:Camera\TrackedCam_Unreal\TrackedCam_Unreal_3-Cam.xcomp
  • Connect the following pins (see screenshot below):
    • Unreal Project module: Out → TrackedCam: Rendered
    • Unreal Project module: B Mask → TrackedCam: B Mask
    • TrackedCam: Out Size → Unreal Project module: Out Size
    • TrackedCam: Control Data → Unreal Project module: Control Data
    • TrackedCam: B Texture → Unreal Project module: B Texture
    • TrackedCam: B Refl Texture → Unreal Project module: B Refl Texture
    • TrackedCam: B Shadow Texture → Unreal Project module: B Shadow Texture 
    • TrackedCam: Preview → Compound output
    • TrackedCam: Out → Compound output
  • Set up the tracked camera input. Please refer to the Tracked Camera Workflow on how to do this.

Aximmetry DE supports 3 methods for compositing the camera image with the virtual set.
The compositing method can be selected by setting the Use Billboards and Allow Virtuals switches on the SCENE node in the INPUTS panel.

Use billboards: OFF, Allow Virtuals: not used

Aximmetry renders the image of the talent on top of the Unreal scene. This method is very easy to set up and has the smallest performance requirements, however, it doesn’t support virtual objects occluding the talent or rendering shadows and reflections of the talent in the Unreal scene.

Use billboards: ON, Allow Virtuals: OFF

Aximmetry renders the image of the talent on top of the Unreal scene. Virtual objects can occlude the talent, reflections, and shadows are supported. The camera image is accurately reproduced without adding noise or distortions.

Note: If there is a translucent object in front of the billboard, the Render To Depth billboard property must be Off to avoid rendering artifacts.

Use billboards: ON, Allow Virtuals: ON

Compositing is done entirely in Unreal. Virtual objects can occlude the talent, reflections, and shadows are supported. With this method, the talent can be lit by the lights in the Unreal scene. Virtual camera movement is also supported. The quality of the talent’s image may be slightly degraded.

Augmented Reality

Aximmetry DE is capable of rendering virtual objects in the Unreal Engine and compositing them onto a tracked camera image.

Unreal setup

  • Create a new Aximmetry Blank Unreal project or open an existing Unreal project.
    If an existing Unreal project is opened:
  • Go to Edit / Project Settings / Engine / Rendering / Default Settings / Advanced and set Frame Buffer Pixel Format and set it to FloatRGBA.
  • Delete all Fog actors (i.e. Atmospheric Fog, Experimental Height Fog) from the scene.
  • Add an AR-tracked camera. Aximmetry / Add Camera / AR with tracked camera
  • Optional: Place shadow catchers (Aximmetry_Shadow_Catcher). There is no restriction on the number of shadow catchers.
  • Optional: Place reflection catchers (Aximmetry_Reflection_Catcher). The maximum number of reflection catchers is 3. Keep in mind that reflections have a very high-performance cost.
  • Place virtual objects in the scene and tag them with the “AximmetryAR” tag. Only tagged objects will show up in the final image
  • If you want to disable Motion Blur:
    • Go to Edit / Project Settings / Engine / Rendering / Default Settings
    • Untick Motion Blur

Please make sure the tag is added to the Actor / Tags array and not the Tags / Component Tags array:

Aximmetry setup

  • Create a new compound
  • Drag in the .uproject file from your Unreal project root directory
  • Select the Unreal Project module and set the Connection parameter to Cooked.
  • Drag an AR camera compound into the flow graph from

[Common_Studio]:Camera\TrackedCam_Unreal\TrackedCam_Unreal_3-Cam.xcomp

  • Connect the following pins (see screenshot below):
    • Unreal Project module: Out → TrackedCam_AR: Rendered
    • Unreal Project module: Alpha→ TrackedCam_AR: Alpha
    • Unreal Project module: Reflection 1, 2, 3 → TrackedCam_AR: Reflection 1, 2, 3
    • Unreal Project module: Shadow → TrackedCam_AR: Shadow
    • Unreal Project module: Shadow Clean Plate → TrackedCam_AR: Shadow Clean Plate
    • TrackedCam_AR: Out Size→ Unreal Project module: Out Size
    • TrackedCam_AR: Control Data→ Unreal Project module: Control Data
    • TrackedCam_AR: Preview → Compound output
    • TrackedCam_AR: Out → Compound output
  • Add tracked camera input. Please refer to the Tracked Camera Workflow documentation on how to do this.

  • In the INPUTS panel, select the SCENE panel and use the Base Cam Transf to position the virtual objects to the right place on the tracked camera image
    NOTE: You can learn more about Camera and Head Transformations here.
  • In the INPUTS panel, select the CATCHERS panel and configure the Reflection Blur, Reflection Strength, and Shadow Strength parameters to match the properties of the real surface to achieve a realistic result
Article content

Loading
Close
Loading spinner icon
1/10