Back to Keying
Introduction
One of Aximmetry's many functions is to merge (to composite) a virtual environment with a real-time or pre-recorded video feed. This is a fundamental part of Virtual Production.
In Aximmetry, we use Billboards to achieve this compositing.
What is a Billboard
The Billboard is a virtual object, a rectangle, that is used to make the camera image interact with the virtual environment. The input and the Billboard are in a tight connection. We use a billboard to display the input's content and place it in the virtual environment.
NOTE: Example of a Billboard location in the stock virtual studio scene ([Studio]:News Room\News Room - VirtualCam_3-Cam.xcomp).
The only exception is if we display the content of an input on a virtual screen. In this case, we don't need a billboard. We have a prepared solution for this in the case of Aximmetry's native render engine, and here you can find information on how you can display input on a virtual screen created with Unreal Engine.
You will need a Billboard for every studio camera you want to use in your project.
NOTE: Billboards are used in both Aximmetry's native render engine and the Unreal Engine workflows. In the case of an Unreal Scene, you can use a maximum of 3 inputs at a time (a maximum of 3 Billboards can be displayed) per virtual camera (because of Unreal limitations). From Aximmetry's perspective, there is no limit on the number of Billboards you can work with. However, most hardware configurations are not able to handle more than 8 billboards at a time.
Billboards in Virtual Camera Compounds with Aximmetry's Native Render Engine
All virtual camera compounds include the Billboards feature.
More on virtual camera compounds here - Virtual Camera Compounds
Billboards are automatically added when you add a virtual camera compound to your project.
You can find the Billboard # panel in the BILLBOARDS Control Board:
The Pin Values panel will show the pins of the billboard # panel when the panel is selected.
NOTE: Selection is indicated by the blue border on the panel:
Each line represents the workflow of an input from a studio camera and a corresponding Billboard with all its settings. In this example, we are using a compound with 3 cameras:
There are as many Billboards as inputs.
Placing the Billboard
We recommend starting with this step. Placing the Billboard in the scene is both the most basic and one of the most important steps.
Put In Front
By default, the Billboard is placed at the origin of the scene.
Select the Billboard panel and find the Put In Front trigger in the Pin Values panel.
After pressing Put In Front, the Billboard will move in front of the currently used camera. This automatically overwrites the Transformation value in the Pin Values panel.
Put On Floor
This trigger automatically detects the nearest virtual floor or model beneath the current location of the Billboard. It then overwrites the height value (the second field in "Pos") in the Transformation value.
NOTE: The detection process begins from a point 1–2 meters above the Billboard's position (bottom of the Billboard). Ensure that there are no objects at this location, as nearby 3D models or effects can interfere with accurate detection.
Transformation
You can change and adjust the Billboard's position in the scene by changing the values of Pos (Position).
NOTE: The Pos (Position) parameters adjust the following axes (from left to right): X (left-right), Y (up-down), and Z (back-forth). The values are measured in meters.
NOTE: The Rot (Rotation) parameters adjust the following axes (from left to right): rX, rY, and rZ. Each parameter controls the rotation around its corresponding axis, measured in degrees.
NOTE: The Scal (Scale) parameter adjusts the size of the Billboard along all axes. It acts as a multiplier of the original values.
NOTE: You can finely modify the values in the Pin Values panel by using the mouse and shortcuts. For detailed instructions, follow the Modifying Number Values section of the Flow Editor documentation.
Vertical Position
The Y value (the middle one) should be set just a bit higher than the virtual floor's Y position. Use the Put On Floor trigger to automatically set the vertical position.
NOTE: If the Put On Floor trigger is not an option, the height of the virtual floor is a piece of information that the graphic designer could have. You could also visually set the Y position by adjusting the height after turning on the AO to the point when the AO is visible, but not any further.
Let's suppose that the virtual floor's Y position is 0.168. This means that we should set the Billboard's Y position to at least 0.169.
In the image below, the Y coordinate of the Transformation of the Billboard is properly set. You can see that the AO below the talent is visible. But as you can also see, it looks as if the talent was floating. At this point, this is completely normal. We will tend to this in the next part.
This is very important for two reasons:
- If it is not set properly, the AO will not be visible (it will be under the floor).
- The reflection of the talent will not be realistic.
Rotation
When you place a Billboard, by default, it will always automatically rotate toward the camera. You can change this setting with the Look at Camera feature. You can find more information on this feature here.
You can use the Rot (Rotation) values to rotate the Billboard if necessary, but this will only work if you turn off the look-at-camera feature.
Maybe you will find it easier to place the Billboard using the Transformation gizmo feature.
Lift
After placing the Billboard and setting its orientation, we recommend continuing with this step.
Usually, on the input, there is some distance between the talent's feet and the bottom of the image:
So if you fit the bottom of the Billboard to the virtual surface, the talent looks as if he were floating in the air:
The further the position of the virtual camera is from the position of the studio camera, the input is taken with, the more 'floating' will be caused.
If you would simply lower the Billboard to fit the talent to the virtual ground, the AO will disappear.
The Lift feature can help to solve this problem.
This offsets the Billboard's bottom so you can fit your talent to the virtual floor and keep the AO as well.
Lowering the value will push the Billboard down:
This is better, but raising the virtual camera a bit will make this look even better:
NOTE: It is always recommended to keep the virtual camera's perspective relatively close to the studio camera's perspective.
Scal - Scale
Sometimes, the input does not appear realistic in size compared to the virtual environment.
You can use the Scale feature to match the image on the Billboard to the real size of the talent.
As you can see, the talent appears way smaller than it should:
Here, the billboard scale has been set properly:
Setting the scale properly is very easy using the size grid feature.
Size Grid on Billboards
Click on the lightbulb icon on the Billboard panel to show the size grid on the Billboard.
1 square on the grid is 50x50 cm by default.
Let's suppose the height of the talent, in reality, is about 2m, so using the grid, we can scale the billboard accordingly. You can see, according to the grid, the talent appears more than 2m tall, but this is due to the fact that we have used the Lift feature before, introducing some offset at the bottom of the billboard, which is added to the height.
This feature also helps your workflow by showing the Billboard's size, orientation, placement, etc.
Again, you can also place, rotate, and scale the Billboard visually, using the Transformation Gizmos.
Mirror (Reflections)
In order to achieve the best results for your virtual production, you must have reflections. The proper setup of reflections requires a few extra steps.
Please follow the link below to gain more information about how to set up these reflections.
Mirror Offset
Look at Camera
With realistic virtual camera movements, you can achieve the best result by using the default setting: At Camera. This will automatically rotate the Billboard toward the virtual camera while the camera is moving. This is necessary to keep the proportions of the content on the Billboard if the virtual camera is moved.
It has 3 values.
At Camera
Keeps the Billboard turned towards the camera when the camera is moving.
Useful for parallax and longer camera movements.
Don't forget that the Billboard is only 2-dimensional, so it is flat. If you move your camera to some extreme position, the image on the Billboard will look distorted.
At Camera Plane
Keeps the Billboard parallel with the camera. Useful for dolly movements. It will look more natural.
Off
With this option selected, the Billboard won't rotate at all.
Flip
Mirrors the billboard along its vertical axis.
Shadows
There is a dedicated document about Shadows.
Transparency
The default value of the Billboard Transparency is Normal (1).
Don't change it.
Highlight Cover
Bloom and flare effects from bright background light sources can be either shown on (Off) or covered by (On) the billboard.
AO
AO, or Ambient Occlusion, is the virtually generated contact shadow visible under the talent, on the virtual floor.
You can find information about AO in the dedicated document on shadows: Ambient Occlusion
Cut
You can adjust the size of the Input inside the Billboard with the Cut sliders.
These sliders simply cut off the input inside the Billboard, so the size of the Billboard will not change.
For example, if you discover some anomaly close to the edge of the Input after properly placing the Billboard in your scene, you can cut off the sides of the input, while not changing the size or position of the Billboard.
These cut sliders are different from the Crop panel's cut sliders. The Crop panel's function is to cut and offset the Billboard, keeping its center in place.
We recommend first using the Crop panel and using the Billboard's Cut feature only if necessary.
Force Distance and Force Distance AO
Objects behind transparent objects are rendered based on the distance from the camera to the center of each object. The rendering process starts with the farthest object and then moves to those closer to the camera.
If a Billboard is placed behind the center point of a transparent object, the Billboard may be rendered before the transparent object. As a result, the Billboard will appear as if it is behind the transparent object.
Force Distance
With this, you can force the center of the Billboard to a distance (closer to the camera) so it will be rendered properly.
NOTE: Keep in mind the distance of the transparent object.
Force Distance AO
This functions the same as Force Distance but applies to the AO (Ambient Occlusion) instead of the Billboard itself.
Meter Opacity
Adjust the opacity of the size grid.
Edge Fade
Fades the edges of the entire billboard. By default, Edge Fade is turned off, as Edge Fade Sides is set to None.
This feature can be useful in various scenarios, for example, providing a smooth fade-out for the talent when exiting the area of the billboard or softening imperfections in keying that result in abrupt edges.
Edge Fade Sides
Specifies which side(s) of the billboard should be faded.
Edge Fade Amount
Controls the strength of the fade effect.
When setting up these Edge Fade parameters, it can be helpful to turn off the keyer.
For example, below the keyer is turned off, and Edge Fade Sides is set to Left:
Billboard Visibility in Different Cameras
Using the visibility panel, you can turn on and off the visibility of the Billboard for virtual cameras 1, 2, 3, etc...
In this example, we are using the VirtualCam_3-Cam compound, which contains 3 Cameras and 3 Billboards, so each VISIBILITY panel contains 3 buttons. Each button represents a virtual camera.
NOTE: Remember that at the beginning, we discussed that each line on the BILLBOARDS control board represents the workflow on a billboard/input pair, so the VISIBILITY 1 panel will toggle the visibility of BILLBOARD 1.
Example 1:
We set both VISIBILITY 1 and 2 (INPUT 1 and INPUT 2) to be seen in CAM 1.
Now you can see the two Billboards in CAM 1, and only in CAM 1.
Example 2:
We set both VISIBILITY 1 and 2 (INPUT 1 and INPUT 2) to be seen in CAM 2.
Now, if we view CAM 1, we will see only Billboard 1 (INPUT 1).
The ON SCREEN info text displays "INPUT 1" at the bottom of the image above. This is the default naming scheme for each input, which can be changed using the Name parameter in the INPUT panel.
But if we switch to CAM 2, we will see Billboard 1 (INPUT 1) and Billboard 2 (INPUT 2).
Billboards in Virtual Camera Compounds with Unreal Engine
A lot of the above-mentioned functions work the same with Unreal Engine-rendered scenes, so here we will only highlight the differences and the additional features of Unreal-rendered Billboards.
In Aximmetry DE, you can choose if the Billboard is rendered by Aximmetry or by Unreal Engine. Both have benefits and disadvantages.
Select the CAMERA & RENDER SETUP Panel in the CAMERAS Control Board. You will find the Allow Virtuals switch in its Pin Values.
Allow Virtuals: OFF
Aximmetry renders the image of the talent on top of the Unreal scene. Virtual objects can occlude the talent, and both reflections and shadows are supported.
The camera image is accurately reproduced without adding noise or distortions.
Allow Virtuals: ON
Compositing is done entirely in Unreal. Virtual objects can occlude the talent, and both reflections and shadows are supported.
With this method, the talent can be lit by the lights in the Unreal scene. Virtual camera movement is also supported. The quality of the talent’s image may be slightly degraded.
Lit
Navigate back to the BILLBOARDS Control Board and select the Billboard you want to work on.
You can only turn this feature ON if Allow Virtuals is ON.
If ON, the lights of the Unreal Scene will affect the Billboard.
In this example, we placed a dozen of strong green spotlights in the Unreal scene. When the Lit is Off, the Billboard is not affected by the lights of the Unreal scene:
But when the Lit is switched to ON, the Billboard is affected by the spotlights and turns green:
Render To Depth
Render To Depth is enabled by default.
This should be turned On in case you are using non-planar reflections in your scene, for example, screen-space reflections or Lumen on which the Billboard's content might be reflected.
This feature, if turned on, will include the billboard in the render depth information. We generally suggest using planar reflections on surfaces where the Billboard content is reflected.
Inverse Tonemap
The tone mapper (post-process) in Unreal can distort the colors of the billboard significantly. This option is used to counter the effect of the tone mapper on the billboard.
You only need to turn this On if Allow Virtuals is also On.
NOTE: Using this feature with heavily processed billboard textures may lead to unexpected results (even if the texture looks fine in Aximmetry). For example, increasing the brightness significantly with an Adjuster module. This is because this feature expects the pixel values to be in a specific range – values outside this range are clipped, and the texture starts to lose detail.
NOTE: Inverse Tonemap is currently not supported in HDR rendering.
Alpha Correction for Bright Background
Very bright backgrounds can make the billboard too transparent. This option is designed to remedy this issue. It is disabled by default.
NOTE: The alpha correction may not handle certain cases correctly. Elements like DOF (Depth of Field) blur and translucent objects behind the billboard can cause rendering errors.
Ray Traced Reflection Intensity
Increase the Ray Traced Reflection Intensity until the brightness of the reflection matches the brightness of the billboard.
Be aware that ray-traced reflections are only used by Lumen Reflections, and only when ray-traced shadows are enabled. Additionally, reflections are ray-traced only if the reflected object (in this case, the billboard) is not visible to the camera. This is because, when the object is not in the camera’s view, the render engine cannot simply copy the reflection from what is already visible; instead, it must calculate the reflection accurately using ray tracing.
NOTE: Standalone Ray Traced Reflections (available in Unreal Engine versions 4 through 5.3) are also affected by this parameter.
For more information about reflections in Unreal, refer to the Reflections paragraph in the Unreal Scene Setup (Green Screen) documentation.
Continue to the Cameras Control Board
Once you have finished setting up the Billboard, you can move on to the Cameras Control Board to set up the virtual cameras:
