Back to Keying
Introduction
One of Aximmetry's many functions is to merge (to composite) a virtual environment with a real-time - or pre-recorded video feed. This is a very fundamental and important part of Virtual Production.
We use Billboards to achieve this composition.
What is a Billboard
The Billboard is a virtual object, a rectangle, that is used to make the camera image interact with the virtual environment. The input and the Billboard are in a tight connection. We use a billboard to display the input's content and place it in the virtual environment.
The only exception is if we display the content of an input on a virtual screen. In this case, we don't need a billboard. We have a prepared solution for this in the case of Aximmetry SE, and here you can find information on how you can display input on a virtual screen created with Unreal Engine.
You will need a Billboard for every studio camera you want to use in your project.
NOTE: Billboards are used in both Aximmetry SE and DE. In the case of an Unreal Scene, you can use a maximum of 3 inputs at a time (a maximum of 3 Billboards can be displayed) per virtual camera (because of Unreal limitations). From Aximmetry's perspective, there is no limit on the number of Billboards you can work with. However, most hardware configurations are not able to handle more than 8 billboards at a time.
Billboards in Virtual Camera Compounds with Aximmetry SE (Aximmetry Rendered)
All virtual camera compounds include the Billboards feature.
More on virtual camera compounds here -
Billboards are automatically added when you add a virtual camera compound to your project.
You can find the Billboard # panel in the BILLBOARDS Control Board.
The pin values panel will show the pins of the billboard # panel when the panel is selected. (Selection is indicated by the blue border on the panel).
Each line represents the workflow of an input from a studio camera and a corresponding Billboard with all its settings. In this example, we are using a compound with 3 cameras.
There are as many Billboards as inputs.
Types of Inputs
Live Input
If both of the buttons on your INPUT panel are turned off, then your current INPUT is offline, so you won't see the Billboard in the scene, because there is nothing to be displayed on the Billboard.
In order to show content on the Billboard, first, you must activate its INPUT.
To activate the INPUT click the button with the power icon on it.
If the input is turned ON, but the input device is not defined, you will see a red rectangle on the Billboard. with the message: "Device not found":
It is a good practice to turn on as many inputs as you are actually using and not more.
Make sure that the Test button is turned off if you want to use a live input (for example a camera feed).
More on using the Test input later.
Select the input device in the Pin Values of the INPUT# panel. Its Pin Values can be seen only if the INPUT# panel is selected
NOTE: If you have already mapped the input device to ID #1, then there is no need to change this.
Pre-recorded Input
You can also use a pre-recorded video (or image, image sequence, etc. more info here) as input. Place the file in the Flow Editor next to the Virtual Cam compound.
Connect the video player's Out pin to the Test Input pin of the Virtual Camera compound. Each Test Input connects to a Billboard with the same number.
Then switch the Test button "On".
Now you can see the video playing on the Billboard.
Billboard Visibility in Different Cameras
Using the visibility panel you can turn on and off the visibility of the Billboard for virtual cameras 1, 2, 3, etc...
In this example, we are using the VirtualCam_3-Cam compound, which contains 3 Cameras and 3 Billboards, so each VISIBILITY panel contains 3 buttons. Each button represents a virtual camera.
NOTE: Remember, that at the beginning, we discussed that each line on the BILLBOARDS control board represents the workflow on a billboard/input pair, so the VISIBILITY 1 panel will toggle the visibility of BILLBOARD 1.
Example 1:
We set both VISIBILITY 1 and 2 (Billboard 1 and Billboard 2) to be seen in CAM 1.
Now you can see the two Billboards in CAM 1, and only in CAM 1.
Example 2:
We set both VISIBILITY 1 and 2 (Billboard 1 and Billboard 2) to be seen in CAM 2.
Now if we see CAM 1, we will see Billboard 1 only.
But if we switch to CAM 2 we will see Billboard 1, and Billboard 2.
Placing the Billboard
We recommend starting with this step. Placing the Billboard in the scene is the most basic and one of the most important steps.
Select the Billboard panel, and find the Transformation pins in the Pin Values.
Transformation
By default, the Billboard is placed at the origin of the scene.
By changing the values of Pos (Position) you can move and place it in the scene.
Vertical Position
The Y value (the middle one) should be set just a bit higher than the virtual floor's Y position.
NOTE: The height of the virtual floor is a piece of information that the graphic designer should have. You can visually set the Y position by adjusting the height after turning on the AO to the point when the AO is visible, but not any further.
Let's suppose that the virtual floor's Y position is 0.168. This means that we should set the Billboard's Y position to at least 0.169.
In the image below, the Y coordinate of the Transformation of the Billboard is properly set. You can see that the AO below the talent is visible. But as you can also see, it looks as if the talent was floating. At this point, this is completely normal, we will tend to this in the next part.
This is very important, for two reasons:
- if it is not set properly, the AO will not be visible (it will be under the floor),
- the reflection of the talent will not be realistic.
Rotation
When you place a Billboard, by default, it will always automatically rotate toward the camera. You can change this setting with the Look at Camera feature. You can find more information on this feature here.
You can use the Rot (Rotation) values to rotate the Billboard if necessary, but this will only work if you turn off the look-at-camera feature.
Maybe you will find it easier to place the Billboard using the Transformation gizmo feature.
Lift
After placing the Billboard and setting its orientation, we recommend continuing with this step.
Usually, on the input, there is some distance between the talent's feet and the bottom of the image:
So if you fit the bottom of the Billboard to the virtual surface, the talent looks as if he was floating in the air:
The further the position of the virtual camera is from the position of the studio camera the input is taken with, the more 'floating' will be caused.
If you would simply lower the Billboard to fit the talent to the virtual ground, the AO will disappear.
The Lift feature can help to solve this problem.
This offsets the Billboard's bottom so you can fit your talent to the virtual floor and keep the AO as well.
Lowering the value will push the Billboard down:
This is better, but raising the virtual camera a bit will make this look even better:
NOTE: It is always recommended to keep the virtual camera's perspective relatively close to the studio camera's perspective.
Scal - Scale
Sometimes the input does not appear realistic in size compared to the virtual environment.
You can use the Scale feature to match the image on the Billboard to the real size of the talent.
As you can see, the talent appears way smaller than it should:
Here, the billboard scale has been set properly:
Setting the scale properly is very easy using the size grid feature.
Size Grid on Billboards
Click on the lightbulb icon on the Billboard panel to show the size grid on the Billboard.
1 square on the grid is 50x50 cm by default.
Let's suppose the height of the talent, in reality, is about 2m, so using the grid we can scale the billboard accordingly. You can see, according to the grid, the talent appears more than 2m tall, but this is due to the fact that we have used the Lift feature before, introducing some offset at the bottom of the billboard, which is added to the height.
This feature also helps your workflow by showing the Billboard's size, orientation, placement, etc.
Again, you can also place, rotate, and scale the Billboard visually, using the Transformation Gizmos.
Reflections
In order to achieve the best results for your virtual production, you must have reflections. The proper setup of reflections requires a few extra steps.
Please follow the link below to gain more information about how to set up these reflections.
Mirror Offset
Look at Camera
With realistic virtual camera movements, you can achieve the best result by using the default setting: At Camera. This will automatically rotate the Billboard toward the virtual camera, while the camera is moving. This is necessary to keep the proportions of the content on the Billboard if the virtual camera is moved.
It has 3 values.
At Camera
Keeps the Billboard turned towards the camera when the camera is moving.
Useful for parallax and longer camera movements.
Don't forget that the Billboard is only 2 dimensional, so it is flat. If you move your camera to some extreme position, the image on the Billboard will look distorted.
At Camera Plane
Keeps the Billboard parallel with the camera. Useful for dolly movements. It will look more natural.
Off
With this option selected the Billboard won't rotate at all.
Flip
Mirrors the billboard along its vertical axis.
Shadows
There is a dedicated document about Shadows.
Transparency
The default value of Transparency is Normal (1).
Don't change it.
Highlight Cover
Bloom and flare effects from bright background light sources can be either shown on (Off) or covered by (On) the billboard.
AO
AO, or Ambient Occlusion, is the virtually generated contact shadow visible under the talent, on the virtual floor.
You can find information about AO in the dedicated document on shadows: Ambient Occlusion
Cut
You can adjust the size of the Input inside the Billboard with the Cut sliders.
These sliders simply cut off the input inside the Billboard, so the size of the Billboard will not change.
For example, if you discover some anomaly close to the edge of the Input after properly placing the Billboard in your scene, you can cut off the sides of the input, while not changing the size or position of the Billboard.
These cut sliders are different from the Crop panel's cut sliders. The Crop panel's function is to cut and offset the Billboard keeping its center in place.
We recommend first using the Crop panel and using the Billboard's Cut feature only if necessary.
Force Distance and Force Distance AO
Objects behind transparent objects will be rendered based on the center of the object's distance from the camera. First, the farthest object will be rendered then the one closer to the camera, etc...
If we place a Billboard behind a transparent object's center, it might be rendered before the transparent object. This will make the Billboard appear as if it was behind it.
Force Distance
With this, you can force the center of the Billboard to a distance (closer to the camera) so it will be rendered properly.
NOTE: Keep in mind the distance of the transparent object.
Force Distance AO
Does the same for the AO.
Meter Opacity
Adjust the opacity of the size grid.
Billboards in Virtual Camera Compounds with Aximmetry DE (Unreal Engine Rendered)
A lot of the above-mentioned functions work the same with Unreal Engine-rendered scenes, so here we will only highlight the differences and the additional features of Unreal-rendered Billboards.
In Aximmetry DE, you can choose if the Billboard is rendered by Aximmetry or by Unreal Engine. Both have benefits and disadvantages.
Select the CAMERA & RENDER SETUP Panel in the CAMERAS Control Board.
You will find the Allow Virtuals switch in its Pin Values.
Allow Virtuals: OFF
Aximmetry renders the image of the talent on top of the Unreal scene. Virtual objects can occlude the talent, reflections, and shadows are supported. The camera image is accurately reproduced without adding noise or distortions.
Allow Virtuals: ON
Compositing is done entirely in Unreal. Virtual objects can occlude the talent, reflections, and shadows are supported. With this method, the talent can be lit by the lights in the Unreal scene. Virtual camera movement is also supported. The quality of the talent’s image may be slightly degraded.
Lit
Navigate back to the BILLBOARDS Control Board and select the Billboard you want to work on.
You can only turn this feature ON if Allow Virtuals is ON.
If ON, the lights of the Unreal Scene will affect the Billboard.
In this example, we placed a dozen of strong green spotlights in the Unreal scene. When the Lit is Off the Billboard is not affected by the lights of the Unreal scene:
But when the Lit switched to ON the Billboard is affected by the spotlights and turned to green:
Render To Depth
Render To Depth is enabled by default.
This should be turned On in case you are using non-planar reflections in your scene, for example, screen-space reflections or Lumen on which the Billboard's content might be reflected.
This feature, if turned on, will include the billboard in the render depth information. We generally suggest using planar reflections on surfaces where the Billboard content is reflected.
Inverse Tonemap
The tone mapper (post-process) in Unreal can distort the colors of the billboard significantly. This option is used to counter the effect of the tone mapper on the billboard.
You only need to turn this On if Allow Virtuals is also On.
NOTE: Using this feature with heavily processed billboard textures may lead to unexpected results (even if the texture looks fine in Aximmetry). For example: increasing the brightness significantly with an Adjuster module. This is because this feature expects the pixel values to be in a specific range – values outside this range are clipped and the texture starts to lose detail.
Alpha Correction for Bright Background
Very bright backgrounds can make the billboard too transparent. This option is designed to remedy this issue. It is disabled by default.
NOTE: The alpha correction may not handle certain cases correctly. Elements like DOF (Depth of Field) blur and translucent objects behind the billboard can cause rendering errors.
Continue to the Cameras Control Board
Once you finished setting up the Billboard, you can move on to the Cameras Control Board to set up the virtual cameras:
