| Please note that this is a BETA version of the document. All information presented is correct but we are working on improving the details. |
Back to Studio Setup Examples (Multi-Machine)
When Do You Need a Multi-Machine System?
When you need multiple independent video outputs, but producing them simultaneously is too much load for a single PC.
The machines can render completely different content or different views of the same content. For example, in a virtual studio system, each machine renders the view of a particular camera, or in a panorama view, each machine renders a slice of the image, and so on.
Architecture
There is a central controlling machine that itself can render as well, and at the same time, it can control several satellite machines. Satellite machines are called “remote renderers” or “renderers” in Aximmetry.
The controlling machine runs a normal Composer, and you can edit and control it through the usual Flow Editor or Control Boards. While the satellite machine runs a slim application named Renderer. It’s a pure render engine, does not provide any UI, and only receives rendering commands from the controlling machine.
For the controlling machine, it’s recommended to use the usual dual monitor configuration. The renderers normally only need a single monitor - of course, it can be a shared monitor using a KVM system.
Each machine must have its own input and output devices in order to produce an output. Every machine's input video and output video are independent of the other machines' video pictures. There are no video streams between machines by default other than optional monitoring.
Licenses
For the controlling machine, you need a Broadcast Edition.
For the satellite(renderer) machines, any license can be used, depending on the input interface of the camera and the camera tracking system you would like to use with the camera connected to the satellite machine. More information on licenses and interfaces here and on tracking systems here.
Setting up the Renderer Machines
Switch to one of the satellite machines.
The Renderer application cannot be started directly, we have to start the Launcher instead. The Launcher has to be running all the time. Its only task is to start, kill, or restart the Renderer upon the command of the controlling machine. This separation serves the operation safety of the system. If something happens with the Renderer, it freezes or crashes, we can handle the situation remotely through the Launcher.
Launcher
So first start the Launcher.
The Launcher’s minimalist UI appears. It's essentially a log window only. The important info here is in the header: the name and the IP (or IPs if you have multiple network cards) of the machine. Any of these can be used later when you set up the controlling machine.
Renderer Config
Though the Renderer itself cannot be started directly, we must be able to set up its inputs and outputs. For this start Renderer Config.
A Startup Configuration window appears which is mostly identical with the Composer’s one. Suppose we want an SDI output and two SDI inputs, one for a camera, and one for a playout coming from the studio system.
Output(s)
Suppose we use the standard assignments: output #1 is a preview, and output #2 is the final picture that should go to an SDI output. If you use a separate monitor for each renderer machine you can use them as preview monitors. In this case, we can set up the usual configuration:
If you do not need a preview monitor per renderer, or you use a KVM system, it’s not necessary to assign any monitors. You only need the final output on the SDI port:
Input(s)
In the Device Mapper we can set up the SDI inputs as usual:
Project Root
In order to get the multi-machine system to work properly you have to ensure that all machines work from the very same shared project folders. You have to place the projects on a single machine of your choice. It can be the controlling machine itself, or a dedicated server machine. Then you have to share the folder containing the projects and set it as the Project Root both on the controlling machine and the renderer machines. You can either use a network drive or a direct network path.
In this example the projects are located on the controller machine in the shared folder named Projects:
IMPORTANT: The libraries that come with Aximmetry, such as the Common and Common_Studio folders, must be shared and included within the Project root folders. Otherwise, for example, the camera compounds will not be loaded on the machines.
Then click Start. The setup window closes and the Renderer is ready to be started by the Launcher.
Do the above setup of Launcher and Renderer for all satellite machines.
Setting up the Controller Machine
On the controller machine, you use the regular Composer application. You can choose that you use the controller machine only for controlling or it can be one of the rendering machines as well. The latter approach can be advantageous because you can do the visual setup (e.g. the placing of the billboards) on the preview monitor in the usual way.
Start the Composer.
Input(s) and Output(s)
If you want the machine rendering, set up the inputs and outputs as usual (presumably for one camera input and one SDI output, similar to the satellite machines).
Remote Renderers
Go to the Remote Renderers section.
If you only use this machine as a controller, turn off Local rendering, otherwise leave it on.
Turn on Remote rendering. Later if you want to use this machine for both content editing and for rendering the live show you can turn on/off this switch at will. That way you won’t have to start all your satellite renderers when you only want to make a graphical change in your virtual set.
We'll describe the Manual start option later in the Operation safety section, now leave it off.
Then list your renderer machines either by their names (if your network setup allows it) or by their direct IP addresses.
Then you have to assign an identifier index to each of them. #1, #2, etc. machines will be referred to later on your control boards as Remote #1, Remote #2, etc.
If you decide not to use one of your machines in the upcoming session, you don’t have to delete its name from the list, simply turn off its index. This way later you can easily switch it on back.
Project Root
If the shared project folders are on this machine then leave the setting as it is. If they’re on a separate server you have to set the project root to that machine.
Starting
Click Start. You’ll see the Remote Engine's Status popup as the renderers are starting. Composer UI won’t show up until all the renderers are started.
Channel Matrix
Open Preferences and go to the Channel Matrix section.
By default, you’ll see the list of all outputs of all the machines and you can assign your channels to each of them individually. (If you’re not familiar with what are channels and outputs please watch this consult Outputs and Channels, Multi-GPU).
In special projects, this gives you total freedom to send specific content to any of the individual outputs. But in the case you want to render the same content on all machines, we recommend using the simpler Unified multi-machine setup instead. That means that Channel 1, 2, etc. always goes to Output #1, #2, etc. on each machine.
Setting up the Control Boards
Some of the stock control boards are prepared for use in a multi-machine environment. E.g. if you are using virtual cameras use VirtualCam_4-Cam.xcomp. We’ll show this setup as an example now.
Inputs
First of all, you have to assign all the INPUT rows to a specific machine in Virtual Camera compounds. The system allows arbitrary configurations, e.g. each machine renders for two cameras. But we recommend sticking with the usual one machine - one camera setup.
The assignment is made via the Engine property of each INPUT panel:
NOTE: Since Aximmetry Version 2024.2.0, the tracked cameras (AR, LED Wall, and tracked green cameras) do not have an Engine setting for the INPUT panels. Instead, they are also defined by the SELECT CAMERA's engine.
IMPORTANT: The Input Device property always refers to the devices of the selected remote machine. Since you probably assign camera input to #1 on all machines you have to select Mapped #1 for all the INPUTs:
Cameras
You also have to assign the machines from the perspective of the cameras: you have to set up which machine uses which camera:
IMPORTANT: The CAM 1, CAM 2, etc. selectors above only serve editing purposes in a multi-machine one-to-one setup and only affect the controller machine. The remote machines always render the camera assigned to them.
You only have to switch between CAM 1, CAM 2, etc. to set up the camera paths and the placement of the billboards on the controlling machine. When you are finished with the setup you always have to switch back to CAM1 so that the controlling machine can fill its role which is rendering the Camera 1 image.
Editing the Camera Paths
You can edit the paths for each camera as usual. You select between CAM 1, CAM 2, etc., and edit the A-B points of each path. On the controller machine’s output and preview monitor, you’ll always see the picture of the selected CAM. Of course, you’ll also immediately see the changes in the output of the remote machine to which the given camera is assigned.
Placing the Billboards
When you select between CAM 1, CAM 2, etc. you’ll also see the Billboards belonging to that camera and you can edit and place them as usual.
However, there’s a catch: by default, you cannot see the actual image of the talent for the cameras that are assigned to remote machines. It’s obvious since their actual SDI input is connected to that machine.
So by default, you’ll see a placeholder image, a schematic figure of an actor. It’s suitable if you only want to adjust the position of the billboard, but not when you want to set up the correct proportions.
But there’s a solution: the system allows sending the input image over NDI from the remote machine to the controlling machine where we do the editing.
NOTE: For the NDI to work, the remote machines need at least Professional license.
For that, you have to specify the name of the remote machine on each INPUT panel:
NOTE: It always has to be the actual machine name, never the IP address. It will be used by the NDI channel system.
NOTE: make sure you run Aximmetry with administrator privileges. Otherwise, you might not receive the NDI signal in Aximmetry.
Set it up for all the inputs you wish to see then turn SHOW REMOTE on:
You can also select the quality of the transfer depending on your network bandwidth and the performance of your machines:
Now you’ll see the actual inputs for each machine.
Remote Engines Status
If for some reason, the connection is lost to some of the remote machines, the Remote Engine Status window pops up and indicates the problematic machine with flashing red.
If the problem is software-based (e.g. crash) or you’ve already fixed the hardware problem (e.g. network connection issue), you can restart the engine on all the disconnected remote machines by the Repair All button or by selecting machine(s) individually and clicking Restart. The current scene also will be automatically loaded into the remote machine and the system will continue to operate normally.
Project and File-sharing Issues
As we mentioned earlier Project Root(s) have to point to the same shared folder both from the central machine's Preferences and each satellite machine's Preferences.
Each time a remote Renderer is started the project sharing is checked by the system. If any Project Roots points to different locations the system stops the remote Renderer and displays the following message:
To remedy the situation, start Renderer Config on the remote machine(s) in question, go to Preferences, and set up the Project Roots properly. Then go back to the controlling machine and click Repair All. The system restarts all the disconnected Renderers and performs the sharing test again.
Another source of problems can be if you use non-project (absolute) paths when referring to a file resource from your scene. In these cases, the system also performs a check if the path points to the same file in the shared folder. The check runs on each loading of the compound and each time you change a file property to a non-project path. If the path is incorrect you will see the following message:
In this case, you have to change the path to ensure accessing the very same file to avoid different content appearing on different machines.
Remote Peeker
When you peek into the content of a pin or a connection in the Flow Editor (by pressing and holding Ctrl and moving the mouse over the pin or connection) you'll only see the content running on the controlling machine. The remote machines usually produce different results for the same pins since they typically render for a different camera.
When you want to find a possible problem within your Flow graph in a multi-machine environment it's useful to also see what's happening on the remote machines.
For that go to the remote machine and click the Peeker button on the Renderer.
A new peeker window appears. From now on whenever you peek anything in the Flow Editor of the controller machine the corresponding content will also appear on the remote machine's peeker window.
Operation Safety
This section helps handle situations when one of the machines suffers a software crash during a live show.
Crash on a Remote Machine
This is the easier case. If one of the remote Renderers ceases to operate you'll get the following message on the controlling machine:
In the unfortunate situation when the output of this machine is currently aired, first you have to switch to another machine on your studio switcher as soon as you can to prevent the interruption of the viewer's experience.
Then click Repair All. The Renderer is restarted on the remote machine and the scene is reloaded into it. When it's output comes alive you can switch back to it and operate normally.
Crash on the Controlling Machine
This is a bit more complicated situation because the controller machine cannot simply reconnect to the running remote Renderers. They have to be restarted and the scene has to be reloaded into them.
Please follow the steps below very carefully to ensure the viewer's experience is as free of interruption as possible.
When the controlling machine crashes the remote Renderers continue to run normally. Therefore in the unfortunate situation when the output of the controlling machine is currently aired, first you have to switch to any of the remote machines on your studio switcher as soon as you can to prevent the interruption of the viewer's experience.
Then start Composer on the controlling machine. On the Startup Configuration window go to the Remote Renderers page and make sure the Manual start option is turned on:
Click Start. The Manual start option prevents Composer from immediately restarting all the remote Renderers, so their picture will continue to be available. You'll see the following message:
Now load the scene in the Composer. Wait until it is completely loaded and its output becomes available.
Then switch to the output of the controller machine on your studio switcher.
Now you can click Repair All on the Remote Engine Status window. All the remote Renderers will restart and they will reload the scene. Wait for all of their outputs to become available. Now you can continue to operate normally.
