PTZ Cameras

Please note that this is a BETA version of the document. All information presented is correct but we are working on improving the details.

Introduction

What is PTZ? 

PTZ is an abbreviation for Pan-Tilt-Zoom. It is a mechatronics solution that can adjust Camera orientation, Zoom, and Focus parameters over software or controllers.

What Type of PTZ Solutions are out There?

There are two main kinds of PTZ solutions

• “Motorized Pan/Tilt heads” + “Zoom Device” 
• All in one “PTZ Camera”

NOTE: “Motorized pan/tilt heads” are not the same as the more advanced “Motion Control Rig” even if they can communicate with the same protocols.

Motorized Pan/Tilt Head

Motorized Pan/Tilt heads are a Pan+Tilt system that can be used with multiple Compatible Cameras.

This is really useful if your production needs to:

• Use different lenses
• An existing camera that you already own and used but the camera itself does not have tracking capabilities
• A very specific camera due to its properties

https://multicam-systems.com/robotic-systems/spirit-head/

Lens Encoder (Zoom Device)

Lens Encoder (Zoom/Focus Device) is a separate device from the Pan/Tilt head, which can provide Zoom and Focus information. So If you use Motorized Pan/Tilt heads and you wish to have Zoom/Focus control. This provides the data to determine the Zoom/Focus values. It is either:

• Only a Zoom device like “Glassmark” (this provides fixed focus only)
• There might exist cameras that use Direct Data connection from the Camera to send accurate Zoom/Focus information. 

You will additionally need parameters for lens distortion. Some Tracking systems can provide this information, while for others Aximmetry can measure it during the Calibration process.

Pan/Tilt Head with Built-in Encoder

https://www.canon.co.uk/cameras/ptz-camera-system/cr-s700r-remote-pan-tilt-power-head/

IMPORTANT: The above example does not have Tracking capabilities however it is possible there is a similar system that might have.

PTZ Camera

PTZ Camera is usually a complete Pan-Tilt-Zoom-Focus system where the camera is built into the mechanism. PTZ cameras are easier to set up as they send Tracking, Zoom, and Focus data together. PTZ cameras have been used by a big portion of our users because of their relatively easy operation and setup process.

"Motion Control Rig"

Motion Control Rig is actually a Robotic Camera System that provides Complete Tracking of the Camera, and they are not considered PTZ systems even though they can still use the same protocols as PTZ cameras. Some of their sub-variants only send the same type of information, but use different protocols.

Sending and Receiving Data to and from PTZ Cameras

Usually, the user can send data to control the system, however, only a handful of systems can provide feedback for accurate tracking.

Receiving Tracking Data from the PTZ System

Aximmetry Supports Receiving PTZ Data from multiple protocols, like

• Free-D
• Visca (obsolete format, limited compatibility with new hardware. Please see your product description if it only receives PTZ controls through Visca or if it can also send back tracking information)

Over these protocols, Aximmetry can provide accurate tracking for your production which you can even record and reuse.

Once your Tracker is Calibrated properly and you added your tracking solution to the list of tracking devices. You can add them as an indexed device or select them in our tracking Compounds and Modules.

Sending Control Data to the PTZ System

A PTZ system can receive commands from a controlling device or automatically control itself based on its programming. There is a wide variety of hardware and software out there for this task. Generally, there are two types of controls.

• the controlling device can send precise PTZ information and the camera can go to that location.
• the controlling device only sends a command to the PTZ camera which itself decides if it can fulfill that command if yes it does. (In the example you cannot zoom more than the maximal Zoom. or you cannot rotate more than +-180 degrees.)

IMPORTANT: Aximmetry does not support Sending PTZ Commands and does not provide any means to control these cameras. Controlling has to be done in:

• The camera’s own software
• Separate controlling hardware
• Separate Control Software like NDI Tools/NDI Studio Monitor

PTZ Camera Installation

When you set up a PTZ camera there are some really important aspects that you have to take care of to receive stable and accurate data from the Camera.

Placement

It is of utmost importance that the camera is completely leveled. We suggest getting either a stand that can automatically level itself, or get a Digital Leveler which can provide you complete accuracy when you set up your stand or mount.

It is suggested that the stand/mount of the camera faces the Green Screen/LED panels at a Right Angle. The camera can Pan to any target even if it is not in the center. This can simplify setting up the Virtual Studio a bit.

The stand of the Camera has to be completely immovable. It should be:

• fixed to the ceiling or wall or
• rooted to the ground.  

If you can't make permanent mounting points and it's on a Mobile Platform. Once it is calibrated, It has to be 100% stable and undisturbed from people and forces.

IMPORTANT: If your camera moves, your calibration will be inaccurate from that moment on. 

NOTE: We also suggest paying double attention to cable management. As even if your camera is safe if someone accidentally kicks the cable it can easily move it.

Record Camera Position

Since PTZ cameras are not complete tracking solutions they usually can't move without the help of another device. The system in itself only gives you rotation information of Pan+Tilt and it doesn't provide Position Data. We need to gather it ourselves by hand or by automatic lens Calibration.

In the case of all-in-one PTZ Cameras, you only need one piece of information, height position. These devices' sensor position is usually at the Pan/Tilt rotation center. The Height position is measured from the floor. It is usually measured at the camera's Pan/Tilt Rotation pivot. Aximmetry usually needs this information in centimeters.

In the case of Motorized Pan/Tilt Heads 

• You need the height position of the Rotation pivot (in centimeters).
• You need the relative position of the camera sensor from the rotation pivot (XYZ translation in centimeters) (see image).

Using Together with Robotic Legs

Some PTZ manufacturers have additional track-dolly technology to raise the platform it is standing on with an extendable leg and send tracking information. Or a complete Robotic arm working together with the PTZ camera. These are usually pretty unique systems that can communicate in a variety of ways. Please always refer to the manual or manufacturer information regarding the setup of these. Usually, the manufacturers have a system to aggregate the different signals into a single data stream per camera.

https://multicam-systems.com/robotic-systems/spirit-lift
https://business.panasonic.it/telecamere-professionali/solutions-videos

Tracker Network

To minimize latency It is recommended to use a separate network for the tracking data as per your office's regular network. This ensures that there is no interference if anyone else uses the network.

In case you only use the network for tracking data you should be able to do it with a single Gigabit connection. (Even though a 100Mbps connection might be enough in theory. We suggest using at least a gigabit connection).

In the case of Free-D Protocol

Always make sure that your firewall doesn't block either the IP address or the ports of the camera. Or the software itself. If you have issues connecting to your tracker, turn off the firewall altogether while you set it up, and remedy the firewall issue if any.

PTZ cameras usually use multiple ports to send information to multiple clients at once. By default, these ports are in consecutive order. For example on multiple models port, 1010 is the default first port. The other ports which these cameras use are usually: 1011, 1012, and 1013.. and the IP might even match. It is not an issue if you use one camera. But if multiple cameras try to communicate on the same port it will create tracking anomalies.

We advise setting each camera's ports distinct from each other. At least a 5-port difference, but we suggest using even more just to be sure.
The reason we suggest making the safe bet and setting each camera to distinct ports is that it ensures no two cameras share the same port and no crosstalk occurs.

NOTE: The best case is that all of these ports are exposed in your PTZ control panel and you can edit them individually, but the feedback is that some systems might only show the 1st port. 

Network Setup for Panasonic PTZ Camera 

In case you have a Panasonic PTZ camera with Free-D support. After you connect the UTP cable, the DHCP gives an IP for the camera. You can reach the camera's control panel through a web browser. 

• [ip-address]/admin/index.html
• [ip-address]/live/index.html
• Or by clicking the small cogwheel icon at the bottom of the NDI tools.

You have the following views: 

• Live
• Setup

For setting up the network settings, Setup / Tracking Data Output. You land on a Setting status panel where you see your basic network info and status. If you use a LAN connection make sure IP (UDP) is on. If not you can turn it on in the Connection type menu:

You should go to Setup / Tracking Data Output / IP out:

Here you can see 4 possible output Clients. You can turn each of them on or off separately.

The Client in this case the computer you run Aximmetry at. That is where the camera will send its Tracking Data.

Each Client has an IP and a port. It is important to note that the system does not prevent you from using the same IP on different ports. By default, these ports are just after each other. It is a common user error to change these IPs to the same IP while having the original ports while using a second camera set up similarly. While the Client selection should help you select the right setup for multiple receivers, we see that it can cause confusion for some of our clients, and they accidentally turn all of them on. Or forget to turn them off. When this happens, we call it crosstalking. The two systems compete with their data and timing and it manifests in cameras jumping away to another location and then back at random frames. Therefore just to be safe we suggest using port settings that cannot accidentally crosstalk. No two Clients should ever use the same Port. And as there are 4 ports in total we suggest having at least a 5-port difference between cameras.

So if the first port is set to 2000 at CameraA. It should be at least 2005 at the next and 2010 at CameraB.

Or 

• 2000, 2005, 2010, 2015 CameraA
• 2100, 2105, 2110, 2115 CameraB.

Or 

• 2000 to 2004 is CameraA
• 2100 to 2104 to CameraB.

NOTE: This also helps you to remember the port settings as you can number your cameras using the 2nd digit of the port number.

Once you set up the ports in your camera, you should set the selected Client's port in Aximmetry as well. You can do it by going into:

Device Mapper / Camera Tracking / Manage Devices... / Camera Tracking / Free-D

Then press the Add... button, and add the desired ports for your Cameras,

Make sure you use the Data Rate you are recording your Video/Sending your Tracking information from the Camera.

and press OK:

Then if you plan to use the Compounds provided by Aximmetry, map them as indexed Camera Tracking devices at:

Device Mapper / Camera Tracking

Video Input

You can receive video signals on multiple Video Input types.

SDI

While professional broadcast cameras can be connected to both SDI and NDI. 

We suggest connecting your camera through SDI if possible as it can reduce the System latency. Also, other devices on a network won`t interfere with the Data transfer. The SDI has a direct connection to the Capture Card and the Machine. While NDI usually goes through our networking devices first. 

NDI

If your solution has to be NDI due to Distance or technical issues. Make sure your NDI has ample Bandwidth.

It's best if they communicate on a completely separate network compared to what is used for PCs and WiFi in your studio.

It's okay to use the same network for tracking, however as a video signal sends a bigger package than that of tracking data, You should use at least a tested Gigabit network with appropriate grade cables. Or preferably use a 2.5 or 10 Gigabit network with even better cables. Try to minimize the signal latency by only using as much cable as necessary and not more. Try not to use too many switches or routers in between the machine and the camera. In case you use a single camera it might be best to directly connect the UTP to the Computer. (though this needs manual connection setup of that network) 

HDMI

Of course, Non-Professional Grade PTZ cameras might not have any SDI connection possibilities, or you use a Photo Camera with mini HDMI output with a PT head and LensEncoder. In this case, use HDMI if available as it can send video data to relatively long distances. 

Other

Some older PTZ cameras might communicate with DVI or even VGA. Aximmetry can receive these signals as long as you have an adequate Windows-compatible capture card for it.

Camera Settings

Each camera manufacturer can have its own solution to communicate with the Camera to edit its settings. The two main types are:

• browser-based settings similar you find for your router. Where you enter a local network address and log in to its control panel.
• controlling software/app that you have to install on your machine to reach the camera's settings. 

The most common settings that you can find in these menus 

• Controls the Networking Parameter of the Camera itself
  • IP address
  • Port
• Controls the camera mechanically/Digitally like
  • Zoom
  • Iris/Exposure
  • Pan/Tilt
• Controls the video signal that comes from the camera
  • Resolution
  • Frame Rate
  • Image Mode
  • Choice of Video Protocol if multiple protocols are available
    • SDI
    • NDI
    • HDMI
    • DVI
• Controls the tracking signal which comes from the camera
  • Tracking Frame Rate
  • Tracking Mode
  • Network ports the tracker communicates with the Client apps.
• Lens Calibration Data
  • It can happen that the camera has a built-in solution for Lens Calibration, in this case, you don`t need to set it up in Aximmetry
• Image Manipulation
  • LUT
  • Color Correction
  • Built-in Effects

Video Mode

When you set up your system for Tracking, it is of utmost importance that you set matching output and input signal modes. So if you are using 1080/25p on your camera use 1080/25p in Aximmetry for the Camera signal, and 25p for the tracking signal.

Usually, all-in-one PTZ cameras do not have their own LCD indicator, and control software varies regarding how and which information is shared with the user. It is not always obvious what actual settings the camera really has. It is absolutely important therefore that you check these settings in the camera's very own control panel through the network every time you start the system. 

The Camera Mode can unexpectedly change for various reasons such as:

• Power Down
• Interference from Control Software
• Firmware Update
• Human interference
• Etc. 

Therefore we suggest always checking the video mode after powering them up.

Calibration

Using Aximmetry Camera Calibrator

To calibrate your PTZ camera, we recommend using Aximmetry Camera Calibrator, as it can automatically calculate your PTZ camera lens.

Using Aximmetry Basic Calibrator.

To calibrate your camera you can also use Aximmetry Basic Calibrator. The Basic Calibrator offers a manual calibration process. 

NOTE: Before you start this project, it's best if you place an object, like a chair into your green, so you can track its movement with the camera.

Lens Calibration

While manufacturers should create a perfect lens… It is not always the case. There might be manufacturing defects. Some lenses are specially tuned to have their own curvatures at specific angles. To have a better or more artistic output. 

By default, this is not ideal for a virtual camera, as it always assumes that we are using a perfect lens. We need to provide additional information about these distortions so the Virtual camera can make the necessary adjustments to match the Real camera's recording.

We also need to calibrate the camera's Focus and Zoom information.

How to Setup PTZ in Aximmetry Basic Calibrator:    

NOTE: When you start Aximmetry Basic Calibrator, make sure your tracking devices are already set up in Aximmetry.

Before starting to record Calibration Points, first, you have to set up your Camera Properties.

If you are using an all-in-one PTZ camera and you set your system upright in the real world, you only have to worry about sensor width and the distance from the ground. These cameras' sensors are usually at the PT (Pan/Tilt) rotation center.

In case you are using a PT head, you also have to add the Delta head position. Delta head position is the translation (distance) of the sensor compared to the PT rotation center.

Sensor Width is the size of the Camera's Light/Image Sensor. 

To get the Sensor Width please check your camera's manual. Make sure to use width and not diagonal size. In case it's a diagonal size please try to backtrack what is the right size. 

NOTE: In the case of all-in-one PTZ cameras the sensor is usually not accessible. However, if you are using a PT head with a Camera that has a changeable lens, you can simply remove the lens and measure the width of the sensor with a ruler.

In case your setup is not perfect, it can still be okay as you can precisely adjust the sensor size at a later point.

PTZ Rotation Center Height From the Floor:

The distance from the ground is relatively easy to Determine. 

In the case of an all-in-one PTZ camera, it's usually the height of the Tilt Axis. And you don’t need to set up Delta Head transformation as it is generally the same as the PT rotation center. 

In the case of PT head, it's also the Tilt Axis but in this case, you have to give Delta Head transformation.

Delta Head Transformation

You can imagine delta head transformation as the translation vector from the PTZ rotation center. It is usually only forward and upward transformation. As the mounts are usually placed on the Pan axis.

Testing PTZ Camera Calibration with an NDI Monitor or Control Pad.

To make a hand calibration you have to be able to check if the positions of the virtual objects are in match the positions of the real objects that the camera records. You can do it by controlling the PTZ camera directly or through pre-recorded commands.

The very best solution to test your camera calibration is if you have a physical controller for your PTZ cameras. Using a mixture of Joystick movement with saved camera positions, which has proper time for movement. 

In case you don't have a physical controller, you can achieve the same with a PTZ camera controlling software like the NDI tools/ NDI monitor.

Best Practices for PTZ Testing

• Recorded States

Most PTZ cameras have the possibility to save PTZ transformation on a control panel to be able to simply animate from one position to another. This is also a really useful tool to test and Hand Calibrate your camera. As these functions allow you to repeat the same process with the same hardware settings, while only the calibration outcome changes. You can easily see the difference between settings to approximate your outcome. 

We suggest using your Numpad for changing between states 1 to 9.

It is also wise to set up 1-3 as Max Zoom out, 4-6 as Middle Zoom, and 7-9 as Full Zoom from the beginning. As it will speed up setting up your calibration points. 

• Keyboard Control

Alternatively, if you use NDI tools to control the PTZ camera or any other software that allows you to use a keyboard to control the camera. It is advised to move up and down from a fixed saved position. Or Left and right from a fix-saved transformation. This ensures that you only have to worry about a single axis during calibration, therefore, having a better chance to see which parameters need to be adjusted. 

• Joystick and Mouse Controls

Last but not least you can directly control your camera with your joystick or mouse, this allows complete freedom of motion. And this is genuinely used as a final test, to make fine adjustments at the end if necessary.

Tracking Delay

Once you have set up your basic properties and you are familiar with controlling your PTZ camera. You should set up your tracking delay. Video and Tracking data come on different protocols and go through different processes. Therefore they have a bit of a time difference in the system, which we can simply sync by providing a tracking delay value. Tracking delay is calculated in frames. 

To set up tracking delay, you should make a sudden move with your camera and see on the screen how much later (or sooner) the 3D environment moves compared to the camera's image. Write in your perceived time value (in frames). Then make another fast movement. Adjust the delay value, then repeat the movement/adjustment process till you feel your 3D and Camera images are starting and stopping their movements at the same time.

Tracking delays can change every time you start your system. So we suggest adjusting your tracking delay values before every show.

Marker Placement

To have a better understanding of what is happening during the calibration, Aximmetry provides a checker textured floor and a set of Marker 3Dobjects 

You can place these  Markers by editing their position in the menu. The best practice is you place them on the legs of the chair we suggested using earlier. It is really easy to see if the two objects are in the same position after rotating the camera.

Calibration Points

To imagine what Calibration Points do is best if you picture a graph. This graph is responsible for adjusting the distortion differences at specific zoom points. By default, the Aximmetry uses the parameters of an "ideal" lens. But these are only default values not for a specific lens.

Usually, we take the first calibration point where the camera is completely zoomed out. This gives a baseline Focal Length and Horizontal Field of View. (Also other fine tunings)

To set up this value you should use the above-mentioned Camera motions to move the camera around. See if your Virtual Marker fits on the Real World Marker. Adjust the Focal Length and Field of view till it matches completely.

Next, if possible we should zoom in Completely and adjust the calibration there as mentioned previously.

With this, we are having a linear transformation between the two endpoints.

At this moment you should see how accurate your zooming is and test if Zoom interpolation Mode 1 or Mode 2 seems to be closer to the right movement. (Depending on your camera it can differ.)

So after deciding on the mode for now.

You should adjust camera calibration as described at Half (Mid) Zoom. Then see if your Zoom mode can still apply.

Usually, 3 points are not enough for perfect calibration. So you should set up the 

• Quarter Zoom
• Three-quarter Zoom

And after this point make multiple Motion and Zoom tests to see if these points are completely accurate or not.

When the 5 main point is as accurate as they can be, you still see imperfections in the in-between steps. It's best to just add more points in between as you feel necessary. But always make sure the first 3-5 points are 100% accurate as it saves time and hassle for you later.

Zoom Interpolation Mode 1 and Mode 2. 

Once you set up your Calibration Points. You might see that between the points the Camera does not calculate Zoom properly. Some camera uses different mechanical and optical behavior from other. For this case, you should try out Interpolation Mode 2 instead of 1. And use the one which is better suited for you.

Receive Tracking information in Aximmetry.
To receive tracking information in Aximmetry. You have to add your tracking devices at the startup/Devices/Manage Devices/Tracking Devices 

Then follow our [Basic lens calibration without lens distortion] and Basic Calibrator

What to Look for When Choosing a PTZ Camera.

There are multiple solutions to achieve a “PTZ Camera” on the market. 

Cheaper models tend to be closed systems that use their proprietary protocols and do not provide tracking data to third-party software. These are usually sold as security cameras.

Some more expensive professional PTZ cameras have the ability to connect to third-party software and send tracking information through open protocols like the commonly used “Free-D” protocol. These cameras are usually sold for Studio and Scientific purposes. 

NOTE: There are some extremely rare cases when you can find non-professional cameras providing tracking features on an open protocol. However, usually, these cameras’ specifications are not clear or consistent. We suggest always inquire information about tracking capabilities through the channels of the Manufacturer before purchasing.