Introduction
This document describes how to set up the Keyer_Advance_B module in Aximmetry . With the module, you can cut out the green or blue background of an image/video. The cut-out region is marked with an alpha map so you can easily composite it with other backgrounds. Through this document, I will mark the property and or pin names with bold. For easy usage, you can use the control board feature of Aximmetry. In this case, the steps will remain the same, but a few things can be accessed differently. I will write down the differences.
Example
Original footage: 441_0195.mov (Anton) 13:26
Cut-out version:
Composite with a background:
Please note that in this example the left and right side will be manually cut out so those regions can be (and have to be) ignored during setup. It can be also true for your videos. Regions – which are not covered with green screen and you don’t want them to be visible in the final composite - have to be manually cut.
General information
This keyer was designed to achieve the best quality with good quality green (blue) screen footage.
Because of that, it is very sensitive to any kind of error. Noise and compression artifacts degrade the quality heavily. If you want to cut out shadows then strong shadows will also degrade the quality.
The parameters of this module have to be set very precisely so we strongly suggest to use a keyboard (and not a mouse) to do this.
Also doing the setup on a moving video is very hard so it is best to freeze a frame. Generally, semitransparent parts need more attention so try to freeze the video where the hair of the subject is more fluffy for example. Of course the more frames you check the result on the better the result can be.
In the current version, the result could look strange, if it is composited with an extremely bright background. (Let me know if this causes some issue in one of your real-life cases.)
Background color
The current version can handle only blue or green background types.
You can set a single color (Background Color) that most represents your green or blue. For this, you can use the pick color tool of the Aximmetry. Generally, you can achieve better quality if the green that you pick is closer (but not too close) to parts of the subject that are semitransparent (like hair). Similarly, if you feel that the quality of some part of the image is not that good you can try to pick the green (or blue) around that region. It is very important that based on your color choice the initial error (noise) can be different. Please ignore this when you pick a color. (see more about this first error later).
However, if you want to achieve the best quality with a stationary camera then you should set a clean plate ( Clean Plate ).Clean plate is an image that only contains the green screen, but not the subject(s). It can’t contain static objects either (like a table for example). If you don’t have an image that only contains the green screen then Aximmetry can help you create one from an image that contains an object in front of the green screen. See the documentation of the clean plate generator module. Control board feature of the Aximmetry will further help you to set a clean plate. See the documentation of the control board. It doesn’t matter what the clean plate contains in areas that you will manually cut out. Also, it is important to mention here too that using a clean plate can cause bigger initial error. Please ignore this. (see more about this later).
With the Use Clean Plate property, you can decide which one do you want to use (solid color or image). Once you set the background color or the background image the system will automatically detect if you have a green or a blue screen.
An example for a clean plate:
Different output types
The out pin of the module will always output the cutout version of the input video. (only if the module is active). The model however contains a second output pin called Monitor. The output of this pin depends on the Monitor Mode property. While you set up the keyer module, it is best if you watch the output of the Monitor pin and change the output type depending on what you set. I will write down later which output I suggest you to use. If you use the control board feature of Aximmetry then you can switch between the normal output and the monitor output quite easily. See the documentation of the control board.
Shadows
First, you have to decide if you want to keep the shadow of the subject. It doesn’t matter if it is a contact shadow, backdrop shadow, or other types of shadow. Depending on your choice a few steps are different in the process so we will discuss these independently.
Cut-out with shadow
If you want to keep the shadow you will need either a clean plate that doesn’t contain very dark regions or you will need a near-perfect green screen. In the latter case, even a soft gradient can cause problems.
Keep Shadow
Set this property on. At this point, you will see a cut out version already, but it will probably contain some noise.
Threshold
We have to increase the Threshold property until all the noise will disappear, but there are a few things that we have to consider.
The default value of the Threshold is zero and you can increase it with 0.01 steps.
I suggest you use the White Comp or the Grey Comp Monitor Mode now. The white background magnifies all kinds of errors so it is easier to fix those. At the same time, one can easily overcompensate these errors that will cause quality degradation overall.
If you see a very dark spot with the white background that is not good, so in that case, the threshold should be increased. If there is a big luminance fluctuation inside the shadow that is not good either, so the threshold should be increased. However, if you see a dim patch somewhere that is not a shadow then you should change it to grey background. If you can’t see this patch with the gray background then probably it is safe to ignore it.
What happens if you set too high value? First: the quality of the transparent object (like glass) will be degraded. Later: the color of the subject starts to turn into dark grey.
It is very important that dark shadows need a bigger threshold so in ideal case you should check the setting with the darkest shadow that the subject can produce (but not more than that).
Among several test, we made often 0.06 was a sweet spot. So if your final result is near to that then probably that is the right setting.
In this example, we set the Threshold to 0.07.
Filter
You don’t have to change it.
Low Cut
In most cases, you don’t have to change it. If you have some artifact that remained in the scene then you can try to eliminate that with this parameter. But be careful, even the slightest value increase will severely degrade the quality of the soft shadows.
Also please note that if you check the result with full white background (White Comp Monitor Mode) sometimes the shadows will look unnatural and often very noisy. This parameter can help you decrease these problems, but please check with your final composite, before you try to fix that.
Hight Cut
If you change to the Matte Monitor Mode then you will see the final alpha.
For example:
In most cases, the subject will not be perfectly opaque.
With the High Cut property, you can fix this.
Start to decrease it with 0.01 steps until you see something like this:
The value is set here to 0.9.
It is still not perfectly opaque, but in most cases, it will not be noticeable in the final composite.
(In fact, it is possible that even with the default value some will not notice the imperfection)
What happens if we set too low value: semi-transparent objects (like hair or glass) became opaque often with a weird color and the subject will start to look like somebody cut out him or her with scissors from a newspaper.
For most videos a value between 0.8-0.9 gives the best result.
Despill Strength
In most cases, you don’t have to change it. The module automatically tries to fix any green or blue color spilling errors.
In a few cases however this property can increase the quality of the colors.
For example: sometimes the green or blue colors are heavily reflected on the hair (or the face) of the subject. In this case, start to increase the value slightly until the green or blue will disappear from the hair (or face) of the subject.
Another example: the color of a fast-moving (motion blurred) object sometimes becomes distorted. It becomes green or blue. You can try to increase this slightly in order to decrease this discoloration. Unfortunately (in the case of motion blur) this parameter can’t always fix this issue in fact it can create so many artifacts that the overall quality of the picture becomes worst.
What happens if you set it too high? The subject starts to become pink or purple.
We suggest using the Gray Comp Monitor Mode for this setup.
Typical values are between 0.5-0.6.
Here is an example where this property could help.
With the value 0.5:
With the value 0.55:
Here is an example where it couldn’t help:
With the value 0.5:
With the value 0.55 (the green was removed but an unnatural edge appeared):
Edge Color Corr Width
In some cases, an edge with unnatural color appears around the subject.
Whit this property you can try to eliminate this problem.
Start to increase the value one by one until the edge disappears. Be careful though this can cause heavy artifacts to appear in some cases so try to set the absolute minimum here. You often have to make some compromise with this setting.
What happens if you set it too high? The edge (or even the inner part) of the subject becomes extremely distorted. Watch the parts carefully where the color of the border of the subject changes.
Sometimes the edge is dark. This is more visible with a bright background. Sometimes the edge is bright. This is more visible with a dark background. I suggest using White Comp, Gray Comp, Black Comp Monitor Mode for this setup depending on the brightness of your final composite.
Typical values: 0-4. In some circumstances, even 10 can work (and will not generate noticeable artifacts).
Here is an example.
Please note that the edge distortion on this example is much less prominent with brighter background (even with grey background)
With the value 0:
With the value 4:
Cut-out without shadow
Please note that the stronger the shadow that you will try to eliminate the more detail of the semitransparent part of the subject (like hair) will be lost.
Keep Shadow
Set this property off. At this point, you will see a cut-out version already, but it will probably contain some noise. Please note that if you use a solid color (Use Clean Plate property is off) then the noise will be less, but this is not necessarily a good thing. We can suppress this specific noise, but the edges (and semitransparent part of the subject) will be more accurate with a clean plate.
Threshold
We have to increase the Threshold property until all the dark noise will disappear, but there are a few things that we have to consider.
The default value of the Threshold is zero and you can increase it with 0.01 steps.
We suggest you use the White Comp Monitor Mode now.
If you see a very dark spot with the white background that is not good, so in that case, the threshold should be increased. If you see big luminance fluctuation inside the shadow or background then please ignore that artifact for now.
What happens if you set too high value? First: the semitransparent part of the subject will disappear. Later the whole subject will disappear.
Typical values: 0.01-0.07. In extreme cases, it can go up to 0.2-0.3.
It this example you can see dark spots in a lot of places.
With the value 0.05: (If you zoom in you can see a patch around the character, but let’s just ignore that for now.)
Low Cut (with Threshold)
Start to increase this until any unwanted shadow disappears and any unwanted patch in the background fades out.
The higher the value the more semitransparent part of the subject will disappear so I suggest you to regularly check the result with White Comp, Gray Comp, Black Comp Monitor Mode for this setup (depending on the brightness of your final composite). If it is possible you should check the result with the very final composite. It is often true that dim shadows or dim patches in the background (or around the subject) will not be visible on the final composite. So set the lowest value that seems good in the final image. (I would like to highlight that with White composite the errors will be heavily magnified).
In some cases (but not always) the higher the Threshold the smaller the Low Cut can be set.
Unfortunately, sometimes the opposite is also true. If you set the Threshold too high it will cut out some part of the hair with a sharp cut that looks unnatural. You can decrease this error with increasing the Low Cut .
If you want to achieve the best quality then try setting this parameter with smaller and bigger Threshold values.
In the following example, we didn’t use clean plate.
Threshold : 0.04, Low Cut : 0
Threshold : 0.1, Low Cut : 0
Threshold : 0.04, Low Cut : 0.38
It is subjective which one looks better, but there can be bigger differences when the subject moves so don’t forget to try this with video.
Hight Cut, Despill Strength, Edge Color Corr Width
These should be set similarly to the Cut-out with shadow case.
Filter
If you experience some flickering and or noise around the border of the subject then you can try to eliminate that with this option. This will however heavily distort the final result so use it only as a last resort.
Please note that Low Cut and High Cut have to be readjusted if you change this value.
Possible problems
There is color distortion around the border of the character
Try to use as much color information in your video as you can. 4:2:0 chroma subsampling is not ideal. Use at least 4:2:2, but the best would be 4:4:4.
Please note that in the current version of Aximmtery we can’t process 4:4:4 SDI input (only 4:2:2), but this might change in the future.
Flickering around the border of the subject
It is caused most probably because of the noise of the input. Inside the keyer, the only thing you can do is using the filter options. This will however degrade the overall quality so try to avoid using this. You can try to prefilter the input with a denoiser, but in most cases, denoisers are trying to reserve edges so it will rarely help with the problems around the edges (border). The best you can do is to reshoot the video. Adding more light to the scene usually decreases the noise radically.
The fluffy parts of the hair flickers
In a lot of cases, it is caused because of some compression artifact. The video compressors are designed to eliminate parts of the images that are less visible for a human. These “less visible” parts however still contain very important information if the image is processed via a computer. Without this information, it is not possible to do a good keying.
You can try to use less compressed video or a different compressor. Video compression artifact removal tools can sometimes help too.
There is a sharp cut in the hair
Probably the Threshold value is too high and it cuts out one part of the hair. Unfortunately, you can’t always decrease the Threshold, because it is possible that this will introduce unwanted artifacts.
The only thing that you can do is that you increase the Low Cut . The cut will remain, but it will be less sharp.
Flickering inside the subject
Please send us a test footage where we can check this issue.
The border of the subject is too sharp
The closer the Low cut and High cut are to each other the sharper will be the border of the subject. This will create an unnatural look of the hair. Unfortunately not much can be done but changing the light settings. High Cut has to be set too low if there was too much green (or blue) color spilling overall on the subject. Try to use less reflective cloth or use more Fill Light. Low cut has to be set too high if either too strong shadow has to be removed or the green screen contains darker regions and no clean plate was used. Try to use more diffuse light sources in the former and try to use clean plate in the latter case.
The border of the subject became very jagged
If the Low cut and High cut are extremely close to each other then the border will be jagged.
However, sometimes the jaggedness became so big as if somebody cut the resolution of the video half. This happens because the chroma information was not correctly interpolated. Please send us an example and we will try to fix it.
Keyer uses too much GPU time
If you cut the shadows then it is true that the higher the Threshold value the faster the calculation is. After you set the Low Cut you can often increase the Threshold without decreasing the quality (until a certain point). Try to increase the Threshold. When you see some quality degradation, try to dial it back a little bit.
Too much part of the hair is cut out
Using clean plate can help with this. If you use solid color picking a color around the hair can also help (just be careful to not pick too close). Also, more hair detail can remain if you don’t cut out the shadows.
