In this tutorial, we deal with receiving input from various external devices and the ways of using these inputs within the system.
Basically, input devices fall into three categories. The first is video and audio inputs. The second is the so-called controllers which enable turning on/off things, starting/stopping animations, and controlling the value of any numeric parameter in the system from external devices. This category includes the MIDI, DMX, GPIO devices, and the keyboard of the PC itself. The third category is the various camera tracking devices used in virtual studio environments. They enable synchronization of the motion of the virtual camera with the motion of the real physical cameras.
Let’s start with the most fundamental input device: the video input. To receive this kind of input we have to use the Video Input module. I wire it out to the output. In the Device property of the module, we can choose the video device we want to receive input from. The most basic category here is the DirectShow devices. Most of the commercial imaging devices like the web cameras and capture cards appear as DirectShow devices in Windows after installing their drivers. For e.g. with a webcam here that is connected to the PC via USB.
The professional capture cards used in studio environments usually have their own interface. However, some of them also appear as DirectShow devices it’s more reliable to use their own interfaces that appear as separate categories in this list. Currently, we have an AJA Corvid SDI card installed which appears as a list of available channels under the AJA category. Currently, a video source is connected to its socket #2, so let’s select Channel 2 as the input device. At this point, we only get a NO INPUT picture. The reason for this is that we also have to specify the exact video mode the source sends. In this case, it's a 1080i50 format signal, so we choose that mode. Now we have our input video.
Of course, this video signal can be used in many ways as we have seen before. For e.g. we can mix it with a background image. Or, since it’s a green background video, we can key it out and put the result onto the background. Let’s select a Keyer compound from the Common Library, wire the video through it, and adjust the settings of the keyer. These settings are discussed in other videos demonstrating the workflow in studio environments. Using the Blender module we can also fade the keyed image in/out.
Receiving audio inputs is done in a very similar manner. First of all the Video Input module itself can also receive audio signal. For e.g. an SDI signal can contain a maximum 16 channel audio track. In this case, the audio will appear on this Audio pin. Currently, it outputs an empty value, so this SDI signal does not contain any audio.
But we can receive independent audio inputs coming through the sound card of our PC or through an external audio capture device connected via USB. To do this we use the Audio Input module. Here we also have to choose a specific device. By default, it’s set to the DirectX Primary Capture Device which always means the device selected as the default recording device in Windows. But of course, we can choose any DirectX device directly, for e.g. the Microphone device I’m currently speaking into. We can see that the signal of my voice appears on the output of the module.
The next device category is the controllers. Suppose we want to control the fade in/out of the input camera image from an external device. Let’s right-click the name part of the corresponding Opacity property. We can see a number of Assign menu items each of them belonging to a specific device type. Now I want a MIDI controlling for this parameter so I select Assign MIDI. The system enters into a learning state waiting for any incoming signal. So I simply have to rotate the desired knob on the controller device and the assignment is made.
A little black triangle in the corner designates the properties that are already assigned to a controller. Hovering over the property name displays the exact identifier of the controller in question. Applying the controller also flashes the property.
All controller signals appear as a value between 0 and 1. Therefore in this case the Opacity parameter is set between 0 and 1. But what if we want a different interval? For e.g. let’s add a Blurrer module. The Radius property of this module has an interval of 0 to 5. Suppose we want to use the whole interval. We can use a Dash Scalar module for this purpose. Its basic function is to project its fader input value onto an arbitrary interval. We wire it into the Radius pin and set a 0 to 5 interval using the Minimum and Maximum properties. After that, we can assign the MIDI controller the In property of the Dash Scalar module. I turn another knob and now I can change the Radius value between 0 and 5.
Not all controller types have a continuous value range. They also can be two-state ones. Some examples of these are the keyboard or the GPIO. Let’s assign one of these to the Background Opacity. For e.g. assign the A key. In this case, if I hold down the A key the property gets a value of 1, if I release the key it gets a value of 0. In practice, we rarely want to use the keyboard this way. Instead, we usually want an on and off behavior for each press. We can do that. In the Assign Keyboard dialog, we can change the mode from Normal to Toggle. Let’s press A again. The system warns us that key A is already assigned to something. We click Remove Conflicts which means that all previous assignments of key A will be removed. From now on each pressing of key A is either turns on or off the background.
There are further modes. Increment and Decrement modes change the value up or down by a 0.1 step. For e.g. let’s assign key S in Incremental mode to the blur Radius. In this case, every press of the key increases the Radius by 0.1. Then let’s assign key D in Decrement mode. Now we have a pair of keys with which we can increment and decrement the value.
We have the Reset and Revert modes that reset the value to its default or to its last loaded value. Let’s assign key W in Reset mode. So now we can use S and D to change the value and W to reset it.
The last mode is Relative. It is only usable with MIDI controllers having “endless” knobs. In this case, the property can decrease or increase relative to its current value depending on if we turn the knob left or right. This can be very useful. If our scene property has some initial value and we start to turn the knob, the value won’t jump to an unpredictable value but changes gradually relative to its initial value.
Besides the keyboard and GPIO, there’s another two-state controller type: the MIDI note. Let’s assign a note to the Background Opacity in Toggle mode. In this case, we can use the MIDI keyboard to turn on/off the background. But MIDI note can also have a Velocity value that carries the information of how strong the key was hit. Let’s assign another note in Normal mode. We still only have 0 and 1 values at pressing and releasing the key. The reason for this is that velocity sensing is off by default. To turn it on we have to open Preferences and then the MIDI Devices section. We can find a “Use note velocity” checkbox here. Let’s check it on. We still don’t see any changes, because the setting does not affect earlier assignments. So let’s redo the assignment. The system warns us about the conflict, let’s remove it. And from now on we can see that by pressing the key we get different values between 0 and 1 depending on how hard we hitting the key. The off value is always 0, of course.
What if we want to cease an assignment? There are multiple ways to do that. In the right-click menu of the property, we can find an Unassign All item that removes all controller assignments for the property. We can also unassign controllers individually using the Unassign submenu where we can see all the assignments listed. As we can see so far we assigned the key A, MIDI note C3, and MIDI note A3 with velocity sensing. The latter is designated by a small “v” letter. So by clicking any of these the given controller is unassigned from the property.
The third way is to use the File / Properties dialog. We can see an Assignment section for each controller device type. Let’s select the MIDI Assignments section. Here we can see what controllers are assigned to what properties in the scene. We can also see if a note was assigned with velocity sensing or not, and after the property name, the assignment mode is displayed if it differs from Normal. For e.g. the assignment of note C3 was made in Toggle mode. Let’s also see the Keyboard Assignments. The assignments can be removed one by one, or all together using the Remove All button.
We can assign controllers not only to the module properties but to the buttons of the Control Boards. There are separate tutorials on the topic of Control Boards, for now, I will quickly assemble a board where the background can be turned on/off with a button. So I add a Pin Collector with a single Logical pin that is defined to be a button. We can see the new Control Board with a button that turns on and off the background. If we right-click the button we get the same assignment menu items as we’ve seen at the properties. Let’s select Assign MIDI. Setting the Mode has no effect here: in the case of Logical buttons, the controller will always work in Toggle mode. Let’s hit a MIDI key. From now on I can toggle the button and the background with that key. Let’s add a proper name to this panel, make it BACKGROUND. If we now enter File / Properties again, we can see the new assignment among the MIDI Assignments: the note is assigned to the On button within the BACKGROUND panel.
Certain controller input types can also be received via dedicated modules. For e.g. for MIDI devices we have a few module types. Let’s see the MIDI Controller module. Wire it into the Blurrer Radius pin. For the module, we can specify a MIDI input device, a MIDI channel, and a specific MIDI controller we want to receive, including all the MIDI Control Change, Aftertouch, Pitch Bend, etc messages. But the much easier way is to use automatic learning here as well. For this, we have to switch on the Learn property. Now we can move the desired knob, fader, key, etc. on our MIDI device. Upon this, the parameters are filled automatically. Furthermore, we can specify a value range, so we don’t have to insert a separate Dash Scalar module. Let’s set the 0 to 5 interval, and turn the MIDI knob to see the result.
Another module is the MIDI Note. It tells which note key is currently pressed. Let’s display its output via a Scalar Peeker module. We can specify a MIDI input device and a MIDI channel manually, or, of course, we can apply an automatic learning. Let’s turn on Learn and hit a MIDI key. The device and channel properties have been set automatically. From now on if I press various MIDI keys we can see the number of the currently pressed one. Also, the Gate output pin tells the Velocity of the last press.
In conjunction with this module, we can use the MIDI Note To Freq module which tells the exact frequency value belonging to a MIDI note. If I wire this value into an Audio Oscillator then I get the initial core of a simple synthesizer. Of course, much more elements should be added to make it a proper synth, but it’s not the topic of this tutorial.
Certain controller types have their own output counterparts as well. For e.g. GPIO signal can not only be received but also sent. It can be done with the GPIO Out module. We have to choose an output Device, a Mode, and the Channel whose state we want to control. The state itself is merely a logical switch since GPIO can only send an on/off state.
Similarly, we can control DMX devices. For e.g. we have a DMX Out Array module that can control ten consecutive channels simultaneously. For the convenient control of RGB lamps we can use the DMX Color Out module instead which sends the three components of color into three consecutive DMX channels.
Our last input device category is the camera tracking. It’s a rather complex topic, and a separate tutorial deals with the details. Here I’ll only show its base elements quickly. This kind of input can be received via a Camera Tracking module. Its main outputs are a Camera Transformation and a Zoom Factor that can be wired directly into a Camera module. Since we haven’t chosen any input device yet the camera resides in a default position. Let’s select a tracking device. Now we are receiving data, our virtual camera becomes synchronized with the real camera. If I now turn the physical camera the virtual camera follows it. The same happens if I’m zooming.
In virtual studio systems, the lens distortion of the used cameras also has to be taken into account, and the rendered image must be corrected accordingly. The related lens data is coming out of the Lens Distortion pin of the module. It can vary depending on the current zoom state of the camera. It can be applied to the rendered image using a Lens Distorter module. Let’s write the lens data into it. We can see the distortion on the Preview. Of course, the current distortion parameters are strongly overdrawn for the sake of the demonstration, this kind of fish-eye lens is rarely used in TV studio environments.
In the examples demonstrated so far we always chose a specific input device directly in the properties of the relevant modules. But we may want to create compounds that are independent of the specific devices used. Instead, we want to refer to the devices as logical ones, for e.g. Camera #1, Camera #2 or MIDI device #1, MIDI device #2, etc. If we later use these compounds we have to assign specific devices to these logical numbers. We can do that in the Startup Configuration in the Device Mapper section.
First, we see the Video mappings here. Let’s make the previously used AJA channel 2 device the logical video device #3. We can also specify a video mode here. That means that the mapping always includes a specific video mode as well. Similarly, let’s make the microphone I’m talking into the logical audio device #5.
If we now start the Composer and add a Video Input module, by selecting the Mapped #3 device we can see the input from the AJA card. The Mode property has no effect in this case since the mapped device carries its own specified video mode. So it’s recommended to leave the property blank. Similarly, if we add an Audio Input module and select the Mapped #5 device we can see the audio signal coming from my mic.
Certain device categories need a preliminary configuration to determine which devices even appear in the list. For e.g. GPIO and Camera Tracking are such device categories. They can receive data through various UDP or serial ports or from various TCP servers. These ports and servers appear as separate devices in the list. For e.g. if we open the list for GPIO we can see a number of devices designated with different serial ports. To determine which ports appear in the list we have to edit a configuration file. This concludes the tutorial of the input devices and controllers.
