Developing guidelines for real-time audio visual composition.
To develop guidelines for real-time audio visual composition through an analysis of (45) mixes of (10) clips from a video recorded during my residency at Signal Culture (1).
White noise is a random signal with a constant power spectral density. In digital image processing, the pixels of a white noise image are typically arranged in a rectangular grid, and are assumed to be independent random variables with uniform probability distribution over some interval.
New media, being unfamiliar, is often experienced as “noise.” There’s a whole genre of new and experimental music called “noise.” It’s about time for some visual noise!
Audio synthesizers first appeared in the late 1960’s, 50 years ago (2), and video synthesizers shortly thereafter (3). Several combined audio video systems existed in studios (4). However, only recently have video modules become widely available. Artists can now assemble their own AVSynths, and begin to understand their potential.
A series of mixes are generated using a (somewhat) value-neutral process so that I can study various combinations of AV tracks and develop guidelines for real-time audio visual composition.
Here’s the studio at Signal Culture –
Here’s the video input –
Control signals are generated using the 2 Serge Universal Slope Generators, and 2 Doepfer Voltage Controlled Oscillators. The Slope Generators and VCOs are cross-patched, “West Coast style.” That is, the output of the first module is patched to the input of second, and the output of the second back into the input of the first. Cross-patching adds a degree of unpredictability to the patch. The outputs of the Slope Generators are patched to the control voltage inputs of the Serge Voltage Controlled Filter, and two of the control voltage inputs of Dave Jones MVIP. The triangle wave outputs of the VCOs are also patched to the MVIP.
Audio is generated by the 2 VCOs, and processed by a Doepfer Ring Modulator and the VCF. The square wave outputs of the VCOs are patched to the Ring Mod, the output of the Ring Mod to the input of the VCF, and the output of the VCF to the Serge Mixer input 1 (5).
Video is input to and processed by the MVIP.
Audio and video are output to the studio monitors and the MacPro for recording.
(10) 2min clips, 01-10, are selected from the recording. The clips were selected sequentially starting 10min back from the midpoint of the recording. No attempt was made to prejudge any selection.
In mixing the clips the second clip is “Overlayed” on the first. Based on the result various effects are used including “Black & White”, “Luma Key”, “Find Edge”, and “Solarize”. The purpose of adding the effects is to clarify the mix. That is, to make sure that both clips are visible and can be seen in relation to their respective audio tracks.
Okay, I have (10) 2min clips from a 30min improvisation, why mix the clips? What do expect to find as a result? Consider improvisation as a means to explore a given situation, as a way of exploring new territory, in this case the AVSynthesizer. The 30min recording “logs” part of that journey.
There are two channels of information in that log, video and audio. For me the AVSynth is unique in that it allows both channels, video and audio, to share control signals. So one might look and listen for the correspondences between the image and sound. Do they correspond in some way? If so, how? That is, to what extent are the the video and audio linked; and what is it that creates that link?
These questions relate most to the individual clips being mixed, so what about the mix itself? Each mix consists of two clips and is, essentially, a duet between the two. This duet is not predetermined in real time but, rather, is a formal construct. It’s value is not as performance but as means of teasing out relationships that might be used in performance. That is, that might help to define the various design elements and attribute, or codes, that define composition on an AVSynth.
There exist lists of both 2D and 3D design elements and principles; and of film codes. So this might be a good place to begin. Here’s a list of some of these 2D and 3D design elements and principles (with attributes) –
– Point (position, size, shape)
– Line (straight, curves, scribble, length, width, orientation, direction)
– Plane (orientation, regular or polygonal), axes, 2D or 3D)
– Color (tint, hue, saturation, value, temperature, transparency)
– Shape (geometric, organic, size, boundary, centroid, balance)
– Texture (smooth, rough)
– Pattern (ordered, random, direction(s))
– Space (2D implied depth, foreground, middle ground, background, close, distant)
– Form (3D implied or real)
– Light (intensity, color, temperature, direct, transmitted, reflected, direction)
– Shadow (intensity, color)
– Movement (direction, regular, random, speed, acceleration)
– Repetition (regular, random)
– Frame (top, bottom, right and left sides)
– Horizon (real or implied, orientation)
– Perspective (one, two or three point, depth)
– Similarity (alike, association, correlation, continuity, support)
– Contrast (dissimilarity, opposition, erasure)
– Harmony (similarity, continuation or extension, repetition, rhythm, phrasing)
– Tension (dissimilarity, asymmetry, expectation, imbalance, unresolved)
– Balance (symmetry, order, equality, informal vs formal)
– Resolution (releasing tension, finding order, completing a phrase pr pattern)
– Hierarchy (order, significance, dependency)
– Scale (relative size)
– Focal Point (balance point within the frame)
– Proportion (relationship between elements, amount, distance, time)
– Dominance (hierarchical relationship between elements, size, intensity, volume)
Sound is another matter, especially as it relates to the moving image. Although the elements differ, similar organizing principle apply. Let’s leave it at that for the moment.
The process of analyzing the mixes is straightforward and begins with writing down what I see and hear. That is, make notes using the lists above as a reference.
I will post individual mixes with stills and comments, over the next few weeks and months.
(1) Signal Culture is located in Owego NY. It’s not the Experimental Television Center. It’s in a different location, several blocks north, with it’s own studio, workshop, library and space for artists to stay. Their residency program includes artists, toolmakers, and researchers.
(4) I worked as one of the first a video animators on a Scanimate system in NYC in the early 1970’s. Although not designed as an AVSynth, I programmed Scanimate for live performance. In the mid 1970’s, as artist-in-residence at the Experimental Television Center I worked on the Paik/Abe Video Synthesizer. For several years I took the PAVS on the road, performing at schools, colleges, community access centers, art galleries, and museums.
(5) This is a simple patch, using traditional audio synth modules. The Serge Universal Slope Generators and Voltage Controlled Filter were on my very first system. The Doepfer Voltage Oscillators and Ring Modulator are on my current system. These are standard modules. The patch is straight ahead, nothing fancy. Control voltage modules patched to sound sources, patched to sound processors, patched to a mixer. The same control voltage modules and sound sources are also patched to the video processor. Any change in the control voltages or the sound sources effects the audio and video processing simultaneously, and will be both heard and seen.