DAVE2

DAVE2 (Digital Audio-Visual Environment pt.2) is a blanket term for the progressive research that I have done with single-camera computer vision applications for the Arctic Region Supercomputing Center (ARSC). The primary goal of these applications is to use the body as an input device, like a mouse or keyboard. This is done by subtracting a background image from new video frames and drawing a box around what's left -- people in the scene. Certain information about this box, such as its width, height, and location, are then used as input for various audio applications.

The information gathered from the cameras is basic numerical data, such as the width of a box, height, and location in the viewport. It is sent out to the network as OpenSoundControl packets, which can be used to control a multitude of applications, including graphical and audio applications. In this case, the information is sent to an audio application written with Max/MSP, a visual audio programming environment.

The primary iteration of DAVE2 consists of four different regions, with objects detected in these regions affecting the audio processing in different ways. You can drag your hand in the left-hand region to pick from a selection of sound samples. Dragging your hand in the top region changes the playback speed of the current sample, and dragging your hand in the right region affects the pitch. This allows users to interact with the environment and create different sounds by simply moving their hands around.

DAVE2.1 (part one) from Dave Krnavek on Vimeo.

DAVE2.1 (part two) from Dave Krnavek on Vimeo.

The most recent version of DAVE2 allows for multiple pages of these region boxes. A separate region, called a page slider, is used to swap between the different pages. In the current implementation, each page consists of different types of effects that can be assigned to the looping sound sample. This lets the user cycle between a multitude of different effects, such as volume, panning, pitch, playback speed, and a delay-feedback loop. It was developed with input from Dr. Morris Palter, percussionist and music professor at UAF.

DAVE2.2 from Dave Krnavek on Vimeo.

DAVE2 was developed using the OpenCV computer vision library. This library works with many platforms, which allows the application to be run on different operating systems, including Windows and Mac OS X. The bounding box information is sent using OpenSoundControl packets to the audio processing program, built with Max/MSP.

Future work in this area may include detecting multiple persons in the scene, enhanced graphics, and research into other uses for this kind of input device.

3DHP and DAVE

3DHP (3-Dimensional Hand Positions) was the focus of my master's project from the computer science department at the University of Alaska Fairbanks. It consists of using two cameras to calculate the location of hand markers in 3D world coordinates. This project was funded by the Arctic Region Supercomputing Center (ARSC) for use in the Discovery lab (their virtual reality lab).

Stereo transformation

Humans use both of their eyes to perceive depth information in our world. Similarly, we can gather information about the location of specific objects if we have two slightly differing views of the object. This project searches for a single object that's visible in a pair of calibrated cameras (we know where the cameras are, along with how they're oriented and some other specifics). Once located, we can use some math transformations to determine it's location in the world. 3DHP uses the HALCON machine vision library to assist with the locating and transforming of these positions.

The calibration plate as seen through the (a) left and (b) right camera

Hand markers

To assist with the locating of the user's hands, special hand markers were created using ordinary light-emitting diodes (LEDs) and ping-pong balls. The ping-pong balls spread the colored light across their surface, which allows us to search for the desired color range. Since we want to use two hands, there are two markers -- red and green.

Output

When the system locates a specific hand marker, it sends a multicast packet out to a network address, specifying the location of the given marker. Multicast packets allow multiple computers to receive the data simultaneously, even outside the local network.

Location

The system is set up in the Discovery lab, which is ARSC's visualization and virtual reality laboratory. The cameras are set in the corners of the center display screen, which gives a view of the user standing on the floor screen. The user's hands are visible while standing on the center screen, which acts as a stage area.

DAVE

An application called DAVE (Digital Audio-Visual Environment) was created to be used with the 3DHP input device. DAVE consists of a group of sound samples that you can drag around the screen. Several parameters such as volume, playback speed, pitch, reverb, and delay/feedback can be manipulated for each sample, depending on their location on the screen. The different parameters are adjusted by simply dragging your hand up and down in the air.

Here's a quick video of DAVE being played in the lab:

DAVE was created using GLUT for the user interface and max/MSP for the audio processing. DAVE was created in the summer of 2005 with the help of Quinton Harris (a student employee at ARSC) and Sean Waite (ARSC summer intern).