A team of digital luthiers from the Music Technology Group (MTG) within the Audiovisual Institute of the “Pompeu Fabra University” in Barcelona have developed what some say is “the musical instrument of the future”. Their invention, which they named “ReacTable”, is an interactive multi-touch tabletop that is in principle, an electronic musical instrument.
“ReacTable” allows several simultaneous performers to share complete control over the instrument by moving and rotating physical objects on its luminous round table surface. This translucent multi-touch tabletop conceals underneath itself a wide-angled camera, which continuously analyzes the surface, tracking the players’ fingertips, as well as the nature, position and orientation of the various objects that are distributed on the table. The objects themselves represent the components of a classic modular synthesizer, while their relative distances and orientation define the topological structure and parameters of the sound synthesizer. A projector, also located just underneath the table, draws dynamic animations on the tabletop, thus providing a visual feedback of the state, the activity and the main characteristics of the sounds produced by the audio synthesizer. The instrument enables users to create complex and dynamic sonic topologies, with generators, filters and modulators – all by simply moving and relating physical objects placed on top of the “ReacTable”.
“ReacTable” relies on an open-source computer vision framework, which was also developed by
MTG researchers. Their
“reacTIVision” software is designed as a toolkit for the development of table-based, tangible multi-touch user interfaces and implements fast and robust tracking of fiducial markers attached onto physical objects as well as multi-touch finger tracking. “ReacTIVision” tracks specially designed fiducial markers in a real-time video stream, converting the incoming frames to black-and-white images by using an adaptive thresholding algorithm. The images are then segmented into a tree of alternating black and white regions (region adjacency graph), which is searched for “left heavy depth sequences” – exclusive sequences that were encoded into the fiducial symbol. When these are detected, the software matches the results against a “dictionary” to retrieve the symbol’s unique ID number, simultaneously calculating the marker’s center point and orientation. All data on the fiducials’ location, orientation and identity is transmitted by
“OpenSound Control” messages to the appropriate client applications. The described image segmentation is also used by “reacTIVision” to identify small round white blobs as finger tips on the surface.
The MTG team is currently in the midst of launching the industrial production of the “ReacTable” and while the product will most likely hit the market only towards the end of this year, several prototypes have already been produced on request for selected science projects, art museums and exhibitions. The inventors have been demonstrating the “ReacTable” by playing live concerts at some major art festivals and conferences – at this stage, two to four players usually perform on the “ReacTable” for around 45 minutes.
The “ReacTable” team was the recipient of a series of prizes for their invention – among these are the “Ars Electronica Golden Nica”, the “Premi de la Cuitat de Barcelona 2007” and two “D&AD Yellow Pencils”. Icelandic singer Bjork successfully used the “ReacTable” during her last “Volta” world tour.
More information on the “ReacTable” can be found
here.
