Credits:

• Production, Art Direction, Illustrations, Animation: SDNA
• Programming, Animation: PixelPusher

Recently I was asked by the visuals company SDNA to work with them on a public art game.  This would take the form of an interactive installation, shown around Worcestershire, as part of the 12 Moves Festival of Dance.  The idea was to get people, and especially kids, dancing and moving about.

UPDATE: An iPhone/iPad version is in the works!

Concept

Ben and Valentina at SDNA had this idea of a fantastical creature, trapped by a Victorian-era collector, who could only be freed by dancing in a particular way.  We discussed a variety of techniques for turing a series of dance movements into actions a computer could recognize, including using the Kinect (which was even more experimental at the time).  We consulted with a choreographer, discussing types of movements and how they might apply to freeing a mystical creature.

As with most projects, our imaginations were ahead of the technology.  Or more specifically, the budget – I like to say that anything is possible, given enough money and time.  Sometimes, that’s a bad thing.  Most of the time, it’s an excellent motivator for simplifying a concept.

After looking at the limitations of the Kinect, especially the limit of the number of people who could interact with it at a single time, we decided to go with augmented reality.  AR, as it’s known, uses computer vision and printed “markers” to tie the real world to the virtual.  In this case, holding up a card printed with the scissors marker in front of the exhibition caused a virtual pair of scissors to appear in the creaturae’s world, inside its case.

The installation ran for a few weeks in unexpected places around Worcestershire – in shopping centres, town halls, and high street storefronts.  The reaction was very positive – people were surprised to see public art in these places, and crowded around it to give it a try.

Here it is in action:

Some more photos are here:

http://www.flickr.com/photos/madpixelist/sets/72157627087176850/

Tech

On the tech side of things, I used the ARToolkitPlus block I ported to Cinder, and also my experimental re-write of Animata, and open source puppetry project.  Animata has some nice features for creating and animating “puppets” – this is what the creature looks like in Animata, with different limbs held together by “bones” (really, more like muscles because they can flex) and joints (the points):

I found Animata a bit clunky and difficult to work with, in the end.  The lack of an “undo” function is a huge handicap, as you have to save often, and when you run the simulation (to animate the joints) your puppet actually changes shape so that you may not be able to get back to the original state.

Luckily, the file format is in XML, and the code is open source, so I was able to get the idea of what it did, and then develop my own version that did some performance-enhancers like caching geometry and textures to increase the speed by a huge margin.

In the future, I’d like to redevelop Animata in Cinder, using the built-in glparams GUI for editing instead of the creaky and aging FLTK upon which Animata currently rests (and which is why it hasn’t been updated for OS X 10.6 yet and may never be).  I’d add joint angle constraints so joints can’t bend the full 360 degrees, and extend joints to have a “rest” position as well as a current position, so when the simulation runs it doesn’t irrevocably change the model (in fact, I’ve already done this to an extent).  Then there are some other features, like the ability to drag joints, cut bones, and more, that I’ve added to make this more game-like.

All this can be done, but first the basic structure of Animata needs to be re-thought, because it has an idea of “layers” consisting of images, with each layer held together by a separate skeleton.  In practice, this makes for horribly confusing projects, because you can’t organize your images into separate layers and join those layers together using a single skeleton.

I’d scrap the idea of a “layer” completely – what it really represents is a collection of images and vertices, out of which you build some skeletons.  Separating the data (vertices, joint positions, bones) from the presentation would enable it to save state – for undo, and also for reverting to original state, for example.  It also opens up some flexibility for writing different renderers, and making Animata more of a library than a self-contained app.  Right now, everything in the Animata code is lumped together – the presentation, the gui, and the data – making it very fragile.  I’ve done a lot of work to rectify that, and hopefully I’ll get some time to put more time into it!