By Ken McGorry
Issue: February 1, 2002

Total immersion for Chinese school kids

The exploiting of digital content in museums is on the upswing, including newly minted masterpieces of interactive digital content exhibited in educational museums. How about an exhibit of digital artwork that's so lively it follows you around the room?

Art Durinski's team of computer animators has taken a museum - Sony's ExploraScience Museum in Beijing, China - and turned the learning/art appreciation experience inside out. Durinski's way with a museum exhibit certainly involves stunning artwork. But, while the grammar school kids and high schoolers in attendance react to the exhibits, the exhibits react to them as well. Moreover, Durinski can change out an exhibit each year, not by storing away a bunch of old canvases, but by enabling a newly created software program.

Beijing's ExploraScience Museum (http://english.explorascience.com.cn/museum_guide/family_members/default.asp) is a joint venture between Sony and the San Francisco Exploratorium. (Sony also exhibits such fun science projects as Aibo the robotic dog and Vaio the laptop computer to the burgeoning Chinese market.)

Last December, ExploraScience opened a wing that offers a truly immersive experience. The new Virtual Aquarium literally surrounds kids with walls consisting entirely of rear-projected seagoing fish and aquatic creatures that bob and swish on the screens - and react to the movements of each visitor. "The Virtual Aquarium is a totally immersive, interactive environment utilizing infrared sensing, interactive computer animation, state of the art audio and high resolution video projection," says LA-based Durinski (durinski@otisart.edu). That is, as you enter the exhibit, a fish swims up to greet you. Your own fish. It seems to be observing you - and it follows you around for the duration of your stay. And, thanks to those specially designed infrared sensors behind a screen called HoloWall, it also knows how tall you are. If you're a small child, the fish swims down to your level. Sony's HoloWall conceals an infrared projector that scans a visitor 15 times per second and is able to sense the person's movements.

Did he mention audio? Yes, the 13 individually designed photoreal fish species, sea horses, sea turtles, a sting ray, shark and other creatures each have their own signature vocalizations thanks to Venice-based sound designer Frank Serafine (www.frankserafine.com).

Serafine, a veteran designer of sound for feature films as well as games, ride films and other interactive projects, created an undersea aural environment for Virtual Aquarium and added an unusual aspect: the fish characters all make their own individual sounds. Serafine also worked with Durinski on the debut of Beijing's ExploraScience a year and a half ago. That time, he relied on the relatively traditional MIDI workflow with Yamaha samplers, tape and timecode. Serafine originally contributed to Durinski's HoloWall projection "floor," which features interactive animated bugs and also a "wall" that allows kids to create colorful animated rings which, when played together, can compose original music.

For this year's Virtual Aquarium, he manipulated the sounds of real land-going animals and reworked them to approximate a whimsical notion of how aquatic animals might sound. Little fish make cute vocalizations that are reworked from recordings of chipmunks. The nasty animated shark gets its voice from what were once alligator sounds. And there's even a reversioned elk to be heard among the sea-going creatures. Serafine's recent interaction with Durinski and company was quite different from the workflow the year before. This time instead of traveling to Tokyo, where the computer animation team was working with a Sony technical team, Serafine used Digidesign Pro Tools and worked long distance, adding his sound design to QuickTime animations which he would FTP back to Tokyo. And he appreciated the use of the project's SGI computers into which his tracks were programmed for surround playback through four or more speakers. "The sound follows you this way and that way," Serafine says. The further away the fish is, the more echo is used; the closer it is, the drier it sounds.

"The thing about these high-end entertainment complexes is that you can go and update the exhibit as often as you like without any big expense because the technology of the exhibit itself doesn't change," says Durinski, who also teaches in LA at the Otis College of Art and Design. "You just go ahead and change the software."

Durinski and his animators used both Maya and Discreet 3D Studio Max to create the colorful 3D denizens and their watery surroundings, complete with filtered rays of light and exotic coral formations.

Now Serafine is experimenting with a desktop Foley program from San Jose's Mindmaker called FlexEffex (www.sontage. com/products/flexeffex.html), which allows the sound designer to create and design all kinds of footfalls (for example), giant, small, crashing, tip-toeing, from one little sound bite. For Serafine and Durinski, Virtual Aquarium was something of a reunion. The two first worked together on Tron 20 years ago - the sci-fi classic concerning good and

evil characters locked in a pitched battle within a computer. Tron 2 is coming and the two men expect to be seeing more of each other, as well as other members of the original effects crew.