Director Charlotte Mikkelborg and Picture This Productions (IG: @picthisltd) recently collaborated with the Mary Rose Museum to create a multi-sensory-augmented game that coincides with the anniversary of the sinking of Henry VIII’s favorite ship back in 1545. Using their smartphone, the Time Detectives: The Mystery of the Mary Rose app immerses players into the sights, sounds and even smells of life on-board the ship almost 500 years ago.
Players become time detectives, and are transported back in history to experience the sights and sounds of the UK’s Portsmouth dockyards. The Mary Rose sank in July 1545, claiming the lives of almost all of its 500 crew members on board. For those playing remotely (via the App Store or Play Store) in the US, the game features photorealistic 3D characters that appear in their environment, almost as if incarnations of the ghosts of the crew.
Back in 2020, Charlotte Mikkelborg (Twitter: @cmikkelborg) directed and produced another interactive historical fiction, the VR narrative Fly, working closely with Neil Corbould's Oscar winning practical effects team (Gladiator, Gravity).
Fly was commissioned by British Airways for their centenary celebrations and narrated by Joanne Froggatt (
Downton Abbey). The project had a sell out run at London's Saatchi Gallery, and also showed at the Tribeca and Cannes Film Festivals in 2020. It was the winner of a Lumieres Award for Best VR Entertainment, and was also voted Best VR Experience of 2020 by the UK government.
“Fly immersed all of your senses in the story,” says Mikkelborg. “It took you on not only an audio-visual journey, but also a haptic and sensory journey. But the downside was that it couldn’t easily scale. Furthermore, VR is expensive to make, as you are creating entire new worlds, and during COVID, those budgets weren’t as available.”
Mikkelborg then began exploring the idea of taking family audiences on immersive story journeys using AR. This led to the concept behind Time Detectives: The Mystery of the Mary Rose.
“We managed to secure funding for the game from Innovate UK, which is the UK government body that supports innovation, as the game had two key innovations,” she explains. “We wanted to introduce volumetrically captured human characters into mobile AR gaming, which hadn’t been done before. We also wanted to augment your reality with, not only the sights and sounds of the past, but also the smells. I have used smell quite often in my experiences (The Journey 2018,
Fly 2020). Not in the gimmicky way that we all might have experienced it once or twice in theme park 4D theatres, but in the subtle way that it pulls our attention in everyday life. Scent is the only one of our senses that speaks directly to the limbic part of our brains that forms memory so, if you use scent right, you are quite literally making more memorable experiences.”
In terms of the volumetrically-captured characters, this was a challenge, as volcap data sets, as generated by the likes of Dimension Studios with their multiple camera setups, are very heavy and so not well suited to mobile gaming.
“So we used a lighter weight volcap method, which involved shooting our characters in front of a green screen, with a DSLR camera and a Kinect Azure close to each other,” she explains.
According to the project’s volcap producer, Dimitri Souzac, the Kinect Azure records the depth video and a low quality RGB video, while the DSLR gives the decent RGB image.
“Syncing together the shots of the Kinect and the main camera, and making adjustments (distortions on the image to make them fit together) gave us, in the end, a good quality image with the depth video,” says Souzac. “In order to reduce errors in the depth map, we then removed our green screen on the original RGB video to get a good mask to work with.”
The video obtained in the end is effectively a stereoscopic top/bottom video, with the RGB video on top and the depth video on bottom, adds Mikkelborg.
“Our characters present effectively as 3D in the game but, if you try and walk all the way around them, you will see they are not completely 3D,” she notes. “However, I think that what you gain in being able to get a sense of being in that character’s presence far outweighs the trade off of not getting full 3D.”
Each of the characters in the game is based on a real character from the ship. The team used DNA and isotope analysis of the skulls and other bones to create the characters. The scientific analysis was provided by lead historian at the Mary Rose, Dr. Alex Hildred, as well as other written evidence dug up by consulting historian on the project, Dr. Hannah Platts from Royal Holloway University London.
“In terms of the main game development, Time Detectives was developed using Unity games engine,” says Mikkelborg. “As anyone working in games or immersive experience production, or even those working in film who’ve utilized virtual production techniques knows, there our two main games engines: Unity and Unreal. While the Picture This Productions team behind
Time Detectives are also fans of Unreal, and are currently using the engine for another project, we chose Unity for
Time Detectives because, in the words of lead developer, Grigor Todorov, ‘When working with complication applications, it takes less time to make something in Unity than it takes in Unreal. There are more tools available for this kind of app in Unity as it’s the main game engine used for mobile game development.’”
For those playing the game at the Mary Rose Museum in Portsmouth, UK, where visitors can see the original ship, artifacts and some of the skeletons raised from the sea in 1982, they also have the chance to wear a mobile scent backpack. The app-based game on each player’s phone connects to their backpack to trigger scent via bluetooth at certain points in the game to further immerse players in that moment of the storytelling.
Emanuela Maggioni is founder of OW: Smell Made Digital, which developed the scent devices.
“Our collaboration with Picture This Productions on the Mary Rose project has been extremely rewarding,” says Maggioni. “As a result, we have both designed and built our first wearable scent-delivery device prototype. As a fully realistic and intuitive part of the augmented reality experience, the scents emitted from each player’s device help to create a significantly deeper immersion in the world of the Tudors. Building on leading scientific research on olfaction and memory, our scent device helps weave the lives and stories of the Mary Rose into the memories of each user, long after the experience itself.”
In Time Detectives, players use their own phone as a magical spyglass to reveal secrets from the past, collecting and interrogating clues, and eavesdropping on conversations to solve the mystery and complete their mission for the King. And there may be more mysteries to come.
“For now, there is just the Mary Rose mystery to solve, but we plan to add more as we feel this uniquely immersive way of bringing history to life has almost limitless potential,” says Mikkelborg. “I, for one, would love to do an experience in the Roman Forum, perhaps investigating the murder of Julius Caesar.”
Time Detectives is available on the App Store for iOS and the Google Play Store for Android.