Academy Award-winning director Robert Zemeckis’ (Forrest Gump, Cast Away) recent film for Universal Pictures and Dreamworks,
Welcome to Marwen, stars Steve Carell, Leslie Mann, Diane Kruger, Merritt Wever and Janelle Monáe and features some unique and innovative visual effects work that is currently on the short list for an Oscar. The film’s plot is centered around the real-life story of Mark Hogancamp (Hogie), a victim of a brutal attack, and how he heals himself through his art and imagination. With all of his memories literally beaten out of him, Hogie creates a fictious Belgium village of Marwen that is populated by life-like dolls. In Marwen, he is a World War II captain and fighter pilot and can be a hero, fight his enemies, and rely on his friends. Through photography, he creates an art installation — a testament to the many women he knows who have influenced him most — while also creating a secret fantasy world for himself that comes to life for Hogie — the dolls featuring the faces of these women — as well as the audience.
Here, VFX supervisor Kevin Baillie speaks with Post.
Let’s talk about the very unique look of lead character Mark Hogancamp’s imagination sequences staring with the opening scene that, if you weren’t already familiar with what the film was about before seeing it, would have some viewers a little confused about what they were seeing. I assume that was intentional?
“It was indeed! Bob [Zemekis] wanted the audience to be eased into realizing that they were witnessing a window into Mark Hogancamp’s imagination. The same techniques that were used in the rest of the film’s imaginary scenes, but the level of ‘plasticizing’ we did to Steve Carell’s likeness as Hogie was dialed back. If you look carefully, you’ll see the over-scaled threading on his costume, doll joints almost hidden by jacket cuffs, etc. It was really quite fun to play with scale in the way that Bob encouraged us to do!”
How did director Robert Zemeckis describe to you what he was looking for with regard to the VFX? You must have been like, ‘Wow, I have to make dolls come to life!?
“Bringing the dolls to life was pretty intimidating, to be honest! Bob wanted the dolls to be the ‘heroic doll version’ of each actor, and wanted them to bear a resemblance to the actor that was just slightly caricaturized. Traditionally, that’s a one-way ticket into the uncanny valley: the zone where a digital creation looks close enough to a human where our brains expect it to be a human, but doesn’t quite get all the way there. A digital character that falls into the uncanny valley triggers off a self-protection mechanism that screams ‘that’s creepy!’ Scientists think it’s our built-in survival instinct that helps us to avoid sick people in the real world. The dolls in Marwen needed to be relatable as alter-egos of people from Mark’s real life, so it was critical that we crossed the uncanny valley, and delivered characters whose souls were alive; that were relatable as characters and conveyed every nuance of the actors’ performances despite being a plastic figure. Cue brain exploding here! It took a lot of testing to find the right recipe for bringing the dolls to the screen.”
Were the imagination sequences animation or CG?
“The imagination sequences were all digitally rendered. The town of Marwen itself was a meticulous recreation of the 1/6th scale miniature designed by production designer Stefan Dechant, built by Dave Asling’s team at Creation Consultants, and lit by our cinematographer C. Kim Miles. The doll bodies were digital, having been obsessively matched to handcrafted physical dolls and their costumes (designed by Joanna Johnston). Their movement was driven by motion capture data from the real actors, who effectively ‘puppeteered’ their dolls. The faces of the dolls were “projected” beauty-lit footage of the real actors, filmed simultaneously with the motion capture of their body performances.”
Can you talk more about how the faces of the actors were applied to the dolls?
“The doll faces were one of the biggest creative and technical innovations for us on Welcome to Marwen. When we filmed our motion capture scenes, we captured the body motion of the actors and the position and lenses of the cameras. It’s important to note that Bob and our DP were designing and operating shots like a normal live action film, using all of the traditional tools like Technocranes and Steadicams, and carefully beauty-lighting every setup. Once we had the digital scene set up in post production, the footage of the actors’ faces was projected onto a disembodied version of the actor’s head. That footage was then transferred onto the hero doll head through a process we invented for this film. We blended the eyes and mouths from the actors over the underlying digital doll renders, whose faces had been hand-keyframed to match the general performance of the actor and, finally, we ‘plasticized’ the result to make it all look like one seamless object. It’s a process unlike anything that’s been done before, so we couldn’t simply hire people off the street who had already done something like it! That’s probably what made the end result so satisfying — not only did we do something fresh and outside-the-box, but it actually worked!”
What were some of the most unique techniques you used to pull off the scenes?
“The use of beauty-lit footage of the actors’ faces to bring the dolls to life presented a ton of challenges, and required us to invent many new techniques. We had to shoot the movie both like a motion capture film and like a normal live-action movie. We needed the lighting and camera moves to be perfect on the mocap stage so that we could project the face footage onto the dolls, which meant pre-planning all of the mocap scenes before we filmed a single frame of footage. To accomplish this, we built a real-time version of Marwen that C. Kim Miles could use to design all of the lighting for the imaginary scenes. He used a custom iPad application to interactively adjust lighting for every setup we planned to shoot. Not only did that serve as ‘storyboards’ for his own lighting team, but it fed into a virtual production process that allowed us to see a real-time version of the imaginary Marwen world live on set. One set of monitors showed us what our production cameras (Arri Alexa 65s) were seeing, and the other monitors showed a beautifully-lit view into Mark’s imagination — moving dolls and all!
“Not only did this virtual production workflow help us confidently make lighting and camera decisions during the shoot, but I really believe that it helped us ‘give back’ to the other departments, and the actors, with a level of context that was really helpful. Normally, as the VFX department, we’re asking those folks for stuff all the time without really being able to succinctly articulate why. Virtual production allows us to actually show them the fruits of their labors right then and there! So, when we ask another department for something, we don’t have to explain how it'll help the movie — they can see it for themselves!
What outside vendors contributed to the visual effect work on the film?
“Atomic Fiction (now Method Studios) was the main VFX vendor on Welcome to Marwen, starting work as early as 2014, followed by a series of digital character tests in 2016 and 2017 to get the movie greenlit. Under the supervision of Seth Hill, they handled 511 of the 655 VFX shots in the film, creating the digital Marwen world and the CG doll versions of 17 characters. Framestore, led by Romain Arnoux and Christian Kaestner, handled 82 VFX shots, focused on the opening sequence of Hogie (Carell) piloting a P-40 through flak-filled skies and his first run-in with a gang of doll Nazis. Method Studios, helmed by Sean Konrad, handled 64 shots, focused primarily on digital set extensions, performance blends, Mark’s ‘drunk effect’ and turning sequences shot in Vancouver into New York City. Creation Consultants, led by Dave Asling, fabricated all miniatures, including 24 hero dolls comprising 17 characters plus backup duplicates of the leads and a set of stunt dolls used in action scenes, the town of Marwen — 14 buildings centered around a courtyard with a fountain, the P-40 aircraft and a DeLorean sports car built out of Legos. Profile Studios handled the mocap stage setup and virtual production workflow, and Day For Nite assisted with previs and postvis.”
What were some of the primary tools used?
“The primary tools used were: Maya for modeling and animation, Photoshop and Mari for texturing, Shotgun for asset and production management. Atomic Fiction used Katana and Renderman for lighting and rendering, whereas Framestore used Arnold to render. Houdini was used for FX simulations. Nuke was the compositing tool of choice for all vendors. Motion Builder and Unreal Engine were used for the virtual production Process. Many other tools were used on a smaller scale or custom-developed for the film.”
What were some of the biggest challenges?
“The biggest challenge was bringing the dolls to life. The more dolls in a scene, and the more complex the camerawork, the harder it was!
“Another challenge was mimicking Hogancamp’s ‘macro’ photographic look. A telltale signature of miniature photography is that focus goes very soft very quickly, and we wanted the imaginary world scenes to give the impression that they were seen through Mark’s lens. To achieve that ‘honest photographic’ look, we rendered almost all shots in a single pass, with depth of field computed ‘in camera’ with Pixar's Renderman. We used digital versions of optical tools, like tilt shift lenses and split diopters, to keep hero actors in focus even when they weren’t perfectly in line with one another. My favorite tech, because it’s new-school born out of old-school, was the variable diopter, which let us sculpt a curtain to vary focus distance across a frame. This allowed us to keep five dolls sitting in a u-shape around the bar, for example, all perfectly in focus while the image still fell off into softness around them.”
What scenes are you most satisfied with?
“While I’m always a fan of action scenes, I particularly love the scenes in Welcome to Marwen that featured nothing but dialogue. For me, there’s just something incredible about seeing these living dolls having a conversation with one another — relating to them not as dolls, but characters. The scene towards the beginning of the film where Hogie is trying to cheer the women of Marwen up, and toasts to them with his coffee mug, is one of these scenes. The floating dust particles in the air, the lighting of the scene, and how relatable I feel the characters are, all combine to make magic in a way that would be impossible without the hard work of many talented individuals. Mimicking reality with VFX is one thing; creating an entirely new reality is something else altogether, and is hugely satisfying!”
What attracted you to this project?
“Robert Zemeckis, plain and simple. With each film Bob does, I’ve seen him push the boundaries of what’s possible in every part of the filmmaking process. I’ve heard people try to pigeonhole him as someone who obsesses over technology, and I feel like that’s really off base. In my experience with Bob over the past 12 years, I’ve seen that he obsesses about every aspect of the films he makes — the writing, the cameras, the lighting, the acting, the set design, the selection of locations, the visual effects, the sound and the list goes on and on. He’s an artist and a collaborator in the truest sense which leads me, and I think the rest of the department heads, to feel uniquely empowered and challenged while working with him. I learn an immense amount with every film that we do together, and I have an intense amount of respect for the tenacity in which he pursues his craft. I feel like we need more filmmakers like Bob. Just think about it — he could rest on his laurels and make commercial movies until the cows come home, but is instead willing to put himself on the line and pour his talents into unusual projects that try to do new, fresh things. With him having that approach I can’t imagine anyone better to learn from, and get more creative satisfaction alongside, than Robert Zemeckis.”