Imagine typing exactly where you want to shoot your film into a textbox, then instantly being transported there. Just a few years ago, this would have been impossible.
Today, however, generative AI is helping to make it a reality. Solutions like Disguise Designer’s AI workflow assistant, Ask AId3n, enable media server operators to create bespoke integration tools for virtual production without having to build them from scratch. One of these integrations could soon bring generative AI into your project, so you can transport an actor from New York to Mars at the click of a button.
For filmmakers, this opens up a whole host of possibilities. And it’s just the beginning. In the future, I believe AI will democratize the entire virtual production workflow. One of the major ways it’s already doing this is by giving filmmakers far more content flexibility, so they may no longer need to build out a full 3D world in a game engine where realtime content adjustments are needed. Advances in NeRFs and Gaussian splats, for instance, are enabling filmmakers to create virtual production-ready scenes from 2D images. That means anyone could record a video or snap photos, then capture them from any angle on an LED stage, with full 3D parallax.
AI world models could soon take this even further. World models won’t just give you the possibility of rendering an LED backdrop from a video or a prompt. They’ll actually generate an entire 3D world that can then be manipulated in realtime. If you’re on-set and the director suddenly wants some birds to fly across the screen, for instance, a world model will let you generate that with just a prompt. World models are still emerging from the research field, but we're excited to start experimenting with them soon.
Of course, there are still obstacles we need to overcome to make the most of AI, like the ethical concerns involved in using AI-generated content, and the fact that many AI video generation tools still only output 720p SDR or HD results, which won’t look good when captured by the 6K or 8K cameras on a virtual production stage.
We've started to see signs of this changing, though, with LumaAI recently adding HDR, 4K and EXR support for their video generation pipelines. New AI innovations are coming out every week to help resolve these types of obstacles. For the low-resolution issue, AI upscalers such as Nvidia’s DLSS techniques are available, which we've been experimenting with using in our RenderStream protocol, and seeing very positive results. For the ethical concerns, I think the missing piece of the puzzle is some sort of certification, such as an AI model that declares the provenance and ownership of its training data, allowing studios to certify its use for production.
This type of certification could enable IP holders to have specific models they trust with their IP, certified to be used on-set, and ensure artists are being properly compensated for their work. That way, we can all focus on what’s most important about the potential of AI in virtual production: helping anyone to tell their story to audiences, faster and in more immersive ways.
Peter Kirkup is Solutions and Innovations Director at Disguise (www.disguise.one). Disguise offers an integrated system of software, hardware and services for creating realtime virtual production environments.