As the Academy Software Foundation (ASWF) continues to grow to encompass new projects and technologies, an important new development this past year was the establishment of a Working Group on Machine Learning, and the first two open source projects underneath it. The latest technology revolution in the VFX and animation world is machine learning (ML), which refers to approaches where instead of programming via step-by-step instructions, we teach computers what to do by providing examples from which to discover their statistical patterns.
It's not always clear what tools we should build and use that are going to empower artists and serve productions. When it comes to ML, the constant hype of the tech industry was causing too much uncertainty and anxiety. Like the Foundation does for other open-source projects, we can take a step forward by collaborating on some ML tools that we all need, leaving more time for custom development where needed in our pipelines. In the process, we can promote a positive vision to rally behind, taking an artist-first, ethics-first approach to building ML tools for productions.
To this end, we established the ASWF Working Group on Machine Learning (WG-ML), and set up communications channels where our engineers can learn together and collaborate on projects that serve us all. Within the first few months, we started two initial open-source projects.
The first new project is the Dailies Notes Assistant (DNA). It was co-sponsored by ILM and Sony Pictures Imageworks (and now has many other collaborators), and its goal is to explore whether we can create a useful tool to aid production coordinators who are taking notes in dailies/review sessions. The goal is not to replace the human note-takers, but to help them manage the deluge of information as they are rapidly switching among multiple shots, hearing and recording feedback from creative leads, associating the right notes with each shot, summarizing the relevant instructions and communicating them to artists, and correctly filing them into existing production tracking systems. Modern voice transcription and large language model (LLM) systems are key enabling technologies.
The second project is being led by WētāFX and has the whimsical name Rongotai Model Train Club (RMTC). The goal of this project is to facilitate data set and model rights tracking as ML models are trained and deployed in a production environment. At any given time, a VFX facility may have multiple projects from different clients in flight, each of which may have different rules and requirements for what data sets may be used to train ML models, the specific ways those models are permitted to be used on the shows and reporting requirements. RMTC aims to help facilities with these important compliance management tasks in a careful way that respects the rights of IP owners.
Both projects are at the early stages, but ASWF aims to promote them as examples of how studios can continue to collaborate on building useful tools that serve the needs of production artists. We are also looking out for the next collaborative open-source project that we would like to coordinate, as well as continuing to act as a neutral forum where studios can share best practices about ML use in production.
Larry Gritz is a software architect and distinguished engineer for Sony Pictures Imageworks, as well as the Academy Software Foundation's (www.aswf.io) TAC Chair.