Artificial Intelligence (AI) has captivated the imagination of many, with visions of hyper-realistic digital creations and fully automated movie productions. However, the reality of AI's capabilities in the visual effects (VFX) industry is much more nuanced. Jory Federighi, a founding member of VFX studio Spruce, recently joined us for an exciting live talk, sharing a comprehensive overview of how AI is being used in VFX today, highlighting its practical applications and limitations.
Contrary to popular belief, generative AI is not yet a dominant force in VFX production. Jory notes that more and more customers are posing the question “Can’t we just use AI?”. He answers yes - and no. What AI can not yet do is produce complete films or even short films. Or at least not sophisticated ones. To share Jory’s example, AI is not yet capable of producing a fully photorealistic short film, such as a 2.5-minute scene of King Kong shaving his body bald. By the way, if anyone does have a copy of that video, send it to Jory, he’d like to see it.
Jory clarified that while AI tools are integrated into their workflows, they do not handle the bulk of creative tasks. “We actually use very little generative AI creative solutions in our typical pipeline” he noted. In fact some of the areas AI is making the largest impact are actually some of the most subtle use cases.
Where AI is truly shining in VFX today is in tedious, time consuming tasks that would be impossible, or highly impractical to accomplish by hand. Jory covers three cases where AI is used in VFX:
One of the more tedious tasks in VFX is creating realistic textures for 3D environments. In a project for Spotify, Spruce needed to create a "painterly town" texture for a green screen session. Traditionally, this would involve sourcing or creating materials manually, a time-consuming process that might even require physically creating textures, involving literally watching paint dry. However, AI tools like diffuse maps, normal maps, roughness maps, and displacement maps allowed Spruce to generate detailed textures based on specific references quickly. This capability enabled them to iterate on materials and compositions efficiently, drastically cutting down the time required for this task.
.png)
In another project, Spruce had to create a realistic record vault as a setting for a video. In order to avoid picturing real-world records, they needed to generate hundreds of fake album coversThis required generating hundreds of fake album covers. In a video that includes thousands of these prop records, there’s no accounting for how much time it would have taken to design album art for each individual one. However, the Spruce team was able to use AI Image Generator, Midjourney, to create unique album covers for every single record, as well as other elements of the design, like an assortment of faded stickers on a milk crate. As Jory flips through some of the thousands of fake albums, he notes some of the oddities one might notice if they look close. Anyone who’s used Midjourney might recognize some of the surrealness of AI generated images- for example, in the case of the image below (right), a dog connected to another dog, or the tandem saddle. Taking the average of its data set, AI image generation doesn’t always get it 100% right, but at-a-glance, it’s often normal enough - making it perfect for a use-case like this one.
.png)
One of the most impactful, and perhaps unglamorous areas where AI has entered the world of VFX is in the time it takes to render animations like the record vault video from the previous example. Jory presents a tool called OpenImage Denoise, which uses AI to significantly decrease the amount of time it takes 3D modeling software Blender to render a video. Take for example a single still from the record vault project. This still image, which might normally take 9 minutes and 45 seconds to render, can be completed in just 55 seconds with OpenImage Denoise. Over the course of the total project of 490 frames, this is an enormous difference, bringing the total rendering time from 200 days down to 18. Again, AI is being used here to accomplish something so time consuming and impractical for a human that it borders on the impossible.
AI: a Practical Solution for Impractical Tasks
While AI is not yet at the point where it can produce complete, high-quality films independently, its strength lies in handling menial, monotonous tasks. This capability allows human artists to focus on creative direction and innovation. As Jory points out, “These tools are not as flashy or fun to play with but are making a huge impact on our daily workflow in VFX now.”
Where AI really shines is in the ways it is able to empower human creativity and decision-making, rather than replace it. As Jory points out, AI is yet to produce entire animations, or even entire 3D props. Jory emphasizes, “Where we really like to use generative AI [is] in situations where they’re not making creative decisions for us or determining the composition. These are the things we like to do as humans. But they really take off the pressure and let us create things that are [otherwise] very time-consuming.”
While AI may not yet fulfill the more sensational expectations set by popular imagination, its current applications are making a serious impact. By automating the tedious aspects of VFX production, AI not only streamlines workflows but also expands the boundaries of what creative teams can achieve within practical timeframes.
For a deeper dive into the presence and impact of AI in the media landscape, watch the full presentation Film and TV Operations in the Age of AI.