An Introduction to Modern VFX Technology
Virtual production (VP) is a cutting-edge filmmaking technique that combines live-action filming with computer-generated imagery (CGI) and real-time visual effects. It’s not new, but the technology has grown leaps and bounds in recent years with excitement from notable projects like The Mandalorian and Star Trek: Discovery, and movies like Avatar: The Way of Water, Thor: Love and Thunder, and The Batman all using the technology. But its use goes beyond larger-than-life sci-fi productions. In 2022, the worldwide market size for virtual production stood at $1.82 billion USD. Projections indicate that between 2023 and 2030, the market will experience a compound annual growth rate (CAGR) of 18.2%.
What’s fueling this exponential demand and adoption?
VP allows filmmakers to blend physical and digital worlds, enabling them to visualize and manipulate virtual environments and characters while capturing live-action footage. This approach has revolutionized the way films, TV shows, and other media content are produced, as it offers greater creative control, cost-efficiency, and flexibility.
Methods of Virtual Production
These are the primary methods used by filmmakers to harness the power of virtual production.
Previsualization (Pre-viz)
Quickly and efficiently developing digital representations of scenes before the actual filming facilitates better project planning for camera angles, lighting, and other elements. One of the biggest benefits of previsualization is the fast pace and low cost of iteration.
Pre-viz has long been a useful tool with legacy software, but it has really grown since the industry started using game engines like Unreal and Nvidia’s Omniverse to speed it up, make it cheaper, plus make it easier to iterate live with a director.
In-Camera Visual Effects
Virtual production enables filmmakers to visualize and adjust visual effects in real-time. In-camera visual effects (ICVFX) is a technique that captures effects during filming using LED screens, providing flexibility and realism during the visual effects process.
ICVFX saves filmmakers time and money by eliminating guesswork, and on-set virtual production (OSVP) allows them to see finished visual effects through the viewfinder or monitor. This real-time technology offers a more efficient and authentic alternative to traditional green screens and post-production methods.
The development of realistic digital environments vs. costly physical sets or location shoots saves money. This provides more flexibility and adaptability in the production process. In recent years, the significant change in VP is the replacement of green screens with large-scale LED video walls to show the environment on stage, often with 360-degree video content. Game engine technology has also made creating and displaying virtual environments cheaper and more accessible to smaller productions.
Orca Studio uses video walls in its studio to results like these:
To get a closer look at how video walls are used together with in-camera effects (ICVFX), you can watch Epic Games giving a demo of “The Volume”—its Virtual Set technology:
All this allows the environment to react realistically to the movement of the camera and reflect real-world lighting onto the actors, enhancing the overall production quality.
Real-Time Performance Capture
This technology records actors’ movements and facial expressions on a motion-capture stage. The captured data is then translated into digital characters and integrated into the virtual environment, resulting in more immersive storytelling. Performance capture as applied to virtual production really shows its strength when applied to the camera. The director of photography (DP) can have a camera rig that shows the actor’s live performance applied to digital characters in a digital world, right in the viewfinder.
This is an alternative to a full-fledged immersive studio wall set – the actor doesn’t get as much immersion in the digital environment, but the DP and director can see everything happening live in the game engine, allowing for real-time adjustments to the environment, performance, lighting, etc. This ultimately reduces post-production costs.
Here’s how ftrack customer Imaginarium Studios helped Netflix use performance capture to create a one-of-a-kind program with dancing monsters.
The Benefits of Virtual Production
As virtual production reshapes how media is made, it brings some enormous advantages for filmmakers and other creatives. Here are the key benefits of adopting this method of production:
- Cost-efficiency: Instead of spending a large budget on building physical sets or traveling to distant locations, filmmakers can create realistic digital environments, reducing overall production costs.
- Greater creative control: Directors can visualize and manipulate virtual environments and characters in real-time, allowing them to experiment with camera angles, lighting, and other elements, resulting in a more refined final product.
- Flexibility and adaptability: Virtual sets can be easily modified or repurposed for different scenes, providing filmmakers with the flexibility to make changes without incurring significant additional costs or delays.
- Enhanced collaboration: VP brings together artists, animators, and live-action teams, fostering cross-disciplinary collaboration and leading to innovative storytelling and visual effects.
- Real-time decision-making: Filmmakers can make creative decisions on the fly, as they can see how the final scene will look while shooting, which helps to avoid costly reshoots and keeps the project on track.
- Environmentally friendly: By reducing the need for physical sets and location shooting, VP minimizes material waste and the environmental impact associated with traditional filmmaking practices.
You can see, there are a lot of wins with this approach. Now let’s compare VP to traditional production
Virtual Vs. Traditional Production
To understand the differences between traditional and virtual production methods, we’ll delve into the distinct approaches across four stages—story development, pre-production, production, and post-production.
Story Development
Virtual Production: In virtual production, story development benefits from the early integration of visual elements and environments, allowing for a more seamless marriage of narrative and visuals. Writers and directors can explore and visualize scenes in a digital space, helping to refine the story and its presentation.
Traditional Production: In traditional production, story development often relies exclusively on scripts and storyboards, with visual elements and environments being explored separately. The integration of narrative and visuals may not be as seamless which makes it difficult for a director to envision how the two will come together. This can lead to costly changes later in the production process.
Pre-production
Virtual Production: Pre-production involves pre-visualization, where filmmakers create digital representations of scenes to plan camera angles, lighting, and other elements. This allows for better planning, creative iteration, and a more precise vision of the final scene—especially when it comes to digital asset creation and environment design.
In the past, pre-visualization was considered a luxury for many production teams because of its high costs and limited accessibility. However, thanks to modern real-time engines, pre-viz has evolved into a more viable and efficient option for filmmakers and production studios. These advanced technologies have made it much faster and easier to translate pre-viz concepts into actual production, enabling teams to make rapid adjustments and review scenes without the need for extensive rendering.
Traditional Production: In traditional filmmaking, pre-production typically relies on storyboarding, location scouting, and set construction. This can be time-consuming and may not provide the same level of flexibility and control as virtual production techniques. While pre-viz has been used here, previous software options have always come at a greater expense of time and crew.
Production
Virtual Production: During production, virtual sets and environments enable efficient shooting with in-camera effects that offer live previews. The movement of the camera can dynamically alter the positioning of the environment so that the camera and video walls work seamlessly together. This allows directors to see how the final scene will look as they shoot, enabling better creative decisions and reducing the need for reshoots. This flexibility allows for significant changes to the story throughout the process because the production is not locked into a certain set of shots or rigid environment. The ability to change scenes, lighting, and weather opens more opportunities for scheduling film shoots to better leverage the time of talent.
Traditional Production: In traditional filmmaking, production involves shooting on physical sets, on-location, or in front of a green screen, with visual effects added during post-production. While blue and green screens can eliminate some logistical issues, they demand a lot of painstaking post-production work. This and the use of physical sets may result in additional reshoots or adjustments if the desired outcome is not achieved during the initial shoot. Any changes in the story after production has begun will require much more time and resources and demand more consideration for continuity as things like lighting and weather change.
Post-production
Virtual Production: Post-production in VP is streamlined, as real-time visual effects and CGI integration have already been addressed during pre-production and production. This reduces the workload in post-production, but also adds time to pre-production. One challenge this presents is that producers need to be more proactive in order to prevent issues, effectively fixing things in pre-production instead of post. This also allows for more creative exploration. Late-stage editorial changes or reshoots are much easier and cheaper for VP.
Traditional Production: In traditional filmmaking, post-production involves the integration of visual effects, CGI elements, and color grading after the shooting process. If green screens have been used, those shots will need rotoscoping and compositing as well. All this is time-consuming and requires extensive coordination between different departments.