2022 the 14th International Conference on Computer and Automation Engineering

Graphics Processing in Virtual Production

2022 14th International Conference on Computer and Automation Engineering (ICCAE) | 978-1-6654-8380-3/22/$31.00 ©2022 IEEE | DOI: 10.1109/ICCAE55086.2022.9762415

Tony Oakden Manolya Kavakli

Academy of Interactive Entertainment AIE Institute
Canberra, Australia Sydney, Australia
tony.oakden@aie.edu.au manolya.kavakli@aieinstitute.edu.au

Abstract— Real-time rendering techniques, developed for com- scenes at their disposal, directors can rapidly iterate and
puter games, offer great opportunities in Virtual Production. Ray experiment with different ideas.
Tracing has been used for CGI movies for many years but it is only
recently that its application in real-time has become practical. This The hardware and software required to render such scenes
is partly due to improved algorithms but mostly advanced hard- was until recently cutting edge and only simple scenes could be
ware such as the Nvidia Geforce RTX 3000 series of cards which rendered in real time or such scenes had to be pre-rendered as
provide hardware support for real-time lighting thus improving movie. With new technology and software, relatively complex
the quality of the rendered images in CGI. We conducted a series scenes can be rendered in real time. Our goal in this paper is to
of tests for rendering of a Virtual Production scene in Unreal game investigate if RTX architecture is crucial for Virtual Production
engine. Images are rendered in 4K and output to a network dis- Studios, despite the additional hardware cost required.
tribution system where the image is broken down into a series of
smaller images each rendered onto LED screens. Results were There are many ways to create a VP system. [2] reviewed
plotted to show the comparison of render times between two the problems with a Virtual Production (VP) Studio technology
graphics workstations using Nvidia RTX A6000 GPU cards and and listed potential solutions for reducing the input lag in a VP
Nvidia RTX A3090 GPU. Our findings state that whilst RTX pro- Studio, especially for the benefit of small enterprises willing to
duces better image quality the gains might not be worth the addi- use VP systems.
tional hardware cost required by the high-end graphic cards. It
might also be optimal to split the rendering of the scene across
In this paper, we will first focus on graphics processing, con-
multiple computers. duct a number of tests with a set of graphics cards used for VP,
discuss our test results, and finally evaluate the performance of
Keywords—graphics processing, virtual production, graphic graphics cards in VP.
cards, performance analysis
II. GRAPHICS PROCESSING
I. INTRODUCTION Ray Tracing has been used for CGI movies for many years
The use of real-time rendering techniques, developed for but it is only recently that its application in real-time has become
computer games, offers great opportunities in film production. practical. This is partly due to improved software, but advances
Modern rendering systems can produce photorealistic scenes in in hardware, such as the Nvidia Geforce RTX 3000 series of
real-time and those can be projected onto screens behind the cards, have provided hardware support for real-time lighting
actors. Virtual Production (VP) integrates virtual and augmented thus improving the quality of the rendered images in CGI when
reality technologies with CGI and VFX using a game engine to compared to traditional real-time rendering techniques [3]. This
enable on set production crews to capture and unwrap scenes in technology has also simplified the processes required to light a
real time [1]. This offers significant advantages over the real-time scene. In traditional rasterized images, lightmaps and
traditional green screen production methods as listed below: light probes must be created and placed by artists. With RT art-
ists are instead free to focus on llighting photorealistic scenes in
First of all, the director can see in real time how the final much the same way as lighting engineers in a traditional film
scene will look without having to wait for post-production. The production.
position of the physical camera can be synchronised with the
camera in the scene in real-time so that as the camera moves the These graphics cards are powered by Ampere, NVIDIA’s
image projected behind the actors updates appropriately. The 2nd gen RTX architecture [4], with new RT Cores, Tensor Cores,
LED screens provide illumination which when used and streaming multiprocessors for the most realistic ray-traced
appropriately provides realistic lighting. Costumes, props and graphics and cutting-edge AI features, promise to revolutionise
other surfaces which are in the scene with the actors pick up real-time rendering over the coming decade. It is expected that
reflections from the screens thus greatly improving the sense RTX architecture [5] will largely replace the current rendering
that these items are in the environment. Artists can update the techniques for all mid to high range hardware applications. It is
3D scene in response to the directors requests in real time whilst likely to become the standard to light the scenes, particularly
the actors are on stage, thus allowing the directory, relevant to virtual production applications [6] where super
cinematographer and lighting engineers to rapidly iterate quality real-time images are essential for believable integration
towards aesthetically pleasing results. Scenes can be animated of live actors and CGI.
and through the use of AI and physics systems, items rendered A. RTX Architecture
can react to the actors movements. With a large library of 3D
RTX architecture (The NVIDIA Turing™ architecture)
This work has been funded by AIE Institute, Sydney, Australia. combined with GeForce RTX™ platform fuses together real-

978-1-6654-8380-3/22/$31.00 ©2022 IEEE 61

Authorized licensed use limited to: PUC MG. Downloaded on July 04,2023 at 18:11:16 UTC from IEEE Xplore. Restrictions apply.
time ray tracing, artificial intelligence, and programmable The hardware components are as follows:
shading. The hardware review includes specifications, features,
and other performance enhancements, as well as comparisons to A. Graphics Workstation1
the other graphics cards for gaming. The graphics processing in The AMD Ryzen CPU is a popular choice for gamers with
RTX architecture allows for significantly more transistors the 5950X a significantly more powerful processor than the
packed into a smaller area than before, using the second more popular 5600 series of CPU frequently used in gaming
generation of consumer ray tracing and third generation deep rigs. The machine has two Nvidia RTX A6000 GPU cards [9]
learning hardware. The process of generating RTX is simpler connected using Nvidia proprietary NVlink bridge [10] which
than lighting using traditional real-time techniques, but there are allows the cards to share the rendering load. This is very much
some problems with the process for which there are currently no a high end card and very rarely found in home gaming rigs even
good solutions. One of these is that the system requires less so with a NVLink configuration.
placement of light probes in the scene which are used in real- Each GTX6000X has
time by the GPU hardware for quantization of the scene. • 48 Gigabytes of GDDR6 memory
B. Light Probes • 10,752 Ampere architecture based CUDA cores
Light Probes provide a way to capture and use information • 84 RT (Ray tracing cores)
about light that is passing through the empty space in your scene • 336 Tensor Cores
[7]. Similar to lightmaps, light probes store “baked” information • Memory bandwidth 768 GB/s
about lighting in a scene. The difference is that while lightmaps B. Graphics Workstation2
store lighting information about light hitting the surfaces in your
scene, light probes store information about light passing through The AMD Ryzen 5900X is a slightly lower spec than the
empty space in your scene. Light probe spacing is critical -when 5900X but in speed tests if very comparable in performance. The
it is too close, frame rates suffer, and when it is too low, visual machine has one Nvidia RTX A3090 GPU. The 3000 series of
quality suffers. The ideal placement of probes is very dependent GPU are frequently found in mid to high range gaming rigs and
on density of geometry and lights. The RTX system Nvidia uses the 3090 is currently highly regarded.
is able to make adjustments to probe positions in real-time but Each GTX3090 [11] has
the heuristics used to make the adjustments are primitive and • 24 Gigabytes of GDDR6X memory
cannot guarantee image quality. • 10,496 Ampere architecture based CUDA cores
III. EXPERIMENTAL PROCESS • 82 RT (Ray tracing cores)
• 328 Tensor Cores
We conducted a series of tests for rendering of a VP scene • Base Clock 1.4 GHz
produced using a computer running the Unreal game engine.
Images are rendered in 4K and output to a network distribution C. Scenes
system where the image is broken down into a series of smaller Data has been collected from the above hardware
images each rendered onto LED screens which are made up configuration using three different test scenes and a variety of
from a number of subpanels. Unreal provides many options to different configurations. In particular we were interested to see
allow for very high quality scenes to be rendered. if machine 1 offered a significant advantage over computer 2 in
Two machines using two different sets of graphics cards terms of frame rate. Several scenes where created in Unreal for
have been tested for use in our system. Both cards have very testing use.
similar architecture. The CUDA and RT cores are most relevant
for graphics processing, the tensor cores are mostly used for
physical simulation or machine learning applications [8]. In the
graphics pipeline the tensor cores can be used to denoise of the
image and some other post processing effects, which can signif-
icantly improve the quality of the final image particularly in real
time raytracing applications.
The key feature of the GTX range of cards is the inclusion
of the RT cores which are designed to accelerate ray tracing to
the point where it can be used in real time for relatively complex
scenes. Both machines run Windows 10 with Unreal 4. The
principle configurations tested where:
• Graphics Workstation 1 – NVLink enabled Fig. 1. Complex scene with complex lighting
• Graphics Workstation 1 – NVLink Disabled
• Graphics Workstation 2
For every combination above tests were performed with the The scenes used for testing and the mesh counts are demon-
following lighting: strated in the Fig. 1-4. Mesh counts, tri counts and lights are
• Real time ray tracing listed under each figure. Complex Scene with complex lighting
• Rasterization is demonstrated in Fig. 1, and with Simple lighting in Fig. 2.

62
Authorized licensed use limited to: PUC MG. Downloaded on July 04,2023 at 18:11:16 UTC from IEEE Xplore. Restrictions apply.
 Mesh Count: 1,878 • Dynamic Lights: 0
• Tri-Count: 49,556,769 Tris • Dynamic Lights (No Shadows): 0
• Dynamic Lights: 1 • Stationary Lights: 2
• Dynamic Lights (No Shadows): 15 • Static Lights: 3
• Stationary Lights: 9 • Mesh Count: 744
• Static Lights: 6 • Tri-Count: 21,612,460 Tris
• Dynamic Lights: 0
• Dynamic Lights (No Shadows): 0
• Stationary Lights: 2
• Static Lights: 3
IV. RESULTS
The tests were run without the physical camera being used
but some movement was simulated using a script which simply
rotates the camera through 360 degrees and moves it along a
track. This provides some degree of averaging across different
scene complexity. In Fig.5, results were plotted to show the
comparison of render times in similar hardware and software
configurations. For all graphs values are in frames per second
Fig. 2. Complex scene simple lighting (FPS). As is expected Ray tracing is the slowest to render and
the simple scene with simple lighting the fastest. However, there
• Mesh Count: 1,878 are some interesting aspects to note here:
• Tri-Count: 49,556,769 Tris
The first is that whilst the 2 A6000 running on the NVLink
• Dynamic Lights: 0
is the fastest hardware it is not significantly so. There have been
• Dynamic Lights (No Shadows): 0
several articles written about relative performance of these cards
• Stationary Lights: 2
and it has been noted that the speed improvement that the link
• Static Lights: 3
brings is subject to a variety of performance constraints. In
particular, performance gains are quite application dependant.
The hardware is still relatively new and so further optimisations
are probably possible to better utilise it.
The second observation is that without the link, when only
one A6000 is running, performance is worse on Graphics work-
station 1 than it is on Graphics workstation 2. This is a surprise
because Graphics workstation 2 is inferior to Graphics work-
station 1 in every way. Again performance comparisons of the
GTX 3090 and the GTX A6000 suggest that the former is
superior for gaming purposes and because we are using Unreal
as the tool for rendering the scenes, and unreal is primarily
aimed at the games market it may be that it is heavily optimised
Fig. 3. Simple scene for the GTX 3090.

Comparison of display update

(average FPS)
60.00
50.00
40.00
30.00
20.00
10.00
0.00
Ray Ray Ray Raster Raster Raster
trace trace Trace simple simple
Fig. 4. Overlapped transparency simple simple lights scene
lights scene
Simple Scene is demonstrated in Fig. 3, and Overlapped
transparency in Fig. 4. A6000 SLI A6000 3090
• Mesh Count: 730
• Tri-Count: 21,607,888 Tris Fig. 5. Comparison of frame rates

63
Authorized licensed use limited to: PUC MG. Downloaded on July 04,2023 at 18:11:16 UTC from IEEE Xplore. Restrictions apply.
 Finally we note that Ray Tracing, even on the top end GTX We had hoped to assess the relative quality of the ray traced
cards, is significantly slower in all configurations that the and rasterised versions of the scenes. Our intention was to get
equivalent scenes rendered with raster. In fact the simple scene experimental subjects to view the output and to rate the quality.
with simple light (one light) takes almost as long to render as the Unfortunately, we have not had time do this yet. The question
most complex scene using rasterization. This indicates that is: “is the extra quality from RTX worth the additional expense
although the hardware is extremely capable the workload of the hardware required for virtual production applications?”
involved in real-time ray tracing is still very high. This may be To do this we need to create some short movies using both RTX
acceptable if the quality of rendered output is substantially better and raster and a variety of hardware and use human subjects to
however, informal tests indicate that in a side by side rate them.
comparison most users struggled to tell the difference between
real-time ray traced images and rasterized ones. Of course this ACKNOWLEDGMENT
is at least in part due to the effort put into making the rasterized We are grateful to Andy Marriott, Managing Director of Sil-
scene look acceptable (baking lights, light probes etc) but still, ver Sun Pictures for giving us permission to conduct testing, and
it is an issue which needs to be considered. Lachlan Emanuel for running the tests and collecting the data.
V. CONCLUSION REFERENCES
In this paper, after a review of graphics processing in VP, we [1] Kavakli, M, Cremona, C., 2022: The Virtual Production Studio Concept
looked in more detail at the hardware used for rendering the – A Game Changer in Filmmaking, IEEE VR 2022: the 29th IEEE
scenes. Experiments were carried out to test the different Conference on Virtual Reality and 3D User Interfaces, 12-16 March, 2022,
Virtual, p.1-10
hardware configurations using a variety of different scenes. The
[2] Kavakli, M, 2022: Requirements for reducing the input lag in a Virtual
results were collated and presented graphically. The high-end Production Studio, HCI INTERNATIONAL 2022, 24TH
hardware did not offer the performance gains we had expected INTERNATIONAL CONFERENCE ON HUMAN-COMPUTER
over the lower end hardware and further research is needed to INTERACTION, 26 June - 1 July 2022, Gothenburg, Sweden
understand why. RXT was shown to be significantly slower, [3] NVIDIA, 2021a. GEFORCE RTX 3080 FAMILY THE ULTIMATE
even on the top end hardware, than rasterization. PLAY, last accessed on 29 Oct 2021, https://www.nvidia.com/en-
au/geforce/graphics-cards/30-series/rtx-3080-3080ti/
In summary we conclude that whilst RTX produces better [4] NVIDIA, 2021b. RTX Global Illumination (RTXGI), last accessed on 29
image quality the gains might not be worth the additional Oct 2021, https://developer.nvidia.com/rtxgi
hardware cost required. It might also be optimal to split the [5] NVIDIA, 2021c. NVIDIA RTX™ platform, last accessed on 29 Oct 2021,
rendering of the scene across multiple computers where the https://developer.nvidia.com/rtx
images destined for the monitors which do not appear in the [6] Unreal, 2021. Storytelling reimagined, last accessed on 29 Oct 2021,
camera frustum are rendered by lower spec computers not using https://www.unrealengine.com/en-US/solutions/film-television
rasterization and a lower resolution but the main screen is [7] Unity, 2020. Unity User Manual 2020.3, last accessed on 28 Oct 2021
rendered by higher end hardware. https://docs.unity3d.com/Manual/LightProbes.html
[8] What-Are-Tensor-Cores-Anyway. (2021, 4 19). Retrieved from Titan
With this paper, we have begun to analyse the data that has Computers: https://www.titancomputers.com/What-Are-Tensor-Cores-
been obtained. Several things need more investigation in future: Anyway-s/1204.htm
[9] a6000-datasheet-us-nvidia. (n.d.). Retrieved from Nvidia.com:
The complexity of the scene test may not be sufficient to https://www.nvidia.com/content/dam/en-zz/Solutions/design-
adequately stress the hardware used in the high-end machine. visualization/quadro-product-literature/proviz-print-nvidia-rtx-a6000-
We intend to create a series of more complex scenes and run datasheet-us-nvidia-1454980-r9-web%20(1).pdf
similar tests to the ones in this paper comparing the results. [10] GLAWION, A. (2021, august). NVLink vs. SLI and Multiple GPUs – Is
it worth it? Retrieved from CGdirector: https://www.gpumag.com/nvlink-
It is interesting that the 3090 performs better than the A6000 sli-difference/
in most of the tests and we intend to investigate further. It may [11] GeForce RTX 3090. (n.d.). Retrieved from tech powerup:
be that the hardware was not configured correctly or some other https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622
problem. A further test would be to physically switch the cards
over and run the tests on the same computer thus reducing the
effect of other hardware (CPU or the other card etc.)

64
Authorized licensed use limited to: PUC MG. Downloaded on July 04,2023 at 18:11:16 UTC from IEEE Xplore. Restrictions apply.