The recent Fortnite World Cup Finals, staged by Epic Games at the Arthur Ashe Stadium in New York City, was a landmark event for the esports movement. Its $30M prize purse saw what is already forecast to be a $1B esports industry next year truly break through to widespread mainstream coverage. And one of the key elements to its sophisticated broadcast operation was a ground-breaking deployment of Ncam technology that allowed for a high level of augmented reality coverage rendered in realtime at 1080p60 in Unreal Engine 4.
The event made use of a two-story octagonal stage covered in more than 100 displays. A huge effort was put into streaming the three-day event to a worldwide audience, with Ncam’s realtime camera tracking technology, Ncam Reality, serving as an integral part of the broadcast operation. Also deployed at the Fortnite Summer Block Party & ProAm that took place at The Forum in Los Angeles in June, Ncam Reality enabled cameras mounted on four 35-foot cranes, one JimmyJib, and a Steadicam to provide an extraordinary amount of augmented reality coverage from the two arenas.
The integration of Ncam Reality into the Finals is the latest development in a continued partnership between Ncam and Epic Games, and showcases how the two companies’ technologies — with Ncam's latest version of Reality 2019.2 feeding camera telemetry to UE4 — enabling augmented reality or virtual graphics to multiple cameras.
“We have been working on some high-profile projects, such as developing a plug-in to work seamlessly with the UE4 engine. So, when Epic required a very special use of AR in extremely demanding environments, we were the standout partner to help deliver such a striking vision,” explains Kingsley Cook, operations director for Ncam.
It was an ambitious set up. The sheer reach of the four 35-foot cranes meant that AR content could be placed all over the arena, and the production team could blend the Fortnite-related content seamlessly into the real world space. In addition to this, a JimmyJib and Steadicam were also mounted with Ncam’s realtime tracking system to create a true rendered perspective of the virtual world from the camera's position.
Graphics were supplied by VFX specialists Nviz to create the perfect solution for the live Fortnite Tournament environment, using Unreal Engine's flexible blueprinting to develop a user interface. This, plus a set of bespoke tools that the Nviz team added throughout rehearsals, kept the tracking and the AR composition practically invisible to the broadcast team and, more importantly, it avoided slowing down the fast-moving pace of the show. They could simply pitch a finished shot when it was ready, treating the AR like a cutaway of the stadium or audience.
The combination of Ncam's rendering features and Unreal Engine's dynamic lighting allowed the Nviz team to place assets in shots in realtime, simply by looking at the final comp, allowing them to deliver hundreds of looks to the broadcast team every day.
“This speed and flexibility meant that our AR/Ncam-equipped cameras could still be called away at any time for their usual purposes while being used to shoot AR when available, without disruption to the rest of the broadcast. We were also able to equip the main cameras (35-foot cranes) with Ncam systems,” says Eolan Power, AR director at Nviz, who also acted as AR director during the events. “Essentially the AR assets would be right where they left them should they need to shoot elsewhere — if they weren’t, we could instantly snap them back using a set of built tools in our UI. This simply wouldn’t be possible without Ncam’s approach to tracking, and without our system’s speed and flexibility.”
Replays of the game action from the tournament were extracted from the engine and, using a portal effect, were rendered within the LED wall in the arena. For the broadcast they were simultaneously pulled out from the background and into the foreground, in front of the talent, to create a memorable visual effect that truly enhanced the viewing experience.
While many tracking systems would have balked at the sheer scale of the auditoriums involved, this was a challenge that the Ncam team met fairly easily.
“Both auditoriums each hold a capacity of beyond 17,000, so the volume of such a space provides challenges to most tracking systems,” he explains. “However, using our multi-sensor markerless technology, there is no limitation to the space our system can track, so it was an obstacle that was easily overcome.
A much more challenging task for the team was the ever-changing light show, which featured a large amount of different elements; a factor that Cook says they had to take very seriously.
“The live show had thousands of lights that were continuously moving and strobing, further enhanced by a laser show for the opening and closing — all of which required AR elements to be positioned at any place within the space by the drop of a hat,” he continues. “On top of this, the auditorium was continuously filled by oil-based smoke machines, with CO2 and confetti canons also being activated for key moments throughout the event.
These elements combined are enough to throw off any optically based-tracking system. Happily, Ncam's fusion technique of combining optical and inertia data together provides the sort of redundancy required to deal with these tricky issues.
“In extreme cases such as these we took redundancy a step further in order to deliver and turned to our latest release, the Ncam Xtreme camera bar — an IR version of our tracking device,” says Cook. “Rigging a handful of IR lights, provided an IR spectrum that illuminated the [stadium], allowing our optical data to hone in on that unaffected spectrum of light, so that any type of lighting state could be achieved.”
The result of Ncam’s efforts was a broadcast stream that blended the real seamlessly with the virtual, with initial estimates suggesting that over two million concurrent viewers watched 16-year-old Kyle Giersdorf (aka ‘Bugha’) win the $3M solo grand prize.
Michael Gay, Epic Games’ director of cinematic production said, "We wanted to bring Fortnite characters to life and make them part of our Fortnite World Cup coverage. We had a unique stage and venue layout that left little room for error. Ncam did an excellent job of reacting to the ever-changing challenges during a live on-site production. From an 80-foot tall Marshmello to Fortnite characters standing right next to players, the addition of AR absolutely elevated the quality of the broadcasts."
“For me, I think the size and complexity of the venues is unparalleled and I’ve personally not seen anything quite like it,” adds Ncam CEO, Nic Hatch. “The integration of real world and AR was excellent. I’m not convinced any other camera tracking technology could be successfully deployed on cranes, jibs and Steadicams, and that combination of cameras, venue size and dynamic lighting superbly demonstrates Ncam’s flexibility and robustness.”
The Fortnite World Cup Finals took place July 26-28 in New York City. Season X of the game was released on August 1 and Epic Games is promising to further develop Fortnite’s competitive scene, announcing the new Fortnite Champions Series as “just a taste of what’s to come.”