Düsseldorf (Germany), 25th November 2020 – Recursive has a long history of developing audio- and video-based hardware and software-focussed design solutions for a number of high-profile brands around the globe, so when a high-end fashion client decided to create a fully immersive, projection-based, real-time rendered experience at its store in Shenzhen, China; Recursive (alongside Green Room) was an obvious choice.

The client, which had seen Recursive’s previous work on the Puma Fifth Avenue store in New York, wanted a company to create a multimedia experience for its customers, which would take them on ‘an enchanted journey of discovery’ through a magical and ethereal landscape with meaningful real-time interactions between the virtual and physical worlds.

CEO of Recursive, David Yates, explained: “We were tasked with developing the system architecture and interpreting the creative intent into a physical solution that could allow four different worlds based on four core themes across the natural world, to be experienced by users, triggered automatically from social interaction and adapting each time to each user’s unique movements within the space.

“We identified early on that we would need to work with a real-time rendering environment and that this would require a method to adapt the content to the shape and layout of the physical room, which is both 360 degrees in coverage and complex in shape, with both convex and concave surfaces.”

The team also identified that the final resolution and canvas size would be exceptionally large, making use of 6 x 4K 21,000 lumen laser projectors across the walls and floor. The real-time nature of this rendering meant that the load would need to be split over several machines that acted together as one, communicating over the network with each other.

This led Recursive to Unreal Engine’s nDisplay system, as it not only renders the environment, but distributes the workload between several workstations in a way that is quick to set up and easy to manage.

The team also made use of VIOSO technology to scan the space and generate the necessary files, in the required formats, to help the Unreal system understand the layout and shape of the nDisplay viewports.

“The projection surface we had to work within the given space is profoundly irregular,” said audio visual consultant, Paul Marshall. “We needed a reliable and flexible projection mapping solution that enables complex projection blending and is capable of synchronising multiple workstations in real time. We used multiple cameras to scan the space and then went through a conversion process to arrive at a single unified MPDCI output.”

The MPCDI files were then brought into nDisplay, then used within the nDisplay configuration in order to assign the outputs to the projectors that were attached to each nDisplay node.

“Outside of the core requirement for real time rendering, there was also a need for other physical elements of the installation to communicate,” added Richard Smith, associate director at Recursive. “Again, nDisplay allows this via Unreal’s blueprint through many network-based protocols such as OSC and UDP thus enabling us to control dynamic lighting, Nano winches, Hazers and audio systems in real time.”

Yates added: “We’ve worked with VIOSO before and knew that they would be our go-to team to work through the issues in mapping this space, as well as providing an output in a format that could be used by a real-time rendering application. VIOSO has an excellent reputation in this field, and after the first meetings, we knew that once again we had the right collaborators for the job.”

The VIOSO system offered the team automated multi-camera, multi-projector calibration, which allowed them to generate the necessary files for the nDisplay configuration.

“Through discussions with the VIOSO team, we arrived at the principle of using the VIOSO software to both scan the space, align this to the 3D model space and output this dataset in an MPCDI v2 format which could then be used by nDisplay,” said Smith. “Given that nDisplay was also essential for the real-time rendering aspects of the project, we paid great attention to the only other instance of this combination of approaches being used together; which was for the Childish Gambino for his Pharos concert series.”

The team achieved the required level of tracking and control through developing their own in-house system based on multiple depth cameras. The system makes use of a point cloud approach to be able to view the user from multiple angles through interpreting the depth data in real time.

“We also needed to be able to use skeletal tracking to drive real time graphical objects within the Unreal environment,” said content production manager George Roboz. “We used the Microsoft Azure Kinect, which translates the user’s skeletal movements into data via Touch Designer. This then drives the nDisplay engine via Unreal’s blueprint system. Once the invited customer enters the room, they drive complex particle systems, deform objects, interact with flora and fauna by reaching out and changing their position in the physical space, while a magical story unfolds around them.”

The real-time interactions across both the virtual and physical makes this space unique both in the user experience, but also in the design and implementation challenges.

“There were lots of stakeholders in this process around the globe, all working on a separate aspect of the project,” explained Marshall. “We had to plan far ahead and support their work; therefore, testing was crucial. The content was developed in the US and the final room was built within the store in Shenzhen with the design and client teams based in the UK.”

Due to the complexities of the setup and the need to ensure the designs would work in reality, the team created a full-sized mock-up space in the UK, where they could test and optimise the workflow and prepare for potential pitfalls.

“We made real friendships as we worked together and sorted out every single technical challenge,” Yates said. “We spent lots of sleepless nights in our home offices, often collaborating with teams all over the globe, but the end product is worth it. We were always blown away by the VIOSO team’s expertise and their willingness to add features to their own software as it was needed. We highly recommend them and can’t wait for the next project when we can work with them again.”