Buckle Up for AC Set Tour App

A Brief History of an iPhone App for American Cinematographer: Employing augmented reality, this new technique will further illustrate the collaborative creative approach of modern productions.

As a contributing writer for American Cinematographer, I often follow many different technologies more or less tangentially related to the art of motion imaging. In June of 2017, I found myself in San Jose at Apple’s Worldwide Developers Conference, learning about the latest and upcoming computing and mobile technologies the juggernauts in Cupertino had cooked up. What caught my interest was the introduction of ARKit, Apple’s new framework for developing augmented reality applications.

For the uninitiated, augmented reality (or AR) refers to the superimposing of computer-generated imagery over the real world. AR is more than just an overlay, the technique also tracks and moves the CG in perspective to match the camera view, so that the overlaid imagery looks like it’s anchored to the ground instead of just floating around like a heads-up display. The most famous recent example is the Nintendo mobile game Pokémon Go, which enables players to see and battle cartoon creatures overlaid onto the world via their device’s camera. ARKit empowers app developers to create AR apps and experiences for iPhones and iPads with a set of built-in protocols and techniques.

I could envision many applications for AR in the world of filmmaking, both regarding production for previsualization/storyboarding/visual effects planning and even in promotion by bringing audiences more directly into movie experiences. I also wondered if we could enhance the coverage AC gives to film in the magazine and online with AR. As part of my interview process on various film projects, I usually ask crew members for any drawings or diagrams they may have from the production. This often leads to complex overhead diagrams showing lighting or camera positions printed alongside the article text and production photos. 

I imagined AR could take this a step further by providing AC readers with a higher level of interactivity. Instead of a flat 2D printed diagram, a 3D set could be placed within the viewer’s perspective they could then manipulate and walk around in all directions. We could also add audio from filmmaker interviews commenting on their approach and experiences. Everything sounded possible with ARKit, but I knew it would require a lot of collaboration to bring the idea to life. 

My first step was to confab with the editors at American Cinematographer and pitch the concept. Managing editor Jon Witmer and associate publisher/web manager David Williams both immediately embraced the idea and gave me the green light to further develop it. The next step was to find a suitable project in order to create a demo version of the app and get the filmmakers involved. At the same time, I knew that to present a real set from a film project we would need to work directly with a postproduction team directly involved with a production.

This led me to The Third Floor, a preeminent visualization studio in Hollywood providing previs, postvis and virtual production for films, shows, games, location-based entertainment and more. The company’s credits include Game of Thrones, Mission Impossible: Fallout, and many of the recent Star Wars and Marvel movies, among many others. I also learned they are heavily into app development on their own for interactive content, virtual reality and a host of other production services. I pitched the concept for an AR app and met with their team, including global head of R&D Addison Bath.

They offered not only to provide a model of a virtual set they’d built for a recent movie but also to develop the app, which would enable users to experience the model along with the filmmaker audio commentary — adding an entirely new dimension to the way AC can report on productions.

Interestingly, in their R&D, The Third Floor already uses many AR and VR technologies to help with the process of filmmaking, so it was a great match. Next, we needed to choose a suitable first project. Disney’s Beauty and the Beast, which featured the showstopper “Be Our Guest” musical sequence (which earned a Visual Effects Society nomination last year in Outstanding Virtual Cinematography for team members including The Third Floor) was a perfect fit.

We worked with Marshall Weinbaum over at Disney PR to obtain the studio’s blessing to include the previs “Be Our Guest” dining room model within the demo version of the app and then I set out to interview the involved filmmakers themselves about the sequence. Typically for the magazine, I do most of my interviews over Skype or cell phone, so the audio quality is not that important because everything gets transcribed for written quotes in the articles. Since these interviews for the app would end up being listened to directly, I knew we would need higher-quality audio.

For each of the interviews, I managed to luck out with help recording the filmmakers wherever in the world away from California they were. We caught up with cinematographer Tobias Schliessler, ASC in production in the UK. The sound mixer on the film he was working on, John Hayes, was gracious enough to record Tobias’ interview in very high quality. 

Next, for theatrical lighting designers Jules Fisher and Peggy Eisenhauer in New York, I got a helping hand from my good friend Jesse Rosen, director of technical development at Abel Cine Tech. Jesse set up a recording session for Jules and Peggy to capture their voices while we spoke over Skype. Finally, we were lucky enough to get Casey Schatz, Virtual Production Supervisor for the film for The Third Floor, talking about the previs itself with his recording setup. Everyone sent their audio files over the Internet, we did some sound editing to get them down to a few minutes each, and with that, we had all of the raw material ready for assembly.

Addison then worked with the model, animation and the audio files to create the app using the Unreal graphics engine along with Apple’s Xcode development environment. For Beauty and the Beast, The Third Floor created the animation that drove the technodolly motion control crane, which is showcased in the app. We did a lot of testing on a variety of iPhones and iPads and ultimately refined the app into the version you can download today. I even got to design the app’s icon myself with input from artists at The Third Floor. 

The new Set Tour app marks an exciting new chapter for American Cinematographer. We’ve just scratched the surface of what is possible with AR interactivity in this initial version of the app. We’re hoping filmmakers and their studio and network collaborators will take an interest in this new way of exploring their art and craft so we can share their knowledge with our readers and immerse them even deeper into the process.

You can download the Set Tour app here, and we look forward to applying this new technology as we report on future productions. If you have ideas, questions or issues regarding the app — or think you have a subject that might be able to employ AR — contact us directly at [email protected].