November 12, 2019

Gemini Man: Building A Virtual Face

Guy Williams from Weta Digital details the creation of the face of the virtual character in Gemini Man, with comments by director Ang Lee and actor Will Smith.

Benjamin B

This is a companion post to the article about my visit to the set of Gemini Man by Ang Lee, with cinematography by Dion Beebe, ASC, ACS. The article discusses the film's exhilarating 120 fps 3D format. This post addresses the other breakthrough technology in the film, the creation of a virtual character's face. In Gemini Man, Will Smith plays an elite hit man who ends up fighting a younger clone of himself, nicknamed Junior, an amazingly convincing computer-generated virtual character. The creation of Junior was overseen by Weta Digital visual-effects supervisor Guy Williams working in collaboration with the film's visual-effects supervisor Bill Westenhofer.

+++

A Cinematic Milestone

To me, the creation of Junior marks a historic moment in cinema. This is the first time I have seen a convincing computer-created human being play a leading role in a movie. In the past, convincing virtual characters have been animals, monsters, aliens, or creatures like Gollum in The Hobbit. Past attempts at virtual human beings have been brief on-screen, and have usually suffered from the "uncanny valley": the closer they got to verisimilitude, the more the characters seemed uncomfortably unreal, as creepy as pretending that cadavers are alive.

The main obstacle to creating a convincing virtual character has been our keen human ability to read faces, to see a glimpse of an actor's soul in their face, and also to see when there's nothing behind the eyes. Watching Gemini Man, I was convinced for most of the movie that Junior was a human being -- with a few less convincing moments, like the college campus scene. It no doubt helps that Junior is intended to be a fairly stiff, laconic character. Nevertheless the achievement marks a turning point: 2019 will be remembered as the year of the virtual human actor showed his face, both in Gemini Man and The Irishman. As Ang Lee notes, the year-long process is presently way too expensive to become commonplace, but that will change...

The filmmakers involved in Gemini Man see the virtual character as a kind of mask for the human actor, a form of intricate digital makeup that extends the actor's range. In this scenario the actor is digital but the captured performance is still human, it's the human spirit in the virtual machine. However, we can all envisage a time when the virtual actor's performance could be animated by artificial intelligence. We will then face a kind of Turing test: is a virtual actor's performance -- his art -- created by a human or a computer? The coming challenge for cinema will be to keep human beings at the heart of the art.

+++

Ang Lee and Will Smith on the set of Gemini Man (courtesy of Paramount)

Ang Lee and Will Smith about Junior

Ang Lee and Will Smith gave a presentation of Gemini Man to the press at Paramount. Here are excerpts of their conversation about creating Junior.

Ang Lee: It’s not technology, it’s artistic endeavor. We want to create a new character. He’s not a robot, he’s a soulful human being.

I’m on a mission to try to discover a new aesthetic for digital cinema, which has dimension and clarity. Can you dream with clarity? That was my new task, and of course a big movie star helps.

Will Smith: One of the big differences between an actor and a movie star is that the movie star puts the movie on their back. But when you work with a director like Ang Lee, you can just be an actor, Ang takes all of that weight, it is such a relief to be able to show up to work with someone, when you know they have it.

Ang Lee: At the end of the day, it’s a heart to heart between two artists. That’s the easy part. When Will has a heart to heart he’s exposed, he’s like anyone. That is precious.

Will Smith: I was Henry for most of the movie, then for the end of the movie I was Junior. And for that you have the motion capture with the helmet and two cameras right in your face. It’s really difficult to connect, as a performer, to be able to make a movie with the 20-year old version of myself. It’s not de-aging. Junior is a 100% digital character, completely CGI. That’s part of what’s so spectacular. They’re using my performance to create the CGI character.

This will never make the actor obsolete because the source is the human heart, the human soul. You’ll never be able to get around those beautiful human mistakes and errors. But in terms of how you’ll be able to deliver and sell the depth of the emotion, it’s really exciting. I’m looking to the movie where there is only the digital version of myself. Will you be able to make a romantic comedy? Will it be that great? I think one day it certainly will.

Ang Lee: To me it’s really simple: Junior in the movie is twice as expensive as Will Smith. It’s so hard, it’s like impossible. Will Smith is easy. With Junior you’re imitating God’s work, that’s how hard it is.

I don’t see technology as just science. I see technology as the material you use for your craft, your artistic endeavor. I welcome the digital era because it’s closer to where our imagination will go. I would not worry that it’s too close to our imagination... A long time before it’s even close to our imagination.

+++

Guy Williams from Weta Digital

I spoke to Weta Digital visual-effects supervisor Guy Williams, who oversaw Junior’s creation in collaboration with visual-effects supervisor Bill Westenhofer.

Benjamin B: Will 2019 be remembered in cinema effects history for Gemini Man and The Irishman ?

Guy Williams: Yes, between those two films, this is the year that a CG human is going to perform front and center. Weta Digital did Gemini Man and ILM did The Irishman.

How many shots of Junior are there?

Williams: It’s just under half the movie. Junior’s not just an action character, he’s an equal performer in the movie. There’s a lot of acting.

Can you give us an overview of the process of creating Junior’s face?

Williams: First we did a FACS [Facial Action Coding System] half-hour shoot with eight video cameras in a 90-degree arc around [Will Smith’s] face to get a full range of expressions. FACS is a capture system based on Paul Ekman’s work cataloguing facial expressions.

So you created a virtual older Will Smith as a starting point for a younger Will Smith?

Williams: Right. We didn’t have access to 23-year-old Will Smith, but we could create a 50-year-old Will. This gave us a baseline for understanding Will Smith, because we could compare [the real and virtual Smiths] one-to-one at his current age. And then we got all the film and television references we could of 23-year-old Will to understand how to bend older Will into younger Will.

When Will is acting with Will, we could have bit the bullet and had a different actor play Will. And use that performance. But you're never going to get a better Will Smith than Will Smith. So we built the system around that idea: that the best way to get a believable Will Smith is to have Will Smith do it.

+++

Facial Puppet

What are you actually modeling in a virtual face?

Williams: We’re modeling the path of the muscles on Will’s face. We don’t try to model the facial expressions; we try to build the finite parts of those expressions with very finite muscle movements — outer brow raiser, inner brow raiser, the brow furrow, the brow lower, nostrils open or closed. We model face shapes by moving each single muscle, and end up with about 70 face shapes for each one of these individual muscles.

When you make an expression, you may fire 10 of these muscle movements. We get those as accurate as we can. Then we end up with this matrix of parts, and to get a certain facial expression, we start firing off the parts of that expression. We call this system, for animating the face, the ‘facial puppet.’

Click image for full frame view (wikipedia)

+++

Performance Capture

How does performance capture come into this?

Williams: We put dots on Will’s face and he wears a helmet with infrared LEDs and two infrared cameras pointing at him, so we can capture 3D points instead of 2D points. We pass these moving points to software that says, ‘I think that these three dots moving like this involve an outer brow-raiser muscle,’ and so on.

So the software gives us the hundreds of moving sliders in the facial puppet to replicate Will’s face. Then the animator goes in and says, ‘Hey, the software was pretty good here, but it didn’t get it right here.’

Then the nuance comes in. You can fire the same muscles to smile twice, but two different performances will have slightly different ways of, say, lips pulling sideways. So the same muscle groups are moving differently for different expressions. You keep trying to build the engine that drives the face, so that it gets easier to get to the right shapes over time.

+++

Thousands of Shapes

Williams: Now the thing that makes the whole facial system so tricky is that when you fire two parts at same time, all of a sudden they don’t look like they came at the same time, so you have to create “compo” (compensation) shapes. So when you fire these two things, you have a special compo shape so that it looks like what happens when Will does it. What started as 50 to 70 shapes now turns into a few hundred shapes. Then you have create the in-between shapes. As he raises his brow, it’s not necessarily a straight line path between the two points, so hundreds of shapes become thousands of shapes.

When you put the parts in the right order, you get something that looks like the face in that pose, but the problem is that it can be ever so slightly off. In that case, we don’t modify the final result, we go in and modify the parts — how you get there. And that was a lengthy process.

+++

Will Smith's Cousin

Was there anything that surprised you about the digital performance?

Williams: Will can smile in a hundred different ways, but there are so many more ways to make a smile. So if we don’t get one of Will’s hundred smiles, then it looks like it might be Will’s cousin or some distant relative. It kind of looks like Will, and it totally looks like a real human being, but you’re just not buying that it’s Will Smith. That turned out to be a big challenge for us.

So the uncanny valley was not so hard to overcome?

Williams: We would be able to deal with making him right off the bat 50% of the time, with a little bit of effort 80% of the time And with a lot of effort 95% of the time. It’s that last 5% of time that took an insane amount of effort.

+++

Human and virtual crying (montage of featurette frames) thefilmbook

Virtual Tears

Can you give an example of a difficult scene ?

Williams: We have a very iconic scene in the movie where Will Smith starts to cry. When you’re crying, more blood travels to parts of your face. You have to start swelling the bottom of his eyes -- it’s really subtle but they get a little bit more puffy. We also made his eyes get slowly bloodshot over the course of the scene, and go a little bit more pink. His nose gets a little bit redder, he sweats ever so slightly more and his forehead gets a little bit shiny. And we had to put the tears in. All these things we had to do well or else it would look like a fake cry.

How many Weta people were involved in building Junior?

Williams: A year in, a few hundred people had worked on it, including modelers for the face, and animators for the body team and face team.

+++

Replacing Humans?

Are we going to replace actors?

Williams: That’s like asking, ‘Are you going to replace an actor if you put them in a costume?’ This is just another tool in the wide toolkit that filmmakers have [in order] to come up with interesting new ways to tell a story.

So far you’re not creating a performance, you're mimicking it in a virtual face.

Williams: A good example is young Muhammad Ali. He’s not able to give us a performance now, so another actor or an animator would have to create a performance. The point is that some performer will create that performance, it’s not like the machines do it. Now actors spend their entire lives crafting their art. Our current school of thought, and we’re very passionate about this, is that the best person to act is an actor.

+++

Unit photography by Ben Rothstein, courtesy of Paramount Pictures.

Film frames from YouTube trailer and featurette.

+++