Acting and Animation

Acting and Animation

There has been debate and discussion lately regarding if motion capture is pure acting.
Here are a couple of latest links to check out:

James Franco requests recognition for Andy Serkis work on RISE OF THE PLANET OF THE APES
Here's the URL if it doesn't work: http://www.deadline.com/2012/01/oscar-exclusive-james-franco-on-why-andy-serkis-deserves-credit-from-actors/

Then, what the Weta effects team did was to essentially “paint” the look of Caesar over Andy’s performance.  This is not animation as much as it’s digital  “make-up.” 

And here’s one of the responses. Be sure to check out the links to the videos by Andy Serkis and Jeffrey Engle


I’ll try to clarify a few things if I may.

Both acting and animation are very difficult to do well and require talent and artistry. Some people try to define the specifics of each one but at the end of the day they are both used to bring life to a narrative character. Whether it’s a live action person or an animated person/animal/object on screen, the audience should be able to relate to the character in some way and be provided some emotion connection.

Acting-
Acting is one of those jobs that looks very easy to those who don’t do it.
Having worked with actors, directed actors and taken both acting and improv classes, I can say with confidence it is very difficult to do well. An actor has to act and react as the character they are supposed to portray. They have to be able to become one with the character. When in that zone the actor is in the moment and the fictional world becomes their world. An actor has to put themselves out there for all to watch and that can make them feel very vulnerable. Actors are not allowed to analyze or observe their performance since that takes them out of the moment. They have to make it look effortless and to feel completely natural even though they are saying written dialog and may be doing the shot multiple times. Subtle facial expressions and body language convey more than the spoken dialog.  Casting of actors for a film is very important since different actors will bring a different take on a character.

Animation-
Animators are frequently given a bit of a brush off from the rest of the entertainment community. Much of what’s animated is aimed at children or young adults so can’t be serious according to some. But take a close look at many animation classics, even for children, and great animation does reverberate emotionally in the viewer. Look at Dumbo, Pinocchio and other Disney classics. Warner Brothers and other studios also brought engaging animated characters to life. These days Pixar and similar 3D animation studios are accomplishing the same thing.

Animators on visual effects projects must achieve a level of realism beyond what happens in most animated films. One’s not better than the other but there are differences. A visual effects animator may have to animate a horse or other animal and make them totally believable as the animal they are supposed to portray. In many cases the animation may be intercut with the footage of the real animal so the match in motion has to be spot on. Visual effects animators may also be called on to animate fantasy creatures or talking, breathing characters.

A true test for a character animator is to animate a simple flour sack drawing or model. Even without a body or face an animator can bring life to the flour sack in such a way to convey happiness, sadness, curiosity and other emotions.

Even though the term may be computer animation an actual artist is the one that does the animation. The computer is just the tool as a pencil was to pre-computer animators. Casting the right animator can be as important as casting the right actor.

Puppets
When I worked at ILM some of the modelers were also puppeteers. As puppeteers they were also in SAG since the actions of a puppet was acting. So when we did leaping laser printers or other things that required puppeting, that was under SAG agreements. I’m assuming the Muppets and any other key puppets would qualify.  In this case it’s still considered acting when moving an inanimate object with hands and rods. But there is a direct connection between the actor and the final performance.

Acting and editing
The act of editing the film of a performance can change it’s impact.
The director guides the actor’s performance on set and selects the most appropriate take of the action. Then in editing specific intercutting is used to both tell the story and to emphasis the character as desired. In the early days of film there was a classic test done where a shot of an actor with a neutral expression was intercut with various scenes. The audience response was the actor was doing a good job of  showing happiness, sadness and other emotions based solely on what it was intercut with.

Manipulation
The development of digital effects technology and artists proficient in its use allows actual manipulation of an acting performance. On STAR WARS: THE PHANTOM MENACE there were scenes that George Lucas requested be modified. An actor might have looked up in a take but the shot was run in reverse so the actor looked downward instead since that was deemed better for the cut. In some cases the scene was split and the sync of one side was slipped relative to the other. This was used to shift the timing of a reaction from one actor to another. In other cases eye blinks were added or removed as desired. Some directors have added tear drops or other modifications to a performance. The point here is that even a live action performance can be modified in post-production in an effort to create a better film experience. What was the actor’s truth on set may not be what appears in the final film. And in most cases the actors would probably not be aware of it. In the future we’ll likely see even more of this as directors and studios seek to take full advantage of the editing process. And don’t think that these types of details are beyond the scope or budget of post-production. Visual effects is already heavily used for things like removing wig netting, making adjustments to makeup (to the tune of over $1 million in some cases) and adding bruises or wounds. (That’s the real meaning of the term ‘digital makeup’.) A few actor adjustment shots will be a drop in the bucket compared to the other work already being done. It’s possible at a future time there will be debate about the implications of these types of modifications and what that means to actors.

The real process
Some actors talk or write about motion capture and 3D animation as ‘painting’ in the image. Painting? Really? That would make sense if you took a time machine back 100 years before computers and tried to explain it I suppose. But these days it makes as much sense as saying Michelangelo ‘doodled’ the David sculpture.

Short form: The performance of the actor is recorded and applied in some form to a Computer Graphic (CG) animated character.

Long form: Potentially dozens of people create a very detailed and fully articulated CG model. Imagine making a Madam Tussaunds wax figure but an order of magnitude more difficult. Character/creature designers, concept artists and animators work with the director to develop the look of the character. In some cases a physical model is sculpted out of clay for the director to review. Next it needs to be modeled head to toe, down to every key wrinkle in the computer. This is part modeling and part sculpting. A type of skeleton is built where every movable joint that is needed is included and each joint has a specific range of motion. The irises in the eyes can open and close, the eyes can move, the chest can expand when breathing, nostrils flare and the tongue is configured so that speech looks correct. In many cases muscles are built and configured to change shape. Texture artists paint every surface of the model, not with just one paint process but with multiple paint versions. One paint version shows how shiny different parts of the model are. Another is used to show dirt and still another may be used to show the subtle skin textures and tiny wrinkles.

Fur and hair may need to be added. The angle of the hair, the length of the hair and style of the hair have to be modeled. Another person may be responsible for defining where things are hard (finger nails, shells, etc) and where things are soft and pliable on the model. A shader writer writes specialized programming code to make the skin look real in different types of lighting and that it have specific translucency depending on where it is on the body. Every major facial and phonetic expression needs to be modeled to aid the animator in adjusting and expressing both emotion and voice. All this is done by a team of skilled artists and technicians who use the computer as a tool as a sculptor may use a chisel.

Every item of clothing and every prop the character handles has to be designed and built. The motion of the clothing is setup to be simulated. Does this clothing item behave like silk, cotton or canvas? Dangling earrings and the hair will be programmed so that they move in realistic manner when the character is moving.

All of that just to create the character model.

Animation Process
A good animator does act out the scene at their desk or within their mind visualizes the performance. Unlike a live action actor, the animator has to also observe and analyze every motion and expression change. And not only the position but also the timing of all of those motions need to be noted. The animator is able to visualize the changes in slow motion, forward and backwards. They have to translate that into key frame positions for the animated character model. If choice of the position of the arm or the eyebrow is wrong or if the time of those motions is wrong, the animation won’t work. Take after take is required to refine the animation. Any dialog is analyzed as well and the facial expression not only has to convey characters emotion at that 1/24 second granularity but the mouth shape, lips and tongue have to match the correct shape for the phonetic sound created in that moment.

Even actions like setting the feet down and walking take work on the computer. Activities in the real world that happen naturally take effort to do on the computer. The ground in the computer has to be built and matched to the ground of the real location and this is done by a person (match mover). If a foot goes too low then it will go into the ground and if it’s too high the foot won’t make contact with the ground.

And once the animation is done someone has to light it much like a Director of Photography does. Someone has to render the character and others have to composite (combine) the image with the original image from the set. There’s an entire team of craftsmen as big or bigger than the live action crew to make all of this happen.

Performance capture
The combination of powerful computers and digital video cameras made it possible to do computer vision. Images and motion could then be analyzed for scientific and medical purposes. The visual effects industry, as usual, looked to take advantage of these new technologies. Motion capture (MOCAP) became a way to reasonably capture 3D motion data, especially human motion, into the computer. This is useful for recording basic human motion for action shots. Facial capture has been developing which makes it possible to capture not only body motion but the entire performance. When the motion captured includes all aspects of a performance then the term ‘performance capture’ may be used instead of motion capture.

The typical system uses multiple markers that are strategically placed on the actor. A number of specialized cameras are placed around on a small to medium sized stage that is lit by subdued light. The multiple views are combined by special computer software to yield 3D information on each joint movement. In these cases only the motion was captured after the main photography had already been complete. Developments over the last few years (VAN HELSING, etc.) have allowed the capture of an actor on the stage or outside while being filmed at the same time. Additional simplified motion capture processes have also been developed (PIRATES 2, etc.) and it’s now also possible to motion capture a number of actors at the same time. These advancements have allowed motion capture actors more freedom and more interaction with fellow actors. (We still use tennis balls and other references at times. If the CG character is only 4 inches tall or is over 20 feet tall then it can be difficult for a real actor to stand in place and provide the correct interaction.)

For interactive performance capture actors are fitted with special suits and act with fellow actors who will remain in the final scenes. This interaction is of course beneficial for both the actors and the director. Another team of visual effects artists then go through and remove the performance actor by literally painting and restoring what would have been behind them. This is a very labor intensive and time consuming process since it involves hand painting frame by frame and creating imagery that isn’t in the original. The animated character or creature is then rendered and composited into the scene.

The problems
It might seem that once the motion data is captured it could simply be applied to the CG model and viola - a moving character that exactly matches the performance of the actor down to the smallest detail. But alas, such is not the case by a long shot. If it were then many of the animated films created today could be done using performance capture but they aren’t. Live action and animated movies are different art forms and what we’re seeing in some cases is a hybrid of the two. There’s still plenty of growing pains.

1. Even with improvement in the motion capture process there is quite a bit of cleanup required. A simple motion of an actor reaching out may have some frames where the arm leaps up or down a few inches. There might also be random frames where the arm goes behind the actors back or through his body. This takes a small team of people to go through and remove these glitches and clean up the data such that the performance is as pure as possible. In some cases large chunks of data may be missing which requires an animator to fill in with the appropriate motion.

2. CG characters seldom match the real actor unless it’s a digital double. Frequently the proportions are changed such that the arms or legs maybe longer or shorter. This means that the stride and interaction of what the actor was doing doesn’t match what it should. The further away this gets from the real actors body the more difficult it becomes to use the data as it is. A Satyr has totally different leg joints and needless to say a four-legged creature or caterpillar can render much of at least the body motion useless. Captured facial performance may not have much use when the facial structure is very different such as on an insect like creature. All this requires an animator skilled in the understanding of motion, performance and animation to try to retain at least a sense of what the original actors performance was providing. In some cases even performance capture data may become a basic reference or simply inspiration with the brunt of the performance created by the animator.

3. Eyes are the window to the soul and a key tool for the actor but actually capturing the eyes, eye blinks and iris changes has not happened in a meaningful way. That means it’s up to an animator to add these types of fine, but critical, details to complete the performance.

4. In the editing and visual effects process it may be determined that some adjustments may need to be made for technical reasons. Placement and timing of the character versus what the performance capture actor was doing. In some cases it may be the directors creative call to modify a performance once it has been reviewed with the rendered creature in place.

The end result is it’s likely a fair bit of performance capture undergoes some manipulation and work by animators and other artists.  In some cases it may be inappropriate to use performance capture simply because the amount of work required is large and the amount of the performance that can be retained is small. Each project has to be evaluated dependent on the creature/character and how cleanly the performance capture can be used.

So you have pure live action acting on one end of the spectrum and pure animation from scratch on the other end. And in-between you have a gray area. One step away from pure animation is to use references. For DRAGONHEART the animators used both stills and clips of Sean Connery to try to incorporate a bit of his personality into the animated performance they were creating. For a project like RANGO they filmed the actors going through the scenes and the animators used this as a reference to create their animation, trying to keep the spirit and emotion of the actors. In some cases they probably followed the actors performance very closely and in other cases they may have ended up deviating quite a bit from the recorded reference. In this film the types of characters (based on a range of animals) would make it impossible to do an exact match of an actor’s performance. It’s up to the animator to re-interpret and adapt the actors art form into an animated art form, much as a screenwriter may have adapt a novel into a screenplay. The core insight may remain the same but changes have to made to deliver it in a different form.

On some of the more extensive performance capture projects  lately only the animators who worked on the project know how much they did or didn’t do. It won’t be 100% nor will it be 0%. True judgement by those not involved directly in the animation would require a side by side comparison of the original footage next to the finished results.  (Not just a shot or two and not just a cut between the two at a specific frame.) How closely does the final performance match the nuances of the original? Did 90% of the shots just require simply cleanup? Did they end up creating or modifying the performance in the majority of the shots? Was the performance used mainly as a reference and inspiration with the animators creating the majority of the final performance? Or does the actor’s full performance shine through the majority of the shots?


Summary
1. Both acting and animation are difficult and require talent. Both bring life to narrative characters. Both are necessary for making projects of these types and both should get proper respect.

2. Animators and visual effects artists are collaborators with live action production, including acting. As such it would be nice to be acknowledged for the contribution these artists bring to a project and a realistic assessment of the amount of creativity they bring, not to mention the amount of work involved behind the scenes.

3. Performance capture is not a pure technology or art. It will likely have some mix of both an actor’s work and an animators work. What the blend is will depend on the specific project. Those reviewing the performance (critics, awards, etc) will have to determine what balance range they’re willing to consider and to provide credit where credit is due.

4. As visual effects continues to progress and as projects push into new territories there will be even more overlap and blurring of the contribution of all involved in making films. The thinking and mind sets will have to keep up with these developments.

5. Performance capture is not Digital "Make-up". As the word Performance indicates, it’s based on performance which at this point involves some balance of an actor and animator collaborating in a manner to bring the character to life.

------------------------------------------------------------------------

[Update Mar. 4, 2012: I received some questions from a reader related to this that I answered. Thought I'd add it to this post]

What is your opinion on the mainstream, out-of-industry perception of performance capture?

Most people outside of visual effects think performance capture is simply hooking up an actor to a CG puppet of sorts. That there's some type of computer program that transforms the actor directly into a cartoon or CG character.

Andy Serkis believes that 'Performance Capture is a tool, it is simply a way of recording an actors performance, no different to a camera'. What are your thoughts on this statement?

Years from now that may be closer to the truth but currently it's not close to reality. The differences are a camera records the actor directly and presents it as it was acted. The resolution, color and other things may be different but that doesn't change the actual performance. The impact of the acting can be influenced by choice of lens, camera angles and later in the edit but what you see is almost always what you get.

In the case of performance capture the raw acting that is captured has to manipulated and in some cases shoe horned in to match a much different CG character. Each one of those involves a creative and technical choice. Performance capture currently also isn't a pure process. Animators and others will have to correct for glitches in the data and oversee sections where the acting performance was lost due to technology limits. This is especially true for facial capture where animators have to adjust and fill in much of the details to get it to match a much different character face. 

Imagine a camera that drops large number of frames during the shot and that produces very fuzzy images. Since the objective is to create a continuous shot that is sharp and uses a different costume, someone else now has to act the part, dressed in the correct costume, using the original as a rough guide. In the end it's not a direct reproduction of the original performance, it is an attempt to somewhat mimic the original performance. 

And of course with a camera alone the director can't change a performance once it's been completed. But with performance capture they can and do change the performance after the fact. If in the edit the director wants a slightly different body action, facial expression or timing change then it's requested and the animation team make the change without involvement of the original actor.

How successful the original performance capture comes through the final images is dependent on many factors. Some shots may be very accurate representation and in other cases it may be very different. That's why looking at the result by themselves or just one or two comparisons of the original performance to the finished results doesn't really confirm how much is the original performance. The only true way is to see all the shots before and after.

In the context of a traditional 'motion capture' shot featuring a digital character, approximately what percentage of the actors body performance would you say remains untouched or unaltered by animators after the fact? 

As mentioned above that varies enormously. What system was used, who was cleaning up the data, how closely does the CG character match the original actor, how much hand work is required and how many changes the director chose to make after the fact. If it's a specific leap and that is being mapped on to a digital double then 75% or more may may be unaltered. (Still cleaned data but more or less matching the actor's body motion) If it's a CG character with a much different body shape (longer legs, bulkier, etc) running and leaning against a lamp post it could be the inverse of those numbers.

In the case of facial capture, approximately what percentage of the actors facial performance would you say remains untouched or unaltered by animators after the fact? 

Same basic issues as the body but facial capture is even more problematic. I would guess that just about all facial capture requires some hand work beyond just a little data cleaning. With body capture there are a limited number of key joints that need to be captured. (i.e. the knee was bent at 35 degrees on this frame). Body language is important but if you capture a basic gait of a walk cycle most people would accept that was the actor even if it wasn't a totally accurate reproduction. Motion capturing the body has also been done for a number of years now so the technology and understanding is much greater. Facial capture is a much newer process and si still being fully explored.

With facial capture you have multiple muscles all acting at the same time to subtly manipulate mouth, eyebrows and basic facial expression.

Think of the range of people's faces and personalities. Each person is the world is someone unique and identifiable by their face. Yet if all you saw of a person is their body it would be almost impossible to pin down with much accuracy. 

Consider the difference of a smirk and a smile. Very different experience for the audience and actor but very difficult to quantify. Maybe one eyebrow is ached a faction of an inch higher. Maybe the corner of the mouth is raised a little more. Maybe the speed at which it happens is slightly different. And it's the type of thing where you may have to already know the person or character to some extent to even tell the difference. Where do you place points and what do you monitor when capturing? How accurate does the CG model muscle structure actually match the actor's facial muscles? And the more different the CG character is (cartoon style, animal, etc) the more difficult this translation is to make. 

For movies we usually try to treat creatures as forms of people (anthropomorphic) and yet most creatures don't match our body form and certainly don't match our facial form. The creature mouths may not even be able to do what we want so someone has to modify the creature CG model to do that and others have to monitor and adjust the performance required to accomplish that at the level required for the specific project. 

As an audience the face of the character is one of our key focal points so it's even more critical than basic body motion. And one of the places we look is the eyes in real life and the movies. It's been said eyes are the window of the soul. Yet trying to capture true eye motion and then to apply this to a CG character is extremely difficult. In most cases animators have to take over for this so one of the key performances tools from the actor is in the hands of someone else.

Where do you think the technology is heading?

I think the technology will continue to develop to make it easier to capture more accurately with less clean up work required.

Facial capture is the area where we will see even larger changes. But just as important as the technology, if not more important, is the requirement for animators to be able to work with this and know how to get the most out of it. The other important item is those in the industry, including directors and actors, need to understand what it can provide and what it can't provide. They have to understand when it should be used, how it should be used and how to evaluate it. And all of that requires an acceptance of it as a collaboration of actor and animators. Ignoring either groups contribution to the results is not only incorrect, it's wrong.

[Update: 7-20-12  Here's an article on motion capture, animation and video games ]

Update 5-13-14  Randall Cook, the Animation supervisor for the Lord of the Rings films, discusses motion capture and Andy Serkis