There has been debate and discussion lately regarding if motion capture is pure acting.
Here are a couple of latest links to check out:
James Franco requests recognition for Andy Serkis work on RISE OF THE PLANET OF THE APES
Here's the URL if it doesn't work: http://www.deadline.com/2012/01/oscar-exclusive-james-franco-on-why-andy-serkis-deserves-credit-from-actors/
Then, what the Weta effects team did was to essentially “paint” the look of Caesar over Andy’s performance. This is not animation as much as it’s digital “make-up.”
And here’s one of the responses. Be sure to check out the links to the videos by Andy Serkis and Jeffrey Engle
I’ll try to clarify a few things if I may.
Both acting and animation are very difficult to do well and require talent and artistry. Some people try to define the specifics of each one but at the end of the day they are both used to bring life to a narrative character. Whether it’s a live action person or an animated person/animal/object on screen, the audience should be able to relate to the character in some way and be provided some emotion connection.
Acting is one of those jobs that looks very easy to those who don’t do it.
Having worked with actors, directed actors and taken both acting and improv classes, I can say with confidence it is very difficult to do well. An actor has to act and react as the character they are supposed to portray. They have to be able to become one with the character. When in that zone the actor is in the moment and the fictional world becomes their world. An actor has to put themselves out there for all to watch and that can make them feel very vulnerable. Actors are not allowed to analyze or observe their performance since that takes them out of the moment. They have to make it look effortless and to feel completely natural even though they are saying written dialog and may be doing the shot multiple times. Subtle facial expressions and body language convey more than the spoken dialog. Casting of actors for a film is very important since different actors will bring a different take on a character.
Animators are frequently given a bit of a brush off from the rest of the entertainment community. Much of what’s animated is aimed at children or young adults so can’t be serious according to some. But take a close look at many animation classics, even for children, and great animation does reverberate emotionally in the viewer. Look at Dumbo, Pinocchio and other Disney classics. Warner Brothers and other studios also brought engaging animated characters to life. These days Pixar and similar 3D animation studios are accomplishing the same thing.
Animators on visual effects projects must achieve a level of realism beyond what happens in most animated films. One’s not better than the other but there are differences. A visual effects animator may have to animate a horse or other animal and make them totally believable as the animal they are supposed to portray. In many cases the animation may be intercut with the footage of the real animal so the match in motion has to be spot on. Visual effects animators may also be called on to animate fantasy creatures or talking, breathing characters.
A true test for a character animator is to animate a simple flour sack drawing or model. Even without a body or face an animator can bring life to the flour sack in such a way to convey happiness, sadness, curiosity and other emotions.
Even though the term may be computer animation an actual artist is the one that does the animation. The computer is just the tool as a pencil was to pre-computer animators. Casting the right animator can be as important as casting the right actor.
When I worked at ILM some of the modelers were also puppeteers. As puppeteers they were also in SAG since the actions of a puppet was acting. So when we did leaping laser printers or other things that required puppeting, that was under SAG agreements. I’m assuming the Muppets and any other key puppets would qualify. In this case it’s still considered acting when moving an inanimate object with hands and rods. But there is a direct connection between the actor and the final performance.
Acting and editing
The act of editing the film of a performance can change it’s impact.
The director guides the actor’s performance on set and selects the most appropriate take of the action. Then in editing specific intercutting is used to both tell the story and to emphasis the character as desired. In the early days of film there was a classic test done where a shot of an actor with a neutral expression was intercut with various scenes. The audience response was the actor was doing a good job of showing happiness, sadness and other emotions based solely on what it was intercut with.
The development of digital effects technology and artists proficient in its use allows actual manipulation of an acting performance. On STAR WARS: THE PHANTOM MENACE there were scenes that George Lucas requested be modified. An actor might have looked up in a take but the shot was run in reverse so the actor looked downward instead since that was deemed better for the cut. In some cases the scene was split and the sync of one side was slipped relative to the other. This was used to shift the timing of a reaction from one actor to another. In other cases eye blinks were added or removed as desired. Some directors have added tear drops or other modifications to a performance. The point here is that even a live action performance can be modified in post-production in an effort to create a better film experience. What was the actor’s truth on set may not be what appears in the final film. And in most cases the actors would probably not be aware of it. In the future we’ll likely see even more of this as directors and studios seek to take full advantage of the editing process. And don’t think that these types of details are beyond the scope or budget of post-production. Visual effects is already heavily used for things like removing wig netting, making adjustments to makeup (to the tune of over $1 million in some cases) and adding bruises or wounds. (That’s the real meaning of the term ‘digital makeup’.) A few actor adjustment shots will be a drop in the bucket compared to the other work already being done. It’s possible at a future time there will be debate about the implications of these types of modifications and what that means to actors.
The real process
Some actors talk or write about motion capture and 3D animation as ‘painting’ in the image. Painting? Really? That would make sense if you took a time machine back 100 years before computers and tried to explain it I suppose. But these days it makes as much sense as saying Michelangelo ‘doodled’ the David sculpture.
Short form: The performance of the actor is recorded and applied in some form to a Computer Graphic (CG) animated character.
Long form: Potentially dozens of people create a very detailed and fully articulated CG model. Imagine making a Madam Tussaunds wax figure but an order of magnitude more difficult. Character/creature designers, concept artists and animators work with the director to develop the look of the character. In some cases a physical model is sculpted out of clay for the director to review. Next it needs to be modeled head to toe, down to every key wrinkle in the computer. This is part modeling and part sculpting. A type of skeleton is built where every movable joint that is needed is included and each joint has a specific range of motion. The irises in the eyes can open and close, the eyes can move, the chest can expand when breathing, nostrils flare and the tongue is configured so that speech looks correct. In many cases muscles are built and configured to change shape. Texture artists paint every surface of the model, not with just one paint process but with multiple paint versions. One paint version shows how shiny different parts of the model are. Another is used to show dirt and still another may be used to show the subtle skin textures and tiny wrinkles.
Fur and hair may need to be added. The angle of the hair, the length of the hair and style of the hair have to be modeled. Another person may be responsible for defining where things are hard (finger nails, shells, etc) and where things are soft and pliable on the model. A shader writer writes specialized programming code to make the skin look real in different types of lighting and that it have specific translucency depending on where it is on the body. Every major facial and phonetic expression needs to be modeled to aid the animator in adjusting and expressing both emotion and voice. All this is done by a team of skilled artists and technicians who use the computer as a tool as a sculptor may use a chisel.
Every item of clothing and every prop the character handles has to be designed and built. The motion of the clothing is setup to be simulated. Does this clothing item behave like silk, cotton or canvas? Dangling earrings and the hair will be programmed so that they move in realistic manner when the character is moving.
All of that just to create the character model.
A good animator does act out the scene at their desk or within their mind visualizes the performance. Unlike a live action actor, the animator has to also observe and analyze every motion and expression change. And not only the position but also the timing of all of those motions need to be noted. The animator is able to visualize the changes in slow motion, forward and backwards. They have to translate that into key frame positions for the animated character model. If choice of the position of the arm or the eyebrow is wrong or if the time of those motions is wrong, the animation won’t work. Take after take is required to refine the animation. Any dialog is analyzed as well and the facial expression not only has to convey characters emotion at that 1/24 second granularity but the mouth shape, lips and tongue have to match the correct shape for the phonetic sound created in that moment.
Even actions like setting the feet down and walking take work on the computer. Activities in the real world that happen naturally take effort to do on the computer. The ground in the computer has to be built and matched to the ground of the real location and this is done by a person (match mover). If a foot goes too low then it will go into the ground and if it’s too high the foot won’t make contact with the ground.
And once the animation is done someone has to light it much like a Director of Photography does. Someone has to render the character and others have to composite (combine) the image with the original image from the set. There’s an entire team of craftsmen as big or bigger than the live action crew to make all of this happen.
The combination of powerful computers and digital video cameras made it possible to do computer vision. Images and motion could then be analyzed for scientific and medical purposes. The visual effects industry, as usual, looked to take advantage of these new technologies. Motion capture (MOCAP) became a way to reasonably capture 3D motion data, especially human motion, into the computer. This is useful for recording basic human motion for action shots. Facial capture has been developing which makes it possible to capture not only body motion but the entire performance. When the motion captured includes all aspects of a performance then the term ‘performance capture’ may be used instead of motion capture.
The typical system uses multiple markers that are strategically placed on the actor. A number of specialized cameras are placed around on a small to medium sized stage that is lit by subdued light. The multiple views are combined by special computer software to yield 3D information on each joint movement. In these cases only the motion was captured after the main photography had already been complete. Developments over the last few years (VAN HELSING, etc.) have allowed the capture of an actor on the stage or outside while being filmed at the same time. Additional simplified motion capture processes have also been developed (PIRATES 2, etc.) and it’s now also possible to motion capture a number of actors at the same time. These advancements have allowed motion capture actors more freedom and more interaction with fellow actors. (We still use tennis balls and other references at times. If the CG character is only 4 inches tall or is over 20 feet tall then it can be difficult for a real actor to stand in place and provide the correct interaction.)
For interactive performance capture actors are fitted with special suits and act with fellow actors who will remain in the final scenes. This interaction is of course beneficial for both the actors and the director. Another team of visual effects artists then go through and remove the performance actor by literally painting and restoring what would have been behind them. This is a very labor intensive and time consuming process since it involves hand painting frame by frame and creating imagery that isn’t in the original. The animated character or creature is then rendered and composited into the scene.
It might seem that once the motion data is captured it could simply be applied to the CG model and viola - a moving character that exactly matches the performance of the actor down to the smallest detail. But alas, such is not the case by a long shot. If it were then many of the animated films created today could be done using performance capture but they aren’t. Live action and animated movies are different art forms and what we’re seeing in some cases is a hybrid of the two. There’s still plenty of growing pains.
1. Even with improvement in the motion capture process there is quite a bit of cleanup required. A simple motion of an actor reaching out may have some frames where the arm leaps up or down a few inches. There might also be random frames where the arm goes behind the actors back or through his body. This takes a small team of people to go through and remove these glitches and clean up the data such that the performance is as pure as possible. In some cases large chunks of data may be missing which requires an animator to fill in with the appropriate motion.
2. CG characters seldom match the real actor unless it’s a digital double. Frequently the proportions are changed such that the arms or legs maybe longer or shorter. This means that the stride and interaction of what the actor was doing doesn’t match what it should. The further away this gets from the real actors body the more difficult it becomes to use the data as it is. A Satyr has totally different leg joints and needless to say a four-legged creature or caterpillar can render much of at least the body motion useless. Captured facial performance may not have much use when the facial structure is very different such as on an insect like creature. All this requires an animator skilled in the understanding of motion, performance and animation to try to retain at least a sense of what the original actors performance was providing. In some cases even performance capture data may become a basic reference or simply inspiration with the brunt of the performance created by the animator.
3. Eyes are the window to the soul and a key tool for the actor but actually capturing the eyes, eye blinks and iris changes has not happened in a meaningful way. That means it’s up to an animator to add these types of fine, but critical, details to complete the performance.
4. In the editing and visual effects process it may be determined that some adjustments may need to be made for technical reasons. Placement and timing of the character versus what the performance capture actor was doing. In some cases it may be the directors creative call to modify a performance once it has been reviewed with the rendered creature in place.
The end result is it’s likely a fair bit of performance capture undergoes some manipulation and work by animators and other artists. In some cases it may be inappropriate to use performance capture simply because the amount of work required is large and the amount of the performance that can be retained is small. Each project has to be evaluated dependent on the creature/character and how cleanly the performance capture can be used.
So you have pure live action acting on one end of the spectrum and pure animation from scratch on the other end. And in-between you have a gray area. One step away from pure animation is to use references. For DRAGONHEART the animators used both stills and clips of Sean Connery to try to incorporate a bit of his personality into the animated performance they were creating. For a project like RANGO they filmed the actors going through the scenes and the animators used this as a reference to create their animation, trying to keep the spirit and emotion of the actors. In some cases they probably followed the actors performance very closely and in other cases they may have ended up deviating quite a bit from the recorded reference. In this film the types of characters (based on a range of animals) would make it impossible to do an exact match of an actor’s performance. It’s up to the animator to re-interpret and adapt the actors art form into an animated art form, much as a screenwriter may have adapt a novel into a screenplay. The core insight may remain the same but changes have to made to deliver it in a different form.
On some of the more extensive performance capture projects lately only the animators who worked on the project know how much they did or didn’t do. It won’t be 100% nor will it be 0%. True judgement by those not involved directly in the animation would require a side by side comparison of the original footage next to the finished results. (Not just a shot or two and not just a cut between the two at a specific frame.) How closely does the final performance match the nuances of the original? Did 90% of the shots just require simply cleanup? Did they end up creating or modifying the performance in the majority of the shots? Was the performance used mainly as a reference and inspiration with the animators creating the majority of the final performance? Or does the actor’s full performance shine through the majority of the shots?
1. Both acting and animation are difficult and require talent. Both bring life to narrative characters. Both are necessary for making projects of these types and both should get proper respect.
2. Animators and visual effects artists are collaborators with live action production, including acting. As such it would be nice to be acknowledged for the contribution these artists bring to a project and a realistic assessment of the amount of creativity they bring, not to mention the amount of work involved behind the scenes.
3. Performance capture is not a pure technology or art. It will likely have some mix of both an actor’s work and an animators work. What the blend is will depend on the specific project. Those reviewing the performance (critics, awards, etc) will have to determine what balance range they’re willing to consider and to provide credit where credit is due.
4. As visual effects continues to progress and as projects push into new territories there will be even more overlap and blurring of the contribution of all involved in making films. The thinking and mind sets will have to keep up with these developments.
5. Performance capture is not Digital "Make-up". As the word Performance indicates, it’s based on performance which at this point involves some balance of an actor and animator collaborating in a manner to bring the character to life.
What is your opinion on the mainstream, out-of-industry perception of performance capture?
Andy Serkis believes that 'Performance Capture is a tool, it is simply a way of recording an actors performance, no different to a camera'. What are your thoughts on this statement?
In the context of a traditional 'motion capture' shot featuring a digital character, approximately what percentage of the actors body performance would you say remains untouched or unaltered by animators after the fact?
In the case of facial capture, approximately what percentage of the actors facial performance would you say remains untouched or unaltered by animators after the fact?
I think the technology will continue to develop to make it easier to capture more accurately with less clean up work required.Where do you think the technology is heading?
[Update: 7-20-12 Here's an article on motion capture, animation and video games ]
Update 5-13-14 Randall Cook, the Animation supervisor for the Lord of the Rings films, discusses motion capture and Andy Serkis