VES Visual Effects Handbook feedback
We are having meetings and starting to work on the revised VES handbook. Most of the feedback we have received has been very general. If you want to see additional information, improvements, changes, etc to the handbook please post here or email to the VES. Is it working for you? Is there anything lacking? How is the Kindle or other eBook version for you? Get in your feedback in NOW.
Townhall meetings
Today there was a VES townhall on the web where members and others were able to ask questions and hear Jeff Okun and Eric Roth discuss VES 2.0
There will be additional townhall meetings.
VES 2.0 feedback
As requested if anyone has something to say regarding VES 2.0, especially solutions and suggestions contact Jeff Okun at: jeffokun at aol.com
You can also contact: leadership at visualeffectssociety.com
VES Forums
The VES has expanded their online site with a beta version of VES Forums. Check your VES emails to get the info on it and some upcoming VES events.
Wednesday, January 25, 2012
Friday, January 13, 2012
Acting and Animation
Acting and Animation
There has been debate and discussion lately regarding if motion capture is pure acting.
Here are a couple of latest links to check out:
James Franco requests recognition for Andy Serkis work on RISE OF THE PLANET OF THE APES
Here's the URL if it doesn't work: http://www.deadline.com/2012/01/oscar-exclusive-james-franco-on-why-andy-serkis-deserves-credit-from-actors/
Then, what the Weta effects team did was to essentially “paint” the look of Caesar over Andy’s performance. This is not animation as much as it’s digital “make-up.”
And here’s one of the responses. Be sure to check out the links to the videos by Andy Serkis and Jeffrey Engle
I’ll try to clarify a few things if I may.
Both acting and animation are very difficult to do well and require talent and artistry. Some people try to define the specifics of each one but at the end of the day they are both used to bring life to a narrative character. Whether it’s a live action person or an animated person/animal/object on screen, the audience should be able to relate to the character in some way and be provided some emotion connection.
Acting-
Acting is one of those jobs that looks very easy to those who don’t do it.
Having worked with actors, directed actors and taken both acting and improv classes, I can say with confidence it is very difficult to do well. An actor has to act and react as the character they are supposed to portray. They have to be able to become one with the character. When in that zone the actor is in the moment and the fictional world becomes their world. An actor has to put themselves out there for all to watch and that can make them feel very vulnerable. Actors are not allowed to analyze or observe their performance since that takes them out of the moment. They have to make it look effortless and to feel completely natural even though they are saying written dialog and may be doing the shot multiple times. Subtle facial expressions and body language convey more than the spoken dialog. Casting of actors for a film is very important since different actors will bring a different take on a character.
Animation-
Animators are frequently given a bit of a brush off from the rest of the entertainment community. Much of what’s animated is aimed at children or young adults so can’t be serious according to some. But take a close look at many animation classics, even for children, and great animation does reverberate emotionally in the viewer. Look at Dumbo, Pinocchio and other Disney classics. Warner Brothers and other studios also brought engaging animated characters to life. These days Pixar and similar 3D animation studios are accomplishing the same thing.
Animators on visual effects projects must achieve a level of realism beyond what happens in most animated films. One’s not better than the other but there are differences. A visual effects animator may have to animate a horse or other animal and make them totally believable as the animal they are supposed to portray. In many cases the animation may be intercut with the footage of the real animal so the match in motion has to be spot on. Visual effects animators may also be called on to animate fantasy creatures or talking, breathing characters.
A true test for a character animator is to animate a simple flour sack drawing or model. Even without a body or face an animator can bring life to the flour sack in such a way to convey happiness, sadness, curiosity and other emotions.
Even though the term may be computer animation an actual artist is the one that does the animation. The computer is just the tool as a pencil was to pre-computer animators. Casting the right animator can be as important as casting the right actor.
Puppets
When I worked at ILM some of the modelers were also puppeteers. As puppeteers they were also in SAG since the actions of a puppet was acting. So when we did leaping laser printers or other things that required puppeting, that was under SAG agreements. I’m assuming the Muppets and any other key puppets would qualify. In this case it’s still considered acting when moving an inanimate object with hands and rods. But there is a direct connection between the actor and the final performance.
Acting and editing
The act of editing the film of a performance can change it’s impact.
The director guides the actor’s performance on set and selects the most appropriate take of the action. Then in editing specific intercutting is used to both tell the story and to emphasis the character as desired. In the early days of film there was a classic test done where a shot of an actor with a neutral expression was intercut with various scenes. The audience response was the actor was doing a good job of showing happiness, sadness and other emotions based solely on what it was intercut with.
Manipulation
The development of digital effects technology and artists proficient in its use allows actual manipulation of an acting performance. On STAR WARS: THE PHANTOM MENACE there were scenes that George Lucas requested be modified. An actor might have looked up in a take but the shot was run in reverse so the actor looked downward instead since that was deemed better for the cut. In some cases the scene was split and the sync of one side was slipped relative to the other. This was used to shift the timing of a reaction from one actor to another. In other cases eye blinks were added or removed as desired. Some directors have added tear drops or other modifications to a performance. The point here is that even a live action performance can be modified in post-production in an effort to create a better film experience. What was the actor’s truth on set may not be what appears in the final film. And in most cases the actors would probably not be aware of it. In the future we’ll likely see even more of this as directors and studios seek to take full advantage of the editing process. And don’t think that these types of details are beyond the scope or budget of post-production. Visual effects is already heavily used for things like removing wig netting, making adjustments to makeup (to the tune of over $1 million in some cases) and adding bruises or wounds. (That’s the real meaning of the term ‘digital makeup’.) A few actor adjustment shots will be a drop in the bucket compared to the other work already being done. It’s possible at a future time there will be debate about the implications of these types of modifications and what that means to actors.
The real process
Some actors talk or write about motion capture and 3D animation as ‘painting’ in the image. Painting? Really? That would make sense if you took a time machine back 100 years before computers and tried to explain it I suppose. But these days it makes as much sense as saying Michelangelo ‘doodled’ the David sculpture.
Short form: The performance of the actor is recorded and applied in some form to a Computer Graphic (CG) animated character.
Long form: Potentially dozens of people create a very detailed and fully articulated CG model. Imagine making a Madam Tussaunds wax figure but an order of magnitude more difficult. Character/creature designers, concept artists and animators work with the director to develop the look of the character. In some cases a physical model is sculpted out of clay for the director to review. Next it needs to be modeled head to toe, down to every key wrinkle in the computer. This is part modeling and part sculpting. A type of skeleton is built where every movable joint that is needed is included and each joint has a specific range of motion. The irises in the eyes can open and close, the eyes can move, the chest can expand when breathing, nostrils flare and the tongue is configured so that speech looks correct. In many cases muscles are built and configured to change shape. Texture artists paint every surface of the model, not with just one paint process but with multiple paint versions. One paint version shows how shiny different parts of the model are. Another is used to show dirt and still another may be used to show the subtle skin textures and tiny wrinkles.
Fur and hair may need to be added. The angle of the hair, the length of the hair and style of the hair have to be modeled. Another person may be responsible for defining where things are hard (finger nails, shells, etc) and where things are soft and pliable on the model. A shader writer writes specialized programming code to make the skin look real in different types of lighting and that it have specific translucency depending on where it is on the body. Every major facial and phonetic expression needs to be modeled to aid the animator in adjusting and expressing both emotion and voice. All this is done by a team of skilled artists and technicians who use the computer as a tool as a sculptor may use a chisel.
Every item of clothing and every prop the character handles has to be designed and built. The motion of the clothing is setup to be simulated. Does this clothing item behave like silk, cotton or canvas? Dangling earrings and the hair will be programmed so that they move in realistic manner when the character is moving.
All of that just to create the character model.
Animation Process
A good animator does act out the scene at their desk or within their mind visualizes the performance. Unlike a live action actor, the animator has to also observe and analyze every motion and expression change. And not only the position but also the timing of all of those motions need to be noted. The animator is able to visualize the changes in slow motion, forward and backwards. They have to translate that into key frame positions for the animated character model. If choice of the position of the arm or the eyebrow is wrong or if the time of those motions is wrong, the animation won’t work. Take after take is required to refine the animation. Any dialog is analyzed as well and the facial expression not only has to convey characters emotion at that 1/24 second granularity but the mouth shape, lips and tongue have to match the correct shape for the phonetic sound created in that moment.
Even actions like setting the feet down and walking take work on the computer. Activities in the real world that happen naturally take effort to do on the computer. The ground in the computer has to be built and matched to the ground of the real location and this is done by a person (match mover). If a foot goes too low then it will go into the ground and if it’s too high the foot won’t make contact with the ground.
And once the animation is done someone has to light it much like a Director of Photography does. Someone has to render the character and others have to composite (combine) the image with the original image from the set. There’s an entire team of craftsmen as big or bigger than the live action crew to make all of this happen.
Performance capture
The combination of powerful computers and digital video cameras made it possible to do computer vision. Images and motion could then be analyzed for scientific and medical purposes. The visual effects industry, as usual, looked to take advantage of these new technologies. Motion capture (MOCAP) became a way to reasonably capture 3D motion data, especially human motion, into the computer. This is useful for recording basic human motion for action shots. Facial capture has been developing which makes it possible to capture not only body motion but the entire performance. When the motion captured includes all aspects of a performance then the term ‘performance capture’ may be used instead of motion capture.
The typical system uses multiple markers that are strategically placed on the actor. A number of specialized cameras are placed around on a small to medium sized stage that is lit by subdued light. The multiple views are combined by special computer software to yield 3D information on each joint movement. In these cases only the motion was captured after the main photography had already been complete. Developments over the last few years (VAN HELSING, etc.) have allowed the capture of an actor on the stage or outside while being filmed at the same time. Additional simplified motion capture processes have also been developed (PIRATES 2, etc.) and it’s now also possible to motion capture a number of actors at the same time. These advancements have allowed motion capture actors more freedom and more interaction with fellow actors. (We still use tennis balls and other references at times. If the CG character is only 4 inches tall or is over 20 feet tall then it can be difficult for a real actor to stand in place and provide the correct interaction.)
For interactive performance capture actors are fitted with special suits and act with fellow actors who will remain in the final scenes. This interaction is of course beneficial for both the actors and the director. Another team of visual effects artists then go through and remove the performance actor by literally painting and restoring what would have been behind them. This is a very labor intensive and time consuming process since it involves hand painting frame by frame and creating imagery that isn’t in the original. The animated character or creature is then rendered and composited into the scene.
The problems
It might seem that once the motion data is captured it could simply be applied to the CG model and viola - a moving character that exactly matches the performance of the actor down to the smallest detail. But alas, such is not the case by a long shot. If it were then many of the animated films created today could be done using performance capture but they aren’t. Live action and animated movies are different art forms and what we’re seeing in some cases is a hybrid of the two. There’s still plenty of growing pains.
1. Even with improvement in the motion capture process there is quite a bit of cleanup required. A simple motion of an actor reaching out may have some frames where the arm leaps up or down a few inches. There might also be random frames where the arm goes behind the actors back or through his body. This takes a small team of people to go through and remove these glitches and clean up the data such that the performance is as pure as possible. In some cases large chunks of data may be missing which requires an animator to fill in with the appropriate motion.
2. CG characters seldom match the real actor unless it’s a digital double. Frequently the proportions are changed such that the arms or legs maybe longer or shorter. This means that the stride and interaction of what the actor was doing doesn’t match what it should. The further away this gets from the real actors body the more difficult it becomes to use the data as it is. A Satyr has totally different leg joints and needless to say a four-legged creature or caterpillar can render much of at least the body motion useless. Captured facial performance may not have much use when the facial structure is very different such as on an insect like creature. All this requires an animator skilled in the understanding of motion, performance and animation to try to retain at least a sense of what the original actors performance was providing. In some cases even performance capture data may become a basic reference or simply inspiration with the brunt of the performance created by the animator.
3. Eyes are the window to the soul and a key tool for the actor but actually capturing the eyes, eye blinks and iris changes has not happened in a meaningful way. That means it’s up to an animator to add these types of fine, but critical, details to complete the performance.
4. In the editing and visual effects process it may be determined that some adjustments may need to be made for technical reasons. Placement and timing of the character versus what the performance capture actor was doing. In some cases it may be the directors creative call to modify a performance once it has been reviewed with the rendered creature in place.
The end result is it’s likely a fair bit of performance capture undergoes some manipulation and work by animators and other artists. In some cases it may be inappropriate to use performance capture simply because the amount of work required is large and the amount of the performance that can be retained is small. Each project has to be evaluated dependent on the creature/character and how cleanly the performance capture can be used.
So you have pure live action acting on one end of the spectrum and pure animation from scratch on the other end. And in-between you have a gray area. One step away from pure animation is to use references. For DRAGONHEART the animators used both stills and clips of Sean Connery to try to incorporate a bit of his personality into the animated performance they were creating. For a project like RANGO they filmed the actors going through the scenes and the animators used this as a reference to create their animation, trying to keep the spirit and emotion of the actors. In some cases they probably followed the actors performance very closely and in other cases they may have ended up deviating quite a bit from the recorded reference. In this film the types of characters (based on a range of animals) would make it impossible to do an exact match of an actor’s performance. It’s up to the animator to re-interpret and adapt the actors art form into an animated art form, much as a screenwriter may have adapt a novel into a screenplay. The core insight may remain the same but changes have to made to deliver it in a different form.
On some of the more extensive performance capture projects lately only the animators who worked on the project know how much they did or didn’t do. It won’t be 100% nor will it be 0%. True judgement by those not involved directly in the animation would require a side by side comparison of the original footage next to the finished results. (Not just a shot or two and not just a cut between the two at a specific frame.) How closely does the final performance match the nuances of the original? Did 90% of the shots just require simply cleanup? Did they end up creating or modifying the performance in the majority of the shots? Was the performance used mainly as a reference and inspiration with the animators creating the majority of the final performance? Or does the actor’s full performance shine through the majority of the shots?
Summary
1. Both acting and animation are difficult and require talent. Both bring life to narrative characters. Both are necessary for making projects of these types and both should get proper respect.
2. Animators and visual effects artists are collaborators with live action production, including acting. As such it would be nice to be acknowledged for the contribution these artists bring to a project and a realistic assessment of the amount of creativity they bring, not to mention the amount of work involved behind the scenes.
3. Performance capture is not a pure technology or art. It will likely have some mix of both an actor’s work and an animators work. What the blend is will depend on the specific project. Those reviewing the performance (critics, awards, etc) will have to determine what balance range they’re willing to consider and to provide credit where credit is due.
4. As visual effects continues to progress and as projects push into new territories there will be even more overlap and blurring of the contribution of all involved in making films. The thinking and mind sets will have to keep up with these developments.
5. Performance capture is not Digital "Make-up". As the word Performance indicates, it’s based on performance which at this point involves some balance of an actor and animator collaborating in a manner to bring the character to life.
There has been debate and discussion lately regarding if motion capture is pure acting.
Here are a couple of latest links to check out:
James Franco requests recognition for Andy Serkis work on RISE OF THE PLANET OF THE APES
Here's the URL if it doesn't work: http://www.deadline.com/2012/01/oscar-exclusive-james-franco-on-why-andy-serkis-deserves-credit-from-actors/
Then, what the Weta effects team did was to essentially “paint” the look of Caesar over Andy’s performance. This is not animation as much as it’s digital “make-up.”
And here’s one of the responses. Be sure to check out the links to the videos by Andy Serkis and Jeffrey Engle
I’ll try to clarify a few things if I may.
Both acting and animation are very difficult to do well and require talent and artistry. Some people try to define the specifics of each one but at the end of the day they are both used to bring life to a narrative character. Whether it’s a live action person or an animated person/animal/object on screen, the audience should be able to relate to the character in some way and be provided some emotion connection.
Acting-
Acting is one of those jobs that looks very easy to those who don’t do it.
Having worked with actors, directed actors and taken both acting and improv classes, I can say with confidence it is very difficult to do well. An actor has to act and react as the character they are supposed to portray. They have to be able to become one with the character. When in that zone the actor is in the moment and the fictional world becomes their world. An actor has to put themselves out there for all to watch and that can make them feel very vulnerable. Actors are not allowed to analyze or observe their performance since that takes them out of the moment. They have to make it look effortless and to feel completely natural even though they are saying written dialog and may be doing the shot multiple times. Subtle facial expressions and body language convey more than the spoken dialog. Casting of actors for a film is very important since different actors will bring a different take on a character.
Animation-
Animators are frequently given a bit of a brush off from the rest of the entertainment community. Much of what’s animated is aimed at children or young adults so can’t be serious according to some. But take a close look at many animation classics, even for children, and great animation does reverberate emotionally in the viewer. Look at Dumbo, Pinocchio and other Disney classics. Warner Brothers and other studios also brought engaging animated characters to life. These days Pixar and similar 3D animation studios are accomplishing the same thing.
Animators on visual effects projects must achieve a level of realism beyond what happens in most animated films. One’s not better than the other but there are differences. A visual effects animator may have to animate a horse or other animal and make them totally believable as the animal they are supposed to portray. In many cases the animation may be intercut with the footage of the real animal so the match in motion has to be spot on. Visual effects animators may also be called on to animate fantasy creatures or talking, breathing characters.
A true test for a character animator is to animate a simple flour sack drawing or model. Even without a body or face an animator can bring life to the flour sack in such a way to convey happiness, sadness, curiosity and other emotions.
Even though the term may be computer animation an actual artist is the one that does the animation. The computer is just the tool as a pencil was to pre-computer animators. Casting the right animator can be as important as casting the right actor.
Puppets
When I worked at ILM some of the modelers were also puppeteers. As puppeteers they were also in SAG since the actions of a puppet was acting. So when we did leaping laser printers or other things that required puppeting, that was under SAG agreements. I’m assuming the Muppets and any other key puppets would qualify. In this case it’s still considered acting when moving an inanimate object with hands and rods. But there is a direct connection between the actor and the final performance.
Acting and editing
The act of editing the film of a performance can change it’s impact.
The director guides the actor’s performance on set and selects the most appropriate take of the action. Then in editing specific intercutting is used to both tell the story and to emphasis the character as desired. In the early days of film there was a classic test done where a shot of an actor with a neutral expression was intercut with various scenes. The audience response was the actor was doing a good job of showing happiness, sadness and other emotions based solely on what it was intercut with.
Manipulation
The development of digital effects technology and artists proficient in its use allows actual manipulation of an acting performance. On STAR WARS: THE PHANTOM MENACE there were scenes that George Lucas requested be modified. An actor might have looked up in a take but the shot was run in reverse so the actor looked downward instead since that was deemed better for the cut. In some cases the scene was split and the sync of one side was slipped relative to the other. This was used to shift the timing of a reaction from one actor to another. In other cases eye blinks were added or removed as desired. Some directors have added tear drops or other modifications to a performance. The point here is that even a live action performance can be modified in post-production in an effort to create a better film experience. What was the actor’s truth on set may not be what appears in the final film. And in most cases the actors would probably not be aware of it. In the future we’ll likely see even more of this as directors and studios seek to take full advantage of the editing process. And don’t think that these types of details are beyond the scope or budget of post-production. Visual effects is already heavily used for things like removing wig netting, making adjustments to makeup (to the tune of over $1 million in some cases) and adding bruises or wounds. (That’s the real meaning of the term ‘digital makeup’.) A few actor adjustment shots will be a drop in the bucket compared to the other work already being done. It’s possible at a future time there will be debate about the implications of these types of modifications and what that means to actors.
The real process
Some actors talk or write about motion capture and 3D animation as ‘painting’ in the image. Painting? Really? That would make sense if you took a time machine back 100 years before computers and tried to explain it I suppose. But these days it makes as much sense as saying Michelangelo ‘doodled’ the David sculpture.
Short form: The performance of the actor is recorded and applied in some form to a Computer Graphic (CG) animated character.
Long form: Potentially dozens of people create a very detailed and fully articulated CG model. Imagine making a Madam Tussaunds wax figure but an order of magnitude more difficult. Character/creature designers, concept artists and animators work with the director to develop the look of the character. In some cases a physical model is sculpted out of clay for the director to review. Next it needs to be modeled head to toe, down to every key wrinkle in the computer. This is part modeling and part sculpting. A type of skeleton is built where every movable joint that is needed is included and each joint has a specific range of motion. The irises in the eyes can open and close, the eyes can move, the chest can expand when breathing, nostrils flare and the tongue is configured so that speech looks correct. In many cases muscles are built and configured to change shape. Texture artists paint every surface of the model, not with just one paint process but with multiple paint versions. One paint version shows how shiny different parts of the model are. Another is used to show dirt and still another may be used to show the subtle skin textures and tiny wrinkles.
Fur and hair may need to be added. The angle of the hair, the length of the hair and style of the hair have to be modeled. Another person may be responsible for defining where things are hard (finger nails, shells, etc) and where things are soft and pliable on the model. A shader writer writes specialized programming code to make the skin look real in different types of lighting and that it have specific translucency depending on where it is on the body. Every major facial and phonetic expression needs to be modeled to aid the animator in adjusting and expressing both emotion and voice. All this is done by a team of skilled artists and technicians who use the computer as a tool as a sculptor may use a chisel.
Every item of clothing and every prop the character handles has to be designed and built. The motion of the clothing is setup to be simulated. Does this clothing item behave like silk, cotton or canvas? Dangling earrings and the hair will be programmed so that they move in realistic manner when the character is moving.
All of that just to create the character model.
Animation Process
A good animator does act out the scene at their desk or within their mind visualizes the performance. Unlike a live action actor, the animator has to also observe and analyze every motion and expression change. And not only the position but also the timing of all of those motions need to be noted. The animator is able to visualize the changes in slow motion, forward and backwards. They have to translate that into key frame positions for the animated character model. If choice of the position of the arm or the eyebrow is wrong or if the time of those motions is wrong, the animation won’t work. Take after take is required to refine the animation. Any dialog is analyzed as well and the facial expression not only has to convey characters emotion at that 1/24 second granularity but the mouth shape, lips and tongue have to match the correct shape for the phonetic sound created in that moment.
Even actions like setting the feet down and walking take work on the computer. Activities in the real world that happen naturally take effort to do on the computer. The ground in the computer has to be built and matched to the ground of the real location and this is done by a person (match mover). If a foot goes too low then it will go into the ground and if it’s too high the foot won’t make contact with the ground.
And once the animation is done someone has to light it much like a Director of Photography does. Someone has to render the character and others have to composite (combine) the image with the original image from the set. There’s an entire team of craftsmen as big or bigger than the live action crew to make all of this happen.
Performance capture
The combination of powerful computers and digital video cameras made it possible to do computer vision. Images and motion could then be analyzed for scientific and medical purposes. The visual effects industry, as usual, looked to take advantage of these new technologies. Motion capture (MOCAP) became a way to reasonably capture 3D motion data, especially human motion, into the computer. This is useful for recording basic human motion for action shots. Facial capture has been developing which makes it possible to capture not only body motion but the entire performance. When the motion captured includes all aspects of a performance then the term ‘performance capture’ may be used instead of motion capture.
The typical system uses multiple markers that are strategically placed on the actor. A number of specialized cameras are placed around on a small to medium sized stage that is lit by subdued light. The multiple views are combined by special computer software to yield 3D information on each joint movement. In these cases only the motion was captured after the main photography had already been complete. Developments over the last few years (VAN HELSING, etc.) have allowed the capture of an actor on the stage or outside while being filmed at the same time. Additional simplified motion capture processes have also been developed (PIRATES 2, etc.) and it’s now also possible to motion capture a number of actors at the same time. These advancements have allowed motion capture actors more freedom and more interaction with fellow actors. (We still use tennis balls and other references at times. If the CG character is only 4 inches tall or is over 20 feet tall then it can be difficult for a real actor to stand in place and provide the correct interaction.)
For interactive performance capture actors are fitted with special suits and act with fellow actors who will remain in the final scenes. This interaction is of course beneficial for both the actors and the director. Another team of visual effects artists then go through and remove the performance actor by literally painting and restoring what would have been behind them. This is a very labor intensive and time consuming process since it involves hand painting frame by frame and creating imagery that isn’t in the original. The animated character or creature is then rendered and composited into the scene.
The problems
It might seem that once the motion data is captured it could simply be applied to the CG model and viola - a moving character that exactly matches the performance of the actor down to the smallest detail. But alas, such is not the case by a long shot. If it were then many of the animated films created today could be done using performance capture but they aren’t. Live action and animated movies are different art forms and what we’re seeing in some cases is a hybrid of the two. There’s still plenty of growing pains.
1. Even with improvement in the motion capture process there is quite a bit of cleanup required. A simple motion of an actor reaching out may have some frames where the arm leaps up or down a few inches. There might also be random frames where the arm goes behind the actors back or through his body. This takes a small team of people to go through and remove these glitches and clean up the data such that the performance is as pure as possible. In some cases large chunks of data may be missing which requires an animator to fill in with the appropriate motion.
2. CG characters seldom match the real actor unless it’s a digital double. Frequently the proportions are changed such that the arms or legs maybe longer or shorter. This means that the stride and interaction of what the actor was doing doesn’t match what it should. The further away this gets from the real actors body the more difficult it becomes to use the data as it is. A Satyr has totally different leg joints and needless to say a four-legged creature or caterpillar can render much of at least the body motion useless. Captured facial performance may not have much use when the facial structure is very different such as on an insect like creature. All this requires an animator skilled in the understanding of motion, performance and animation to try to retain at least a sense of what the original actors performance was providing. In some cases even performance capture data may become a basic reference or simply inspiration with the brunt of the performance created by the animator.
3. Eyes are the window to the soul and a key tool for the actor but actually capturing the eyes, eye blinks and iris changes has not happened in a meaningful way. That means it’s up to an animator to add these types of fine, but critical, details to complete the performance.
4. In the editing and visual effects process it may be determined that some adjustments may need to be made for technical reasons. Placement and timing of the character versus what the performance capture actor was doing. In some cases it may be the directors creative call to modify a performance once it has been reviewed with the rendered creature in place.
The end result is it’s likely a fair bit of performance capture undergoes some manipulation and work by animators and other artists. In some cases it may be inappropriate to use performance capture simply because the amount of work required is large and the amount of the performance that can be retained is small. Each project has to be evaluated dependent on the creature/character and how cleanly the performance capture can be used.
So you have pure live action acting on one end of the spectrum and pure animation from scratch on the other end. And in-between you have a gray area. One step away from pure animation is to use references. For DRAGONHEART the animators used both stills and clips of Sean Connery to try to incorporate a bit of his personality into the animated performance they were creating. For a project like RANGO they filmed the actors going through the scenes and the animators used this as a reference to create their animation, trying to keep the spirit and emotion of the actors. In some cases they probably followed the actors performance very closely and in other cases they may have ended up deviating quite a bit from the recorded reference. In this film the types of characters (based on a range of animals) would make it impossible to do an exact match of an actor’s performance. It’s up to the animator to re-interpret and adapt the actors art form into an animated art form, much as a screenwriter may have adapt a novel into a screenplay. The core insight may remain the same but changes have to made to deliver it in a different form.
On some of the more extensive performance capture projects lately only the animators who worked on the project know how much they did or didn’t do. It won’t be 100% nor will it be 0%. True judgement by those not involved directly in the animation would require a side by side comparison of the original footage next to the finished results. (Not just a shot or two and not just a cut between the two at a specific frame.) How closely does the final performance match the nuances of the original? Did 90% of the shots just require simply cleanup? Did they end up creating or modifying the performance in the majority of the shots? Was the performance used mainly as a reference and inspiration with the animators creating the majority of the final performance? Or does the actor’s full performance shine through the majority of the shots?
Summary
1. Both acting and animation are difficult and require talent. Both bring life to narrative characters. Both are necessary for making projects of these types and both should get proper respect.
2. Animators and visual effects artists are collaborators with live action production, including acting. As such it would be nice to be acknowledged for the contribution these artists bring to a project and a realistic assessment of the amount of creativity they bring, not to mention the amount of work involved behind the scenes.
3. Performance capture is not a pure technology or art. It will likely have some mix of both an actor’s work and an animators work. What the blend is will depend on the specific project. Those reviewing the performance (critics, awards, etc) will have to determine what balance range they’re willing to consider and to provide credit where credit is due.
4. As visual effects continues to progress and as projects push into new territories there will be even more overlap and blurring of the contribution of all involved in making films. The thinking and mind sets will have to keep up with these developments.
5. Performance capture is not Digital "Make-up". As the word Performance indicates, it’s based on performance which at this point involves some balance of an actor and animator collaborating in a manner to bring the character to life.
------------------------------------------------------------------------
[Update Mar. 4, 2012: I received some questions from a reader related to this that I answered. Thought I'd add it to this post]What is your opinion on the mainstream, out-of-industry perception of performance capture?
Most people outside of visual effects think performance capture is simply hooking up an actor to a CG puppet of sorts. That there's some type of computer program that transforms the actor directly into a cartoon or CG character.
Andy Serkis believes that 'Performance Capture is a tool, it is simply a way of recording an actors performance, no different to a camera'. What are your thoughts on this statement?
Years from now that may be closer to the truth but currently it's not close to reality. The differences are a camera records the actor directly and presents it as it was acted. The resolution, color and other things may be different but that doesn't change the actual performance. The impact of the acting can be influenced by choice of lens, camera angles and later in the edit but what you see is almost always what you get.
In the case of performance capture the raw acting that is captured has to manipulated and in some cases shoe horned in to match a much different CG character. Each one of those involves a creative and technical choice. Performance capture currently also isn't a pure process. Animators and others will have to correct for glitches in the data and oversee sections where the acting performance was lost due to technology limits. This is especially true for facial capture where animators have to adjust and fill in much of the details to get it to match a much different character face.
Imagine a camera that drops large number of frames during the shot and that produces very fuzzy images. Since the objective is to create a continuous shot that is sharp and uses a different costume, someone else now has to act the part, dressed in the correct costume, using the original as a rough guide. In the end it's not a direct reproduction of the original performance, it is an attempt to somewhat mimic the original performance.
And of course with a camera alone the director can't change a performance once it's been completed. But with performance capture they can and do change the performance after the fact. If in the edit the director wants a slightly different body action, facial expression or timing change then it's requested and the animation team make the change without involvement of the original actor.
How successful the original performance capture comes through the final images is dependent on many factors. Some shots may be very accurate representation and in other cases it may be very different. That's why looking at the result by themselves or just one or two comparisons of the original performance to the finished results doesn't really confirm how much is the original performance. The only true way is to see all the shots before and after.
In the context of a traditional 'motion capture' shot featuring a digital character, approximately what percentage of the actors body performance would you say remains untouched or unaltered by animators after the fact?
As mentioned above that varies enormously. What system was used, who was cleaning up the data, how closely does the CG character match the original actor, how much hand work is required and how many changes the director chose to make after the fact. If it's a specific leap and that is being mapped on to a digital double then 75% or more may may be unaltered. (Still cleaned data but more or less matching the actor's body motion) If it's a CG character with a much different body shape (longer legs, bulkier, etc) running and leaning against a lamp post it could be the inverse of those numbers.
In the case of facial capture, approximately what percentage of the actors facial performance would you say remains untouched or unaltered by animators after the fact?
Same basic issues as the body but facial capture is even more problematic. I would guess that just about all facial capture requires some hand work beyond just a little data cleaning. With body capture there are a limited number of key joints that need to be captured. (i.e. the knee was bent at 35 degrees on this frame). Body language is important but if you capture a basic gait of a walk cycle most people would accept that was the actor even if it wasn't a totally accurate reproduction. Motion capturing the body has also been done for a number of years now so the technology and understanding is much greater. Facial capture is a much newer process and si still being fully explored.
With facial capture you have multiple muscles all acting at the same time to subtly manipulate mouth, eyebrows and basic facial expression.
Think of the range of people's faces and personalities. Each person is the world is someone unique and identifiable by their face. Yet if all you saw of a person is their body it would be almost impossible to pin down with much accuracy.
Consider the difference of a smirk and a smile. Very different experience for the audience and actor but very difficult to quantify. Maybe one eyebrow is ached a faction of an inch higher. Maybe the corner of the mouth is raised a little more. Maybe the speed at which it happens is slightly different. And it's the type of thing where you may have to already know the person or character to some extent to even tell the difference. Where do you place points and what do you monitor when capturing? How accurate does the CG model muscle structure actually match the actor's facial muscles? And the more different the CG character is (cartoon style, animal, etc) the more difficult this translation is to make.
For movies we usually try to treat creatures as forms of people (anthropomorphic) and yet most creatures don't match our body form and certainly don't match our facial form. The creature mouths may not even be able to do what we want so someone has to modify the creature CG model to do that and others have to monitor and adjust the performance required to accomplish that at the level required for the specific project.
As an audience the face of the character is one of our key focal points so it's even more critical than basic body motion. And one of the places we look is the eyes in real life and the movies. It's been said eyes are the window of the soul. Yet trying to capture true eye motion and then to apply this to a CG character is extremely difficult. In most cases animators have to take over for this so one of the key performances tools from the actor is in the hands of someone else.
I think the technology will continue to develop to make it easier to capture more accurately with less clean up work required.Where do you think the technology is heading?
Facial capture is the area where we will see even larger changes. But just as important as the technology, if not more important, is the requirement for animators to be able to work with this and know how to get the most out of it. The other important item is those in the industry, including directors and actors, need to understand what it can provide and what it can't provide. They have to understand when it should be used, how it should be used and how to evaluate it. And all of that requires an acceptance of it as a collaboration of actor and animators. Ignoring either groups contribution to the results is not only incorrect, it's wrong.
[Update: 7-20-12 Here's an article on motion capture, animation and video games ]
Update 5-13-14 Randall Cook, the Animation supervisor for the Lord of the Rings films, discusses motion capture and Andy Serkis
[Update: 7-20-12 Here's an article on motion capture, animation and video games ]
Update 5-13-14 Randall Cook, the Animation supervisor for the Lord of the Rings films, discusses motion capture and Andy Serkis
Tuesday, December 13, 2011
VES and VFX trade organization
There's a LinkedIn discussion going on regarding Vfx trade organizations and the Vfx foundation.
LinkedIn discussion
This also includes some potshots at the VES. I wrote up a response but thought it best to post here as a reference.
I'm on the board of the VES but I don't speak for the Ves so these are simply my personal views. Bob and Colin have brought up some points but I'd like to clarify a few things.
I don't see what's to be gained by bashing the VES. The Ves was formed 15 years ago to be an honorary society, similar in many ways to the ASC and the Academy of Motion Pictures. To honor professionals, to provide education and to advance the art of visual effects where possible with standards and research. And to that end, it's accomplished that. Educational programs are frequently held around the world. A couple of weeks ago there was an event covering the latest on performance capture. Many of these event videos are online at the website (upgrades in progress) The Ves created the Ves handbook and is looking to do more in this area in the future.
I pay dues to the Ves because I do this for a living. I'm a professional and the Ves is the largest group of visual effects professionals. Just as I buy books and magazines and pay to attend events related to my professional. It's a business expense.
I don't begrudge paying dues to the Academy and wouldn't begrudge paying into the ASC if I was a member. If you don't want to join the Ves then don't. If you're a Ves member and wish to see improvements then join a committee. Most of the Ves is volunteer based and that's how the Ves handbook came to be.
There are those that want the Ves to be a trade organization or a union. But that ship sailed when the Ves was founded. It may sound like an easy thing to change the structure and intent of the Ves (or other existing non profit) but it is not. To totally switch to a different type of organization would likely require dissolving the Ves and creating a totally new and different organization. The current members wouldn't be members under a trade organization and the same might apply as a union. The US government treats unions differently than standard non-union groups regarding legal issues. The same likely applies to a trade organization. There are also tax issues that would change. If a cinematographer has an issue getting paid he/she doesn't go to the ASC and tell them to change their entire focus and structure to accommodate the cinematographers needs. If a studio has an issue they don't go to the Academy and demand they become something totally different because the studio has a new need.
The Ves has tried a few times to get companies together to discuss a trade organization but many visual effects companies are very competitive and fiercely independent and didn't want to even consider such a thing. The Ves has also been in talks with the union.
Since none of these things happened (as of yet) the Ves decided to do what it could do to fill some of these gaps and still be true to the type of organization it is. The Ves is proceeding with doing what it can. Is there still room for a trade group or union? Yes, but at this point those aren't formed. So once again, what's the point of bashing the Ves?
Scott Ross has his work cut out for him to try to come up with a plan and sell it to all the major companies. Just as the union has to create a plan and sell to workers.
Trade organizations are typically made up of similar companies with similar needs. They don't tend to be made up of individuals. A Vfx trade organization would likely be based to some extend on the AICP and be about trying to standardize the billing and client relationship part of business. That's a different need than most workers.
Outsourcing
Here in the US we've been hurting because of the amount of tax incentives and outsourcing going on elsewhere. Some of the thinking is that the trade org or the foundation would be able to solve this problem. However both the proposed trade org and vfxfoundation are international, same as the Ves. You'll notice most unions and trade groups are regional (state, country, etc) Part of the reason for that is so they are all on the same page and can advocate their government to do things that would benefit them. (It's also cleaner from a legal perspective as well.) Being international means you can't hurt one subgroup while aiding another. I don't think London Vfx companies would be thrilled about the trade group they pay money into using that money to lobby California for more tax incentives. And we're now to the point many companies have satellite companies in other areas of the world and what were once local companies are now owned by large companies elsewhere. All of which makes it difficult to try to reduce or balance outsourcing.
Expenses
The trade group is budgeted at $3 million a year. That's a lot of money. Certainly more than the Ves. The foundation is at the other end of the spectrum. It doesn't plan to ever charge any membership fees. I've never belonged to any medium to large membership organization (professional or hobby) that didn't charge for membership. Sure, you can get a bunch of people together on the web without charging but much beyond that will require some funds at some point. Real expenses start happening when you're trying to service a number of people. Office supplies, web registration, web updates, legal fees, etc. Not everyone can volunteer full time for any length of time. Will job postings cover these costs? If you want sponsors then someone is going to have to spend time contacting companies and trying to make arrangements. One of the benefits of some type of paid membership is you separate the serious from those who aren't serious. Those who simply join everything on a lark.
Trade organization
One of the potential benefits if there was a trade organization it might it simpler for a worker group (union or other group) to negotiate. One group to deal with instead of dozens of smaller companies.
The downside is a trade organization could use it's strength in numbers to avoid any worker groups. One of the member companies could simply state that they were planning on cutting down certain worker benefits or thought certain types of jobs in Vfx were paid too much and they weren't going to be raising their rates for the next few years. There are legal issues with collusion but it can also be a gray area.
There are those that think if a trade organization benefits companies with more profits, that those profits will trickle down to the workers in benefits and pay. That's unlikely to happen. If companies make more profit then they will likely award their management and return more to their investors. They might put more money into equipment but they will still layoff people at the end of the project.
The hope is that if there is a trade organization it will create a stronger industry and if that trade organization is setup correctly more companies will be profitable and stable. And likewise it's possible it may raise the bar on bidding work and doing the work. All,of these do indirectly benefit the worker but it's important to not confuse the purpose and aim of a trade organization with a worker organization.
Ultimately it would be good for all of visual effects if in addition to the Ves there was a true trade organization and a workers group.
LinkedIn discussion
This also includes some potshots at the VES. I wrote up a response but thought it best to post here as a reference.
I'm on the board of the VES but I don't speak for the Ves so these are simply my personal views. Bob and Colin have brought up some points but I'd like to clarify a few things.
I don't see what's to be gained by bashing the VES. The Ves was formed 15 years ago to be an honorary society, similar in many ways to the ASC and the Academy of Motion Pictures. To honor professionals, to provide education and to advance the art of visual effects where possible with standards and research. And to that end, it's accomplished that. Educational programs are frequently held around the world. A couple of weeks ago there was an event covering the latest on performance capture. Many of these event videos are online at the website (upgrades in progress) The Ves created the Ves handbook and is looking to do more in this area in the future.
I pay dues to the Ves because I do this for a living. I'm a professional and the Ves is the largest group of visual effects professionals. Just as I buy books and magazines and pay to attend events related to my professional. It's a business expense.
I don't begrudge paying dues to the Academy and wouldn't begrudge paying into the ASC if I was a member. If you don't want to join the Ves then don't. If you're a Ves member and wish to see improvements then join a committee. Most of the Ves is volunteer based and that's how the Ves handbook came to be.
There are those that want the Ves to be a trade organization or a union. But that ship sailed when the Ves was founded. It may sound like an easy thing to change the structure and intent of the Ves (or other existing non profit) but it is not. To totally switch to a different type of organization would likely require dissolving the Ves and creating a totally new and different organization. The current members wouldn't be members under a trade organization and the same might apply as a union. The US government treats unions differently than standard non-union groups regarding legal issues. The same likely applies to a trade organization. There are also tax issues that would change. If a cinematographer has an issue getting paid he/she doesn't go to the ASC and tell them to change their entire focus and structure to accommodate the cinematographers needs. If a studio has an issue they don't go to the Academy and demand they become something totally different because the studio has a new need.
The Ves has tried a few times to get companies together to discuss a trade organization but many visual effects companies are very competitive and fiercely independent and didn't want to even consider such a thing. The Ves has also been in talks with the union.
Since none of these things happened (as of yet) the Ves decided to do what it could do to fill some of these gaps and still be true to the type of organization it is. The Ves is proceeding with doing what it can. Is there still room for a trade group or union? Yes, but at this point those aren't formed. So once again, what's the point of bashing the Ves?
Scott Ross has his work cut out for him to try to come up with a plan and sell it to all the major companies. Just as the union has to create a plan and sell to workers.
Trade organizations are typically made up of similar companies with similar needs. They don't tend to be made up of individuals. A Vfx trade organization would likely be based to some extend on the AICP and be about trying to standardize the billing and client relationship part of business. That's a different need than most workers.
Outsourcing
Here in the US we've been hurting because of the amount of tax incentives and outsourcing going on elsewhere. Some of the thinking is that the trade org or the foundation would be able to solve this problem. However both the proposed trade org and vfxfoundation are international, same as the Ves. You'll notice most unions and trade groups are regional (state, country, etc) Part of the reason for that is so they are all on the same page and can advocate their government to do things that would benefit them. (It's also cleaner from a legal perspective as well.) Being international means you can't hurt one subgroup while aiding another. I don't think London Vfx companies would be thrilled about the trade group they pay money into using that money to lobby California for more tax incentives. And we're now to the point many companies have satellite companies in other areas of the world and what were once local companies are now owned by large companies elsewhere. All of which makes it difficult to try to reduce or balance outsourcing.
Expenses
The trade group is budgeted at $3 million a year. That's a lot of money. Certainly more than the Ves. The foundation is at the other end of the spectrum. It doesn't plan to ever charge any membership fees. I've never belonged to any medium to large membership organization (professional or hobby) that didn't charge for membership. Sure, you can get a bunch of people together on the web without charging but much beyond that will require some funds at some point. Real expenses start happening when you're trying to service a number of people. Office supplies, web registration, web updates, legal fees, etc. Not everyone can volunteer full time for any length of time. Will job postings cover these costs? If you want sponsors then someone is going to have to spend time contacting companies and trying to make arrangements. One of the benefits of some type of paid membership is you separate the serious from those who aren't serious. Those who simply join everything on a lark.
Trade organization
One of the potential benefits if there was a trade organization it might it simpler for a worker group (union or other group) to negotiate. One group to deal with instead of dozens of smaller companies.
The downside is a trade organization could use it's strength in numbers to avoid any worker groups. One of the member companies could simply state that they were planning on cutting down certain worker benefits or thought certain types of jobs in Vfx were paid too much and they weren't going to be raising their rates for the next few years. There are legal issues with collusion but it can also be a gray area.
There are those that think if a trade organization benefits companies with more profits, that those profits will trickle down to the workers in benefits and pay. That's unlikely to happen. If companies make more profit then they will likely award their management and return more to their investors. They might put more money into equipment but they will still layoff people at the end of the project.
The hope is that if there is a trade organization it will create a stronger industry and if that trade organization is setup correctly more companies will be profitable and stable. And likewise it's possible it may raise the bar on bidding work and doing the work. All,of these do indirectly benefit the worker but it's important to not confuse the purpose and aim of a trade organization with a worker organization.
Ultimately it would be good for all of visual effects if in addition to the Ves there was a true trade organization and a workers group.
Tuesday, November 15, 2011
Dream Quest
Below are a few videos with visual effects by Dream Quest, a visual effects company I co-founded in late 1979. These are from the first 5 years. The company was later sold to Disney and changed it's name to the Secret Lab, which Disney then closed. All effects were done using 35mm film and opticals since digital effects were yet to be possible.
Great examples of cheesy car spots from the 80's.
If people are interested I could do some articles at some point on pre-digital technology and methodologies. (Motion control, matte paintings, motion graphics, etc)
Basic demo reel with new audio commentary
Great examples of cheesy car spots from the 80's.
If people are interested I could do some articles at some point on pre-digital technology and methodologies. (Motion control, matte paintings, motion graphics, etc)
Basic demo reel with new audio commentary
Friday, November 11, 2011
Visual Effects Union - IA efforts one year later
I know most of you would rather just focus on the technical issues of visual effects and many of you have an aversion to even thinking about unions after being bombarded by myths and inaccurate information. I urge you to spend some time to get informed before the opportunity passes you by.
There are a number of problems with the visual effects industry currently. Unions and Trade Organizations are some of the different approaches that visual effects artists should seriously research so they understand the issues. These issues impact everyone in visual effects and will have major impacts on the future, no matter where you live. Do you want a career in visual effects doing what you love to do or do you want your future determined by others? Will there be a real future in visual effects where artists have a balanced life, can do it until they retire if they wish and can actually continue to work where they live?
Well it's been a year since the IA announced it's plan to unionize visual effects workers and two years since they start researching the possibility of unionizing this industry. ... And still nothing has happened.
Mission not accomplished.
vfxsoldier covers this in IATSE VFX Organizing Effort One Year Later
and
fxGuide covers it in their article IATSE… One Year Later
These articles do a good job of detailing what the situation is so I won't repeat what they've already said.
As I stated in one of my previous articles, the union has a 1-2 year window to do this. So far they've blown past 1 year and already people are having a major issue with them. A year from now will anything have changed? Will it be too late? I'm still mystified why the union doesn't get serious and take a real stand. Either do it or don't do it. They've made the most tepid and ineffective approach to accomplishing anything.
I've posted a number of articles related to the unions and visual effects. But it is now entirely in the unions hands to come forward with their information and to convince everyone involved that they in fact have a workable plan for both artists and companies. They can no longer rely on bloggers and others to help get their message out, they have to take on that burden themselves since they're the ones in charge of the effort.
Suggested reading on this blog:
Lower down on the right side of this blog under the heading: VFX INDUSTRY - STATE OF THE INDUSTRY are a list of posts in chronological order. There's also a Labels section where you can click on any keyword and get all the articles that relate.
Highlights:
Pass Me a Nail discusses some of the problems.
Using the Nail lists some of the possible solutions and outlines what the union can do if they want to succeed.
Unions covers the basics of the union and what it can offer.
Response is about some of the problems and provides my response to some of the union comments.
---
A commentor on vfxsoldier listed an article, The Last Hurdle for Corporate Capitalism: Deunionizing America, that's worth checking out.
[ update: Visual Effects Union, take 2 ]
Related post:
Visual Effects Guilds
There are a number of problems with the visual effects industry currently. Unions and Trade Organizations are some of the different approaches that visual effects artists should seriously research so they understand the issues. These issues impact everyone in visual effects and will have major impacts on the future, no matter where you live. Do you want a career in visual effects doing what you love to do or do you want your future determined by others? Will there be a real future in visual effects where artists have a balanced life, can do it until they retire if they wish and can actually continue to work where they live?
Well it's been a year since the IA announced it's plan to unionize visual effects workers and two years since they start researching the possibility of unionizing this industry. ... And still nothing has happened.
Mission not accomplished.
vfxsoldier covers this in IATSE VFX Organizing Effort One Year Later
and
fxGuide covers it in their article IATSE… One Year Later
These articles do a good job of detailing what the situation is so I won't repeat what they've already said.
As I stated in one of my previous articles, the union has a 1-2 year window to do this. So far they've blown past 1 year and already people are having a major issue with them. A year from now will anything have changed? Will it be too late? I'm still mystified why the union doesn't get serious and take a real stand. Either do it or don't do it. They've made the most tepid and ineffective approach to accomplishing anything.
I've posted a number of articles related to the unions and visual effects. But it is now entirely in the unions hands to come forward with their information and to convince everyone involved that they in fact have a workable plan for both artists and companies. They can no longer rely on bloggers and others to help get their message out, they have to take on that burden themselves since they're the ones in charge of the effort.
Suggested reading on this blog:
Lower down on the right side of this blog under the heading: VFX INDUSTRY - STATE OF THE INDUSTRY are a list of posts in chronological order. There's also a Labels section where you can click on any keyword and get all the articles that relate.
Highlights:
Pass Me a Nail discusses some of the problems.
Using the Nail lists some of the possible solutions and outlines what the union can do if they want to succeed.
Unions covers the basics of the union and what it can offer.
Response is about some of the problems and provides my response to some of the union comments.
---
A commentor on vfxsoldier listed an article, The Last Hurdle for Corporate Capitalism: Deunionizing America, that's worth checking out.
[ update: Visual Effects Union, take 2 ]
Related post:
Visual Effects Guilds
Wednesday, October 26, 2011
People, not computers, create visual effects
There's an impression that is perpetuated in the media that visual effects are created entirely by computers. The human artists are left out of this narrative. The term CGI is now being used interchangeably with visual effects. The feeling among the layperson is visual effects must be easy because it's being done using computers.
These days’ writers use computers to write both novels and scripts. Computers are now in many cameras that cinematographers and photographers use. Yet we don't say the computer created the script or that the computer did the photography.
Computer graphics is one of the tools used in making visual effects and it is powerful tool with a wide scope but it’s not the only tool or only ingredient to creating effects. We also use the live action with the actors, footage of bits and pieces to combine, miniatures as necessary, our hands (with a mouse, tablet, keyboard, pencil, paint brush, etc) and our most important tool, our eyes.
The truth is visual effects is an incredibly labor intensive process. There is far more effort and time put into each shot than most people imagine, including those in production. It's inconceivable to the average person that at 24 frames per second there is still some handwork done on individual frames and people are tasked with tracing images among other time consuming work.
When BACK TO THE FUTURE 2 came out people thought that there really were hover boards being developed by Mattel. They were convinced there was no way people were tracing and painting on every frame. After all, computers were available so all of that handwork surely had to be automated by then. This was in 1989.
Years ago visual effects artists had to make fake computer graphics. For ESCAPE FROM NEW YORK we used wood and plastic blocks with white tape to represent a wire frame New York. Motion graphics cameras layered and moved artwork to give the illusion of metal logos and glints. Robert Abel's did extensive hand animation to simulate computer graphics for commercials. The public at that time were lead to believe computers were creating some great visuals. Even the motion control systems used on CLOSE ENCOUNTERS and STAR WARS didn't use true computers. They were all built from pure hardwired electronics.
Computers eventually were able to catch up to the fake computer graphics images we were creating. Computers and related software now have become one of the main tools for visual effects artists but they're still just tools that must be used by those skilled enough to do so. Anybody can type on a computer or pickup a paint brush but the ability to truly create something of professional quality in nay field takes hard work, skill, experience and some talent to master.
And yet we in the visual effects industry and the software vendors tend not to make that distinction clear. We enjoy showing off our newest tools and typically have images prepped for doing demos or dog and pony shows that don't involve the time consuming work. This leaves the press or those being presented, the impression that’s it’s easy and simple. When people see a painting they know what's gone into it to create the final results. When they see a finished visual effects shot they have no idea how much work was involved. Usually it's just wiped away with a 'it's CG' comment, as if the tool represents the work involved. The media doesn't credit brushes for the work the painter has done but they are quick to congratulate the software and hardware for creating the visuals.
With each new feature in Photoshop or new image processing demo done on YouTube people think another problem for visual effects has been solved. Many of these types of processes may work fine on a still image but can be of little value when done for a moving image projected 40 feet across. While some of these developments have made things faster and easier (I can remember hand tracking on a rear projection digitizing screen for BLUE THUNDER before we had digital compositing), they have yet to solve all of our technical issues. The computer is excellent at doing many things but most of what we do still takes a trained eye of an operator to achieve the results necessary.
The reality is it takes a lot of skilled and hardwork to create visual effects shots. In many cases the visual effects crew may eclipse the size of the live action crew, yet producers and studios still don't know why it costs so much. Few actually see the full effects crew working on their projects. Take a look at the number of credits on a visual effects film and know that this doesn't actually cover all the people that worked on the effects, just the ones that were contractually required.
Yes, computers are getting faster every year but just as a computer running twice as fast doesn’t allow a writer to create a script in half the time, it doesn’t mean shots can be produced in half the time. It’s the time reviewing, thinking and modifying that tends to take the most time. Same as on a live action set. The time the cameras are rolling is only a tiny amount of the time required to shoot a film.
Computers have made certain tasks quicker but the complexity and finessing of the work has continued to outpace the speed of the computer. It still takes hours or days in some cases to simply render certain shots. And this helps explain why a major change to a shot is a setback in terms of time and amount of man hours lost. We understand there will be creative decisions as the film is being fine tuned in post but filmmakers should be clear it's no different than asking a live action crew to do several days of reshoots at different locations. Part of this problem as mentioned before is the entire visual effects crew tends to be hidden from view so this amount of time and effort is not obvious to those outside of visual effects.
In the end thank a visual effects person for the work you see on the screen, not the computer.
Thursday, October 13, 2011
Rotoscoping - The Basics
Mike Seymour @ fxguide has written up a great article on Rotoscoping.
It covers the history, the process and the list of the various roto tools available.
I'm glad he spoke to Tom Bertino and Jack Mongovan. Both of them worked in the ILM roto department and there have been countless unsung heros there that did and do fantastic work. Dragonheart is an example of an all roto show. Quite a bit of the successful work you see today has to credit the hard working roto and paint teams. And as pointed out in the article, good roto requires good artists. While roto is considered a starter position it requires real skill and experience to do it well. Many people at ILM continued to work for years in roto because they enjoyed it and did it very well. I can't say enough good things about what they brought to vfx production. Many starting in visual effects now take all of this for granted.
I'm interviewed about Commotion, a roto/paint and compositing tool I developed and that was used around the world until purchased by Pinnacle (and then by Avid)
Links here related to Rotoscoping:
Rotoscoping - Part 1 video
Rotoscoping - Part 2 video
Rotoscoping Hair article
In real production:
Walking pants breakdown - Part 1 video (approaching a vfx sequence)
Walking pants breakdown - Part 2 video (actually rotoing and compositing)
Commotion article (I'll try to expand this in more detail in the future)
I've gotten many emails and tweets about restarting Commotion since there's still a lack of some of the tools and techniques that Commotion had over 15 years ago. As flattering as it is I'd probably prefer to focus on new challenges. Thank you.
It covers the history, the process and the list of the various roto tools available.
I'm glad he spoke to Tom Bertino and Jack Mongovan. Both of them worked in the ILM roto department and there have been countless unsung heros there that did and do fantastic work. Dragonheart is an example of an all roto show. Quite a bit of the successful work you see today has to credit the hard working roto and paint teams. And as pointed out in the article, good roto requires good artists. While roto is considered a starter position it requires real skill and experience to do it well. Many people at ILM continued to work for years in roto because they enjoyed it and did it very well. I can't say enough good things about what they brought to vfx production. Many starting in visual effects now take all of this for granted.
I'm interviewed about Commotion, a roto/paint and compositing tool I developed and that was used around the world until purchased by Pinnacle (and then by Avid)
Links here related to Rotoscoping:
Rotoscoping - Part 1 video
Rotoscoping - Part 2 video
Rotoscoping Hair article
In real production:
Walking pants breakdown - Part 1 video (approaching a vfx sequence)
Walking pants breakdown - Part 2 video (actually rotoing and compositing)
Commotion article (I'll try to expand this in more detail in the future)
I've gotten many emails and tweets about restarting Commotion since there's still a lack of some of the tools and techniques that Commotion had over 15 years ago. As flattering as it is I'd probably prefer to focus on new challenges. Thank you.
Monday, October 10, 2011
VES Board of Directors
If you are a Visual Effects Society (VES) member you should have already received email regarding the Board of Director's nomination forms. If not, make sure the VES has your current email or check your spam filters. They are due back this Friday.
Feel free to nominate yourself or someone you would think would do a good job (with their approval of course). Usually it involves being in LA once every other month or so for a meeting. Details should be on the VES web site.
Sometimes people complain about the board being made up of vfx supervisors, producers and owners. We do have people working in other positions on the board and that' swhy I'm encouraging people who are interested in getting involved to go ahead and nominate themselves. Certainly names you've heard before tend to be more likely to be voted on by other members but I do think members are more open to voting for people in all phases of production.
You can also volunteer to be on one of the various committees even if you're not on the Board of Directors.
Don't forget the Annual Member Meeting is Oct. 20 in Los Angeles.
Feel free to nominate yourself or someone you would think would do a good job (with their approval of course). Usually it involves being in LA once every other month or so for a meeting. Details should be on the VES web site.
Sometimes people complain about the board being made up of vfx supervisors, producers and owners. We do have people working in other positions on the board and that' swhy I'm encouraging people who are interested in getting involved to go ahead and nominate themselves. Certainly names you've heard before tend to be more likely to be voted on by other members but I do think members are more open to voting for people in all phases of production.
You can also volunteer to be on one of the various committees even if you're not on the Board of Directors.
Don't forget the Annual Member Meeting is Oct. 20 in Los Angeles.
Tuesday, September 27, 2011
VES Visual Effects Bill of Rights – Now what?
VES Visual Effects Bill of Rights – Now what?
By now most of you have heard and hopefully read the VES Bill of Rights. If not check it out here.
This first step was to try to define where we want to go in terms of working experience for visual effects workers worldwide. The next step is to try to implement what we can and to encourage steps to make it possible. The details are still being worked out. I’m hoping we end up with more concrete approaches and specific working conditions guidelines.
As always if you have input, feedback on the Bill of Rights or have suggestions and solutions, please send email to VES Leadership. You can post here in addition to emailing if you wish to open it to discussion by all.
The VES Membership meeting is October 20 in Los Angeles and most of the world wide sections can be linked in. The Bill of Rights will be covered. See the VES website for info.
Most people seemed to be positive about the VES making these issues more public and to at least start the ball rolling. As some have pointed out the VES doesn’t have Collective Bargaining, nor is it a union or a trade organization. Yes, that’s true, which can make it tricky but we are the largest organization of visual effects workers. The VES has been in discussions with all 3 groups of players in this industry: studios, visual effect companies and the artists themselves. Hopefully we can help broker some arrangements that will help our industry based on the bill of rights.
A few have suggested it’s a distraction. From what? The VES stand does not preclude a real union or a real trade organization. If anything, the bill of rights should make some issues clearer for everyone. As always I’m hoping by providing information and inspiration here people will join in to help find solutions.
There have been a few that suggest the VES is an elite group made of elite members. The VES is an honorary society. You need to be working in visual effects for 5 years to be a member of the VES and need 2 members to submit letters for you. I don’t think of that as elite. It does mean that VES members are experienced professionals. The main reason the VES has gotten involved in these types of issues is because there are problems in the industry and members were asking the organization to get involved and help find solutions. No other group seemed to be making progress in this area. And the approach for the VES is to try to make solutions apply to all visual effects workers.
IA Union of visual effects artists – The IA would certainly be the natural fit for visual effects workers since they cover most of the crafts in motion pictures, including the camera crew. The IA spent a year ‘researching’ visual effects industry and has now spent almost another year with someone spearheading the effort to unionize the industry. Unfortunately that has yet to result in anything. You would think they would like to get the word out to as many visual effects workers as possible and that they would try to sell the idea of the union with a clear and concise guide of the benefits and costs. They should have also been selling the idea to the visual effects companies as well. But to date most of that hasn’t happened and it hasn’t seemed like the IA has put much into this process. Many visual effects workers have either not heard of the effort or now assume it’s not happening. For more info on the IA check out their blog here.
Meanwhile the Art Directors Guild (union) has taken a definitive stand to bring in previs artists as part of their union. See their website here. And the Vancouver IA has a good website and info here. There’s also a movement for the motionographers union.
Will the IA get going or should there be an independent group that forms a new type of union?
David Rand wrote a response to the VES Bill of Rights here.
Dave is correct that the current bidding process is broken. For more info on some of the business models in visual effects check out a previous post here.
To fix this process will require many visual effects companies to get on the same page because ultimately only they can control the situation. Most visual effects companies are very competitive and fiercely independent. The VES has been encouraging the companies to meet and discuss. Many of the companies are in as much denial about issues as the workers and studios. Those doing well (especially if they’re in a location with tax incentives) see no reason to change. Why should they bother fixing the leaking roof when it’s sunny out? And of course once it’s starts raining it will be too late. The days of milk and honey will not last forever for any location. Scott Ross points out that if 4-5 of the major visual effects companies got together they could lay out some basic guidelines or requirements with their clients.
Currently there’s also been discussion among Indian visual effects workers about their situation that doesn’t sound too far off from the ones in the U.S. As I’ve said before many of these are global issues and do in fact affect you no matter where you are. And they of course make some of the same errors and false ideas as other here do.
1. Unions are only for laborers. We’re artists.
Guess what? The director’s are covered by the DGA (union). The Writers are covered by the WGA (union). The actors are covered by SAG (union). The Cinematographers are covered under the ICG (union). And so on for just about every position in motion pictures except visual effects. Are none of those other people artists? Do you gain anything by being a starving artist? Do you gain anything by not having health care insurance? Do you gain anything by not having a united group of similar artists? Can you change things by yourself and will the company change at your lone request?
2. Unions? Look at what happened to American automobile industry.
Stereotyping everything certain is not a solution nor is ignoring details of history.
Please see this previous post Using the Nail
3. Producing good work is the solution
Producing good work will certainly help you get work but it alone will not guarantee employment nor will it guarantee you fair treatment.
4. Working for free
Many starting out in this business thinks that they’ll work for free to prove themselves and then the companies will hire them. You’ve already devalued yourself when you choose to work for free. Do you think the company that hires free labor will suddenly start paying people what they’re worth and stop the practice of hiring more free labor? Each wave of new workers comes in and is willing to work for free which means those with experience now will either have to continue to work for free or will have to move on. Some visual effects companies are run by people without the passion for visual effects. Some can’t grasp the simple business solution that by hiring experienced and qualified people, treating them properly and paying them properly, they will have a true business that grows and can increase productivity and profits. Providing a quality product is of value. By simply hiring free labor they have forever tied themselves to the mediocre and will just continue being in a race to be the cheapest provider. And that’s a game that cannot be won. There will always be somewhere else cheaper, either by cost of living or incentives.
5. Working as independent contractors
One of the notions expressed is to be a remote freelancer for a visual effects company in another country. Why would a visual effects company in another country hire someone directly in another country? What experience would they have had with that person directly? Most of the major films are covered by restrictions and guidelines so images and other movie data aren’t leaked out. Can you set up to qualify? Can you do an entre shot yourself (animation, lighting, composting, roto) or will they be sending just one step of a shot to an independent worker in another country every day or every few hours? If that were to work there will be websites where artist bid on how much to do a shot. The lowest bid would likely get selected. Every independent contractor is now in a race to the bottom themselves.
As always if you have input, feedback on the Bill of Rights or have suggestions and solutions, please send email to VES Leadership. You can post here in addition to emailing if you wish to open it to discussion by all. Personally I’d prefer suggestions and solutions over complaints and reasons why none of this will work.
By now most of you have heard and hopefully read the VES Bill of Rights. If not check it out here.
This first step was to try to define where we want to go in terms of working experience for visual effects workers worldwide. The next step is to try to implement what we can and to encourage steps to make it possible. The details are still being worked out. I’m hoping we end up with more concrete approaches and specific working conditions guidelines.
As always if you have input, feedback on the Bill of Rights or have suggestions and solutions, please send email to VES Leadership. You can post here in addition to emailing if you wish to open it to discussion by all.
The VES Membership meeting is October 20 in Los Angeles and most of the world wide sections can be linked in. The Bill of Rights will be covered. See the VES website for info.
Most people seemed to be positive about the VES making these issues more public and to at least start the ball rolling. As some have pointed out the VES doesn’t have Collective Bargaining, nor is it a union or a trade organization. Yes, that’s true, which can make it tricky but we are the largest organization of visual effects workers. The VES has been in discussions with all 3 groups of players in this industry: studios, visual effect companies and the artists themselves. Hopefully we can help broker some arrangements that will help our industry based on the bill of rights.
A few have suggested it’s a distraction. From what? The VES stand does not preclude a real union or a real trade organization. If anything, the bill of rights should make some issues clearer for everyone. As always I’m hoping by providing information and inspiration here people will join in to help find solutions.
There have been a few that suggest the VES is an elite group made of elite members. The VES is an honorary society. You need to be working in visual effects for 5 years to be a member of the VES and need 2 members to submit letters for you. I don’t think of that as elite. It does mean that VES members are experienced professionals. The main reason the VES has gotten involved in these types of issues is because there are problems in the industry and members were asking the organization to get involved and help find solutions. No other group seemed to be making progress in this area. And the approach for the VES is to try to make solutions apply to all visual effects workers.
IA Union of visual effects artists – The IA would certainly be the natural fit for visual effects workers since they cover most of the crafts in motion pictures, including the camera crew. The IA spent a year ‘researching’ visual effects industry and has now spent almost another year with someone spearheading the effort to unionize the industry. Unfortunately that has yet to result in anything. You would think they would like to get the word out to as many visual effects workers as possible and that they would try to sell the idea of the union with a clear and concise guide of the benefits and costs. They should have also been selling the idea to the visual effects companies as well. But to date most of that hasn’t happened and it hasn’t seemed like the IA has put much into this process. Many visual effects workers have either not heard of the effort or now assume it’s not happening. For more info on the IA check out their blog here.
Meanwhile the Art Directors Guild (union) has taken a definitive stand to bring in previs artists as part of their union. See their website here. And the Vancouver IA has a good website and info here. There’s also a movement for the motionographers union.
David Rand wrote a response to the VES Bill of Rights here.
Dave is correct that the current bidding process is broken. For more info on some of the business models in visual effects check out a previous post here.
To fix this process will require many visual effects companies to get on the same page because ultimately only they can control the situation. Most visual effects companies are very competitive and fiercely independent. The VES has been encouraging the companies to meet and discuss. Many of the companies are in as much denial about issues as the workers and studios. Those doing well (especially if they’re in a location with tax incentives) see no reason to change. Why should they bother fixing the leaking roof when it’s sunny out? And of course once it’s starts raining it will be too late. The days of milk and honey will not last forever for any location. Scott Ross points out that if 4-5 of the major visual effects companies got together they could lay out some basic guidelines or requirements with their clients.
Currently there’s also been discussion among Indian visual effects workers about their situation that doesn’t sound too far off from the ones in the U.S. As I’ve said before many of these are global issues and do in fact affect you no matter where you are. And they of course make some of the same errors and false ideas as other here do.
1. Unions are only for laborers. We’re artists.
Guess what? The director’s are covered by the DGA (union). The Writers are covered by the WGA (union). The actors are covered by SAG (union). The Cinematographers are covered under the ICG (union). And so on for just about every position in motion pictures except visual effects. Are none of those other people artists? Do you gain anything by being a starving artist? Do you gain anything by not having health care insurance? Do you gain anything by not having a united group of similar artists? Can you change things by yourself and will the company change at your lone request?
2. Unions? Look at what happened to American automobile industry.
Stereotyping everything certain is not a solution nor is ignoring details of history.
Please see this previous post Using the Nail
3. Producing good work is the solution
Producing good work will certainly help you get work but it alone will not guarantee employment nor will it guarantee you fair treatment.
4. Working for free
Many starting out in this business thinks that they’ll work for free to prove themselves and then the companies will hire them. You’ve already devalued yourself when you choose to work for free. Do you think the company that hires free labor will suddenly start paying people what they’re worth and stop the practice of hiring more free labor? Each wave of new workers comes in and is willing to work for free which means those with experience now will either have to continue to work for free or will have to move on. Some visual effects companies are run by people without the passion for visual effects. Some can’t grasp the simple business solution that by hiring experienced and qualified people, treating them properly and paying them properly, they will have a true business that grows and can increase productivity and profits. Providing a quality product is of value. By simply hiring free labor they have forever tied themselves to the mediocre and will just continue being in a race to be the cheapest provider. And that’s a game that cannot be won. There will always be somewhere else cheaper, either by cost of living or incentives.
5. Working as independent contractors
One of the notions expressed is to be a remote freelancer for a visual effects company in another country. Why would a visual effects company in another country hire someone directly in another country? What experience would they have had with that person directly? Most of the major films are covered by restrictions and guidelines so images and other movie data aren’t leaked out. Can you set up to qualify? Can you do an entre shot yourself (animation, lighting, composting, roto) or will they be sending just one step of a shot to an independent worker in another country every day or every few hours? If that were to work there will be websites where artist bid on how much to do a shot. The lowest bid would likely get selected. Every independent contractor is now in a race to the bottom themselves.
As always if you have input, feedback on the Bill of Rights or have suggestions and solutions, please send email to VES Leadership. You can post here in addition to emailing if you wish to open it to discussion by all. Personally I’d prefer suggestions and solutions over complaints and reasons why none of this will work.
Wednesday, August 31, 2011
Letter to Animation Guild
The Animation Guild received an email for someone in the VFX industry who documents some of the issues faced by typical vfx workers. This is somewhat of an extreme but there's certainly a basis of truth.
If any of this sounds familiar then I urge you to make your voice heard. Don't expect someone else to come and fix the problem. Don't expect the company you work for to fix the problems and don't expect the studios to step forward and fix the problems.
Email VES Leadership group
If any of this sounds familiar then I urge you to make your voice heard. Don't expect someone else to come and fix the problem. Don't expect the company you work for to fix the problems and don't expect the studios to step forward and fix the problems.
Email VES Leadership group
Saturday, August 27, 2011
VFX Wages Discussion
This post is in response to a comment on a previous post. See this post and follow the comments there for the full original comment. (Toward the end from Anonymous) Unfortunately once I wrote this response it was bigger than a comment could be so I've done it as a post.
Here's part of that comment (focused for this post) but request you read it all to see it in context:
"Here is my thought on this: We got greedy.
During boom times artist rates went up and up. I now make more money than most doctors. I know plenty of others who do just as well. We are the most expensive part of production on most films. While single individuals may get larger paychecks- the director, producer, lead actor, a massive amount of money goes to VFX.
We are too expensive. I make, on average, 5 to 10 times more than other Americans. Why?! It just happened that way over time, I didn't ever expect to be doing this well as an employee of another company. I am well beyond my own expectations."
We got greedy? I don't recall a mass rally outside vfx company offices calling for huge pay increases. I don't recall any company just deciding one day they will pay more simply because vfx workers would like more.
I'm still not clear why people are apologetic for being paid above a national average as if it were a sin. Especially if you're actually a skilled and knowledgable worker creating something or performing a real service that will ultimately be very profitable. According to some of these people we're all so well paid none of us should get health care or any other benefits.
I'd like to think that decades of experience in a very difficult creative and technical industry would amount to something. That the time and effort put into keeping up with quickly changing software, images and techniques would be of some value. I'd like to be paid more than I was 20+ years ago. Some movies makes hundred of millions of dollars (and some even more than a billion dollars). Movies that we helped create. I'd like to think that those of us who do this are paid enough money to have a reasonable home in the cities we work in, that we can afford to raise children, that we don't have to worry about health care bills, that we don't have to scrape the bottom of the barrel to try to retire one day, and that we we're paid enough so we don't have to worry if we don't get a call for a project next week. Is that being greedy? I don't think so.
There will always be people who make more than you and people who make less than you. Bill Gates isn't always the top of the list. You may not have made any money last year but there's likely someone who hasn't made money for the last 2+ years.
The average US CEO of a public company makes over $10 million a year. [To put that in perspective many CEOs earn as much in 1 day as a typical US worker does in an entire year. 1 Day] Are they that much smarter? Harder working? Working longer hours? From my experience you could actually replace most CEO's with a rock and it wouldn't make a bit of difference to the bottom line since it tends to be the workers and managers that handle the actual running and in many cases the non-CEOs are the ones that are forward thinking. (Steve Jobs is a rare exception) I suspect many of these CEO's have assistants that probably make more than a CG supervisor. At the other end of the spectrum are people working multiple jobs cleaning toilets and digging ditches just to try to make a living for themselves and their family. There are some teachers and regional pilots that make less than the poverty level of income. Is any of that fair? No, but all we have at most is a tiny bit of control of our own wages.
I know someone who works in non-vfx that probably puts in half the hours I do and is paid 2-4 times what I make. I know a non-vfx software engineer who was very well paid at a permanent position. More than you are. He received a $300,000 bonus when they shipped on time. When was the last time you got paid a bonus? Any type of bonus? [Supreme Court Clerks now receive over $280,000 signing bonus when they go to work at a law firm.] Most of the time people put in an incredible number of hours at the end of a project and may not even receive a thank you from the company. If the company is really feeling up to it they might have a wrap party as a thank you.
Drive around Los Angeles and look at the pricing of homes in areas you wouldn't mind living in. Base price of a reasonable house (small 2 bedroom) in LA is over $800,000. Many start at $2 million and go up. There are a lot of people in LA making a lot of money and they're not vfx people. Do you think all of these people are going without health care and pensions? Do they all feel guilty?
“We are the most expensive part of production on most films.”
Well we’re a large expense IF the show is a VFX show. I’d hardly call vfx the most expensive part of most films. Most films don't have extensive vfx. Most films burn through $100,000-$300,000 per day while shooting. Most films spend tens of millions of dollars on advertising and promotion. Most films employee at least some actors making $2 million to $20 million. On most of the vfx projects I’ve worked on, vfx typically use less than ½ the budget, in many cases 1/3 to ¼. Published movie budgets are seldom accurate.
“While single individuals may get larger paychecks- the director, producer, lead actor, a massive amount of money goes to VFX.”
I cover a few other people below. You act as if it’s our fault and something we should be ashamed of. A studio chooses which projects they feel will make money in the box office. Most of the top 20 moneymaking movies have made extensive use of vfx. It wasn’t the only reason for their financial success but vfx is part of the reason and with some films, it’s very high on the reasons. The studio chooses to make a vfx film. It’s not something we’re pushing on them. A studio makes tradeoffs and may choose to spend more on vfx than on A actors on some projects and just the opposite on other projects. And the thing about vfx is it’s not just one person. You’ve lumped in all the vfx people costs (which can be hundreds of people) against individuals. When you compare the costs of a full shooting crew on a large vfx movie (including 2nd unit and all the support teams) shooting for 6 months then it’s a much different balance between the ‘expensive vfx crew’ and the shooting crew. The live action crew can be just as expensive, if not more expensive.
Film business: In addition there are quite a few others in the film business making above the national average by week/day/hour. Studio executive, jr executives, editors, DPs, production designers, stunt supervisor, special effects supervisors, sound mixers, DI colorists, etc.
I haven't checked salaries lately but suspect most of the crew make above the national average as well. And why do people make good money in Hollywood? Because they're working on large projects with very good profits if done well. A VFX heavy film will likely make hundreds of millions in profit if done correctly. These people have developed skills and experience that can't be simple bought or learned in a course. They go from project to project so they're not permanently employed. They freelance. They have to be paid more simply to average out and be able to afford the same thing that someone is permanently employed can afford. They put in long hours. Shooting days are 12hr days. Shooting weeks can be 5-7 days. They live in LA and similar areas where the cost of living and housing is above the national average.
Realize the national averages really haven't gone up much in the last 20-30 years. Most pay has been relatively stagnant with 2% increases a year, if that. Even while many companies have become much more profitable. So where does the extra money go to? The CEO, upper management and shareholders. CEO's 30 years ago made approx 30x their average employee. It's now over 300x. GE made record profits last year. Paid no taxes. What do they want to do? Cut wages and benefits of their workers.
Why were Verizon workers on strike? Verizon is making very good profits, paying their upper management very, very well and not paying taxes. And yet they're still asking their workers to reduce there benefits. "That loss of health benefits and other givebacks in the proposed contract would net Verizon annually about $1 billion, or $20,000 per worker, according to the unions." ref And these are benefits that had already been agreed to by both sides and awarded. Now the company wants to reduce them.
From the sounds of it you're in a very sweet spot. You seem to be permanently employed, paid a very high salary and get paid overtime. "I now make more money than most doctors." Really? You make more than most doctors? What you’re describing is not the average or typical for the majority of vfx workers.
If we're going to even try to compare any of this to the national average there are a few things to do. So let's take a look at what an average vfx worker deals with. First calculate rates 1 1/2x for over 8 hrs and 2x for over 12 hrs to work out an average 40hr week. Most of us work 50 hrs to 90 hr weeks but for comparison we have to put it relative to a 40hr workweek. Don't forget those who work a flat rate (no overtime pay). Use the same formula and see how that works out, especially once you hit 90hrs+ a week. VFX workers who do work a lot of overtime end up sacrificing time with their families and their health in the long run that many other jobs do not.
"So many of my friends are unemployed right now."
Even at ILM it was common for people to be laid off for 3-6 months during the 'slow' periods and now we're seeing people go for even longer stretches. So cut the cut the pay period from 12 months to 6-9 months. Now calculate a yearly average or 5-year average.
Oddly when you take that high salary and try to stretch it from 6 months to cover a year, it doesn't look nearly as large. And in the case of most vfx workers there's no guarantee when they will be re-hired. It may seem like the perfect time to take an extended vacation but if you don't know whether you're working again in a few weeks or a year from now it's hard to plan.
Now not everyone in vfx is covered for health care or pension. Even if they were covered if they're off for an extended time or have to switch companies they have to start over. So calculate in the cost of Cobra insurance or self-insurance. Also calculate a pension fund. Not cheap. And we're adding this because most full time jobs include these as part of their compensation.
If a vfx worker has to work outside of town for a given period and has to cover related expenses (travel, boarding, phone calls, etc) then deduct those as well from the vfx wages. A percentage of non-vfx workers work for companies that offer other benefits. (Discounts on products, discounts on services, profit sharing plans, stock options, bonuses, etc) So now compare the full compensation package of the national average with the average vfx worker and what it costs to create that same level of benefits. That average vfx salary that may have looked huge on the surface is likely to be much closer to the national average than it first appeared.
Are there 1000's of people who could do what you do? Can they step into the job you're currently at and do just as well? Do you have years of experience, expertise and skills that means you're much more likely to do something much faster and to avoid the pitfalls? Do you have to take responsibility for the project or a team of people? Do you have to manage people? Do you save money for the company even with your salary? When I work I don't feel bad about being paid well because not only do I accomplish what needs to be done I usually end up saving the vfx company or studio a few times my salary just on avoided expenses. (Avoiding or minimizing costly sets, having the option not to travel the entire cast and crew halfway around the world, improving the pipeline, minimizing the amount of overtime that I can, doing multiple tasks, doing my own mockups, etc)
Companies can and do at times hire poor employees/managers that not only don't accomplish what they need to but that cause losses by making bad decisions. In the end the roto and paint team may have to solve problems because of mistakes made early in the process.
So how does this all work in today’s vfx world? Yes, workers in China and India are paid less but they typically have a lower cost of living. And even in these places as the workers get more skills and experience their wages are going up.
Why don’t all vfx workers in California cut their wages in half? Certainly that would have an impact and counter the imbalanced tax incentives? Yes, but probably not in the way most people might expect. First, many vfx companies would be unlikely to pass all of that savings on to the studios. They would want to keep some to increase their profits and increase salaries of their management. Second, the studios would ask the vfx companies for even lower bids, because after all, the workers cost less. Those places in other countries and areas would be forced to follow suit because the companies that employee them and the studios would point out those in LA don’t make as much. End result could simply be evenly lower bids with the tax incentives still in place and the work still going else where. And now everyone is paid less. Doesn't seem to be much of an advantage.
Why did we end up with the salaries in vfx that we do?
" It just happened that way over time.." Nope.
First off there is no standard. People tend to assume there is because some of the bigger companies have somewhat similar rates. But without a union there’s really no standard rates and people doing the same job at different companies (or even within the same company) could be paid drastically different rates. (and of course different locations and countries differ as well).
Companies don’t just randomly choose to pay a higher rate for workers. Nor do they tend to volunteer to pay increases and other job incentives.
One of the reasons why people are paid the amount they are because when digital vfx started it was made up mainly of union people. ILM was all union when I went to work there. Say what you will about the unions but the fact is in many industries you’re paid a reasonable rate because a group of people organized and asked for given rates. There was a shift to digital as time progressed but many of the people shifted to similar jobs in the digital world if they could. That also meant that 2D animators (with skills and existing experience) tended to move into 3D. Once again people who were already up to speed and able to accomplish the work are worth far more than a number of less expensive people with no experience and that would require training and hand holding.
Those working in VFX are also paid what they are because similar jobs in other industries might pay well. Digital vfx were ramping up as Silicon Valley was at full speed. Whether it’s in graphic arts, software development or other areas, if companies wish to either hire someone from that other industry or want to retain people from going to another industry, then they have to pay a competitive rate. Do you think everyone at Apple, Google, Microsoft and Adobe are being paid minimum wage or even the national average?
And the other reason for the pay to be the way it is because there was a lack of skilled, experienced talent at the beginning and some companies would essentially compete to get key people. The result is there was a spike at one point but much of that has already been scaled back. Will there be more scaling back in the future? Possibly.
But don’t forget the US Justice Dept found that there was collusion between ILM and Pixar regarding non-competing for animators. So that created artificially lower rates for animators and prevented some opportunities for them.
As stated the reason why the rates are where they are elsewhere is due to the rates that have been paid to key vfx people in California. We’re now seeing a flood of vfx students trying to be employed. Many of these people are willing to work for anything, including free and minimum wage. Some would be happy to pay the first year if necessary. What they don’t understand is by working free or cheap they can end up pushing all wages down, including their future earnings. They may find next year, once they’ve gained more experience, they aren’t given a raise simply because this years batch of students are willing to work for even less. And some producers prey on these types of people and employ essentially students to do their work for free or cheap.
The point here is that we should be looking for solutions to the problems of the vfx industry. Outsourcing and tax incentives are some of the problems. Overtime and overtime pay is another issue. Being forced to be an independent contractor is another problem. I'm sure we could list many more but I'm hoping we focus discussions on potential solutions; throwing out and discussing possible ideas. Simply shrugging our shoulders and saying "woe is me" is not a solution.
It's not easy. This is a global problem but hopefully we'll be able to come up with some ideas to balance this out.
Thanks.
Here's part of that comment (focused for this post) but request you read it all to see it in context:
"Here is my thought on this: We got greedy.
During boom times artist rates went up and up. I now make more money than most doctors. I know plenty of others who do just as well. We are the most expensive part of production on most films. While single individuals may get larger paychecks- the director, producer, lead actor, a massive amount of money goes to VFX.
We are too expensive. I make, on average, 5 to 10 times more than other Americans. Why?! It just happened that way over time, I didn't ever expect to be doing this well as an employee of another company. I am well beyond my own expectations."
We got greedy? I don't recall a mass rally outside vfx company offices calling for huge pay increases. I don't recall any company just deciding one day they will pay more simply because vfx workers would like more.
I'm still not clear why people are apologetic for being paid above a national average as if it were a sin. Especially if you're actually a skilled and knowledgable worker creating something or performing a real service that will ultimately be very profitable. According to some of these people we're all so well paid none of us should get health care or any other benefits.
I'd like to think that decades of experience in a very difficult creative and technical industry would amount to something. That the time and effort put into keeping up with quickly changing software, images and techniques would be of some value. I'd like to be paid more than I was 20+ years ago. Some movies makes hundred of millions of dollars (and some even more than a billion dollars). Movies that we helped create. I'd like to think that those of us who do this are paid enough money to have a reasonable home in the cities we work in, that we can afford to raise children, that we don't have to worry about health care bills, that we don't have to scrape the bottom of the barrel to try to retire one day, and that we we're paid enough so we don't have to worry if we don't get a call for a project next week. Is that being greedy? I don't think so.
There will always be people who make more than you and people who make less than you. Bill Gates isn't always the top of the list. You may not have made any money last year but there's likely someone who hasn't made money for the last 2+ years.
The average US CEO of a public company makes over $10 million a year. [To put that in perspective many CEOs earn as much in 1 day as a typical US worker does in an entire year. 1 Day] Are they that much smarter? Harder working? Working longer hours? From my experience you could actually replace most CEO's with a rock and it wouldn't make a bit of difference to the bottom line since it tends to be the workers and managers that handle the actual running and in many cases the non-CEOs are the ones that are forward thinking. (Steve Jobs is a rare exception) I suspect many of these CEO's have assistants that probably make more than a CG supervisor. At the other end of the spectrum are people working multiple jobs cleaning toilets and digging ditches just to try to make a living for themselves and their family. There are some teachers and regional pilots that make less than the poverty level of income. Is any of that fair? No, but all we have at most is a tiny bit of control of our own wages.
I know someone who works in non-vfx that probably puts in half the hours I do and is paid 2-4 times what I make. I know a non-vfx software engineer who was very well paid at a permanent position. More than you are. He received a $300,000 bonus when they shipped on time. When was the last time you got paid a bonus? Any type of bonus? [Supreme Court Clerks now receive over $280,000 signing bonus when they go to work at a law firm.] Most of the time people put in an incredible number of hours at the end of a project and may not even receive a thank you from the company. If the company is really feeling up to it they might have a wrap party as a thank you.
Drive around Los Angeles and look at the pricing of homes in areas you wouldn't mind living in. Base price of a reasonable house (small 2 bedroom) in LA is over $800,000. Many start at $2 million and go up. There are a lot of people in LA making a lot of money and they're not vfx people. Do you think all of these people are going without health care and pensions? Do they all feel guilty?
“We are the most expensive part of production on most films.”
Well we’re a large expense IF the show is a VFX show. I’d hardly call vfx the most expensive part of most films. Most films don't have extensive vfx. Most films burn through $100,000-$300,000 per day while shooting. Most films spend tens of millions of dollars on advertising and promotion. Most films employee at least some actors making $2 million to $20 million. On most of the vfx projects I’ve worked on, vfx typically use less than ½ the budget, in many cases 1/3 to ¼. Published movie budgets are seldom accurate.
“While single individuals may get larger paychecks- the director, producer, lead actor, a massive amount of money goes to VFX.”
I cover a few other people below. You act as if it’s our fault and something we should be ashamed of. A studio chooses which projects they feel will make money in the box office. Most of the top 20 moneymaking movies have made extensive use of vfx. It wasn’t the only reason for their financial success but vfx is part of the reason and with some films, it’s very high on the reasons. The studio chooses to make a vfx film. It’s not something we’re pushing on them. A studio makes tradeoffs and may choose to spend more on vfx than on A actors on some projects and just the opposite on other projects. And the thing about vfx is it’s not just one person. You’ve lumped in all the vfx people costs (which can be hundreds of people) against individuals. When you compare the costs of a full shooting crew on a large vfx movie (including 2nd unit and all the support teams) shooting for 6 months then it’s a much different balance between the ‘expensive vfx crew’ and the shooting crew. The live action crew can be just as expensive, if not more expensive.
Film business: In addition there are quite a few others in the film business making above the national average by week/day/hour. Studio executive, jr executives, editors, DPs, production designers, stunt supervisor, special effects supervisors, sound mixers, DI colorists, etc.
I haven't checked salaries lately but suspect most of the crew make above the national average as well. And why do people make good money in Hollywood? Because they're working on large projects with very good profits if done well. A VFX heavy film will likely make hundreds of millions in profit if done correctly. These people have developed skills and experience that can't be simple bought or learned in a course. They go from project to project so they're not permanently employed. They freelance. They have to be paid more simply to average out and be able to afford the same thing that someone is permanently employed can afford. They put in long hours. Shooting days are 12hr days. Shooting weeks can be 5-7 days. They live in LA and similar areas where the cost of living and housing is above the national average.
Realize the national averages really haven't gone up much in the last 20-30 years. Most pay has been relatively stagnant with 2% increases a year, if that. Even while many companies have become much more profitable. So where does the extra money go to? The CEO, upper management and shareholders. CEO's 30 years ago made approx 30x their average employee. It's now over 300x. GE made record profits last year. Paid no taxes. What do they want to do? Cut wages and benefits of their workers.
Why were Verizon workers on strike? Verizon is making very good profits, paying their upper management very, very well and not paying taxes. And yet they're still asking their workers to reduce there benefits. "That loss of health benefits and other givebacks in the proposed contract would net Verizon annually about $1 billion, or $20,000 per worker, according to the unions." ref And these are benefits that had already been agreed to by both sides and awarded. Now the company wants to reduce them.
From the sounds of it you're in a very sweet spot. You seem to be permanently employed, paid a very high salary and get paid overtime. "I now make more money than most doctors." Really? You make more than most doctors? What you’re describing is not the average or typical for the majority of vfx workers.
If we're going to even try to compare any of this to the national average there are a few things to do. So let's take a look at what an average vfx worker deals with. First calculate rates 1 1/2x for over 8 hrs and 2x for over 12 hrs to work out an average 40hr week. Most of us work 50 hrs to 90 hr weeks but for comparison we have to put it relative to a 40hr workweek. Don't forget those who work a flat rate (no overtime pay). Use the same formula and see how that works out, especially once you hit 90hrs+ a week. VFX workers who do work a lot of overtime end up sacrificing time with their families and their health in the long run that many other jobs do not.
"So many of my friends are unemployed right now."
Even at ILM it was common for people to be laid off for 3-6 months during the 'slow' periods and now we're seeing people go for even longer stretches. So cut the cut the pay period from 12 months to 6-9 months. Now calculate a yearly average or 5-year average.
Oddly when you take that high salary and try to stretch it from 6 months to cover a year, it doesn't look nearly as large. And in the case of most vfx workers there's no guarantee when they will be re-hired. It may seem like the perfect time to take an extended vacation but if you don't know whether you're working again in a few weeks or a year from now it's hard to plan.
Now not everyone in vfx is covered for health care or pension. Even if they were covered if they're off for an extended time or have to switch companies they have to start over. So calculate in the cost of Cobra insurance or self-insurance. Also calculate a pension fund. Not cheap. And we're adding this because most full time jobs include these as part of their compensation.
If a vfx worker has to work outside of town for a given period and has to cover related expenses (travel, boarding, phone calls, etc) then deduct those as well from the vfx wages. A percentage of non-vfx workers work for companies that offer other benefits. (Discounts on products, discounts on services, profit sharing plans, stock options, bonuses, etc) So now compare the full compensation package of the national average with the average vfx worker and what it costs to create that same level of benefits. That average vfx salary that may have looked huge on the surface is likely to be much closer to the national average than it first appeared.
Are there 1000's of people who could do what you do? Can they step into the job you're currently at and do just as well? Do you have years of experience, expertise and skills that means you're much more likely to do something much faster and to avoid the pitfalls? Do you have to take responsibility for the project or a team of people? Do you have to manage people? Do you save money for the company even with your salary? When I work I don't feel bad about being paid well because not only do I accomplish what needs to be done I usually end up saving the vfx company or studio a few times my salary just on avoided expenses. (Avoiding or minimizing costly sets, having the option not to travel the entire cast and crew halfway around the world, improving the pipeline, minimizing the amount of overtime that I can, doing multiple tasks, doing my own mockups, etc)
Companies can and do at times hire poor employees/managers that not only don't accomplish what they need to but that cause losses by making bad decisions. In the end the roto and paint team may have to solve problems because of mistakes made early in the process.
So how does this all work in today’s vfx world? Yes, workers in China and India are paid less but they typically have a lower cost of living. And even in these places as the workers get more skills and experience their wages are going up.
Why don’t all vfx workers in California cut their wages in half? Certainly that would have an impact and counter the imbalanced tax incentives? Yes, but probably not in the way most people might expect. First, many vfx companies would be unlikely to pass all of that savings on to the studios. They would want to keep some to increase their profits and increase salaries of their management. Second, the studios would ask the vfx companies for even lower bids, because after all, the workers cost less. Those places in other countries and areas would be forced to follow suit because the companies that employee them and the studios would point out those in LA don’t make as much. End result could simply be evenly lower bids with the tax incentives still in place and the work still going else where. And now everyone is paid less. Doesn't seem to be much of an advantage.
Why did we end up with the salaries in vfx that we do?
" It just happened that way over time.." Nope.
First off there is no standard. People tend to assume there is because some of the bigger companies have somewhat similar rates. But without a union there’s really no standard rates and people doing the same job at different companies (or even within the same company) could be paid drastically different rates. (and of course different locations and countries differ as well).
Companies don’t just randomly choose to pay a higher rate for workers. Nor do they tend to volunteer to pay increases and other job incentives.
One of the reasons why people are paid the amount they are because when digital vfx started it was made up mainly of union people. ILM was all union when I went to work there. Say what you will about the unions but the fact is in many industries you’re paid a reasonable rate because a group of people organized and asked for given rates. There was a shift to digital as time progressed but many of the people shifted to similar jobs in the digital world if they could. That also meant that 2D animators (with skills and existing experience) tended to move into 3D. Once again people who were already up to speed and able to accomplish the work are worth far more than a number of less expensive people with no experience and that would require training and hand holding.
Those working in VFX are also paid what they are because similar jobs in other industries might pay well. Digital vfx were ramping up as Silicon Valley was at full speed. Whether it’s in graphic arts, software development or other areas, if companies wish to either hire someone from that other industry or want to retain people from going to another industry, then they have to pay a competitive rate. Do you think everyone at Apple, Google, Microsoft and Adobe are being paid minimum wage or even the national average?
And the other reason for the pay to be the way it is because there was a lack of skilled, experienced talent at the beginning and some companies would essentially compete to get key people. The result is there was a spike at one point but much of that has already been scaled back. Will there be more scaling back in the future? Possibly.
But don’t forget the US Justice Dept found that there was collusion between ILM and Pixar regarding non-competing for animators. So that created artificially lower rates for animators and prevented some opportunities for them.
As stated the reason why the rates are where they are elsewhere is due to the rates that have been paid to key vfx people in California. We’re now seeing a flood of vfx students trying to be employed. Many of these people are willing to work for anything, including free and minimum wage. Some would be happy to pay the first year if necessary. What they don’t understand is by working free or cheap they can end up pushing all wages down, including their future earnings. They may find next year, once they’ve gained more experience, they aren’t given a raise simply because this years batch of students are willing to work for even less. And some producers prey on these types of people and employ essentially students to do their work for free or cheap.
The point here is that we should be looking for solutions to the problems of the vfx industry. Outsourcing and tax incentives are some of the problems. Overtime and overtime pay is another issue. Being forced to be an independent contractor is another problem. I'm sure we could list many more but I'm hoping we focus discussions on potential solutions; throwing out and discussing possible ideas. Simply shrugging our shoulders and saying "woe is me" is not a solution.
It's not easy. This is a global problem but hopefully we'll be able to come up with some ideas to balance this out.
Thanks.
Monday, August 22, 2011
VES 2.0 feedback requested
The VES 2.0 group (Visual Effects Society) continues to meet and explore options for the vfx industry.
I'd like to once again ask all vfx artists and companies to submit their thoughts, ideas, solutions and concerns to the VES Leadership group. Email VES Leadership group You don't have to be a member to submit. You can also post in the comments here or email me directly. **
If you're in the vfx industry you do really owe it to yourself and your future to submit your thoughts. Now is the time, not 6 months from now.
Please consider reading the links on the right side under VFX INDUSTRY - STATE OF THE INDUSTRY heading if you haven't already. I posted these to try to provide information and different perspectives of some of the issues. Pass Me That Nail covers some of the key problems with the vfx industry. Using the Nail covers some of the VES 2.0 issues.
I also urge vfx artists to become informed and try to make open minded and balanced decisions and comments. It's easy to repeat the same phrases and become narrow minded, especially under the circumstances. Also keep in mind incentives will not last forever anywhere. Don't assume smooth sailing where you're currently working to last your lifetime. What you perceive as only a US problem could be your problem in the next year or two.
For more info: Visual Effects Society
VES 2.0 Letter to VFX Industry
From Shoot magazine report on VES 2.0
[**Update: Please check out the comments below (you may have to click on the comments link.
I'm listing a few of the types of things to consider in your note and included a few questions from the IA (snippets from the comments)
What are the problems people are facing? You may have some issues I haven't touched on or you may have a totally different priority.
How would you like to see the vfx industry structured?
What should the vfx companies be doing differently to make it better for the workers?
What should the studios be doing to make it better for the workers?
What about the studio and vfx company relationships?
People writing in should mention their views on the IA and why.
What's their views on a vfx trade organization?
From the IA:
"..we continue to seek out some direction as to the type of organization the rank and file really want?
One integrated visual effects organization?
An independent guild?
Affiliation with several different IATSE locals (matching job-skills) such as camera, editorial, animation, or art directors?
We could charter a guild and have it function as a separate entity, benefitting from the experience and strength of the International, and, where appropriate, adopting the Major Studios Basic Agreement. We just need some direction from the affected parties."]
There's also a relatively new take on this which is to focus on management.
While I agree that improving management would help I don't see it being the all in one solution.
Some vfx companies are managed better than others but I don't expect to cut the required time in half or another 20% in profits simply by improved management. Many vfx companies are already reasonably run and while there's room for improvement it's unlikely to be huge. The variables from directors and studios tend to be larger than these gains. Likewise don't assume just because a company is more efficient that it will make more money (it may just get more work) or that benefits, wages and working conditions will improve for employees. The trickle down theory doesn't work. Most general US companies are making 20% more in profits the last couple of years but continue to reduce their workforce and reduce employee benefits. Those added profits go to the executives and shareholders, not the people being more efficient.
What we'll need is improvements/changes to the 3 key parts of this issue (Studios, vfx companies and vfx workers) to get a balance that works and is sustainable.
[Please click on Comments link below if comments aren't automatically shown in your browser]
I'd like to once again ask all vfx artists and companies to submit their thoughts, ideas, solutions and concerns to the VES Leadership group. Email VES Leadership group You don't have to be a member to submit. You can also post in the comments here or email me directly. **
If you're in the vfx industry you do really owe it to yourself and your future to submit your thoughts. Now is the time, not 6 months from now.
Please consider reading the links on the right side under VFX INDUSTRY - STATE OF THE INDUSTRY heading if you haven't already. I posted these to try to provide information and different perspectives of some of the issues. Pass Me That Nail covers some of the key problems with the vfx industry. Using the Nail covers some of the VES 2.0 issues.
I also urge vfx artists to become informed and try to make open minded and balanced decisions and comments. It's easy to repeat the same phrases and become narrow minded, especially under the circumstances. Also keep in mind incentives will not last forever anywhere. Don't assume smooth sailing where you're currently working to last your lifetime. What you perceive as only a US problem could be your problem in the next year or two.
For more info: Visual Effects Society
VES 2.0 Letter to VFX Industry
From Shoot magazine report on VES 2.0
[**Update: Please check out the comments below (you may have to click on the comments link.
I'm listing a few of the types of things to consider in your note and included a few questions from the IA (snippets from the comments)
What are the problems people are facing? You may have some issues I haven't touched on or you may have a totally different priority.
How would you like to see the vfx industry structured?
What should the vfx companies be doing differently to make it better for the workers?
What should the studios be doing to make it better for the workers?
What about the studio and vfx company relationships?
People writing in should mention their views on the IA and why.
What's their views on a vfx trade organization?
From the IA:
"..we continue to seek out some direction as to the type of organization the rank and file really want?
One integrated visual effects organization?
An independent guild?
Affiliation with several different IATSE locals (matching job-skills) such as camera, editorial, animation, or art directors?
We could charter a guild and have it function as a separate entity, benefitting from the experience and strength of the International, and, where appropriate, adopting the Major Studios Basic Agreement. We just need some direction from the affected parties."]
There's also a relatively new take on this which is to focus on management.
While I agree that improving management would help I don't see it being the all in one solution.
Some vfx companies are managed better than others but I don't expect to cut the required time in half or another 20% in profits simply by improved management. Many vfx companies are already reasonably run and while there's room for improvement it's unlikely to be huge. The variables from directors and studios tend to be larger than these gains. Likewise don't assume just because a company is more efficient that it will make more money (it may just get more work) or that benefits, wages and working conditions will improve for employees. The trickle down theory doesn't work. Most general US companies are making 20% more in profits the last couple of years but continue to reduce their workforce and reduce employee benefits. Those added profits go to the executives and shareholders, not the people being more efficient.
What we'll need is improvements/changes to the 3 key parts of this issue (Studios, vfx companies and vfx workers) to get a balance that works and is sustainable.
[Please click on Comments link below if comments aren't automatically shown in your browser]
Thursday, August 04, 2011
2D to 3D Conversions
2D to 3D conversions
2D to 3D conversion is the process of converted a film from 2D (normal film) to a 3D (or stereo) film for viewing with a stereographic viewing system. (polarized glasses, shuttered glasses, special screen, etc)
Conversion had gained a bad name after Clash of the Titans and a few other films that had been rushed through at the last minute. Just as you can’t do a 2000 shot vfx show in 6 weeks with great quality, you can’t do a quality conversion of an entire movie in 6 weeks.
Most people think of a 2D to 3D conversion as a factory type of operation but it’s far from it if it’s to be done right. Much of my past year was spent working for Legend3D dealing with some interesting and very technically challenging problems. Legend had completed work on Alice (Tea party sequence and Drink Me sequence) and done the conversion on the first 3 Shrek films to be released for 3D TV.
Legend was starting on Transformers: Dark of the Moon and would need help with the extensive visual effects shots. Legend 3D ended up converting over 77 minutes of footage for Transformers. Most of the vfx shots were converted in a 4-5 month time span. The converted footage was a mix of visual effects shots (from ILM and DD) and non-vfx shots. In most cases these would be shots intercut back and forth with original stereo footage shot on location. This raised the complexity much more than if the entire film had been converted. Just as doing vfx in a live action film is a different task than creating an animated film (or virtual sequence).
This was also more difficult than doing a library title or a film that was completed. The same issues vfx companies face of ever changing edits and creative directions was in full force. In addition the vfx shots were changing and so we had to follow very closely and track all changes such that we could turn around the final converted shots within a day or two of deliver of the last vfx shots.
While at Legend I helped them to develop the pipelines required to do this type of work efficiently and at as high of quality level as possible. Legend had some proprietary software and techniques for different stages in the process. I wrote a number of specialty Nuke plugins and scripts to help leverage these proprietary software packages with existing software. I worked with a number of artists and developers at Legend to review and analyze the shots as they were delivered and worked closely with ILM and DD to get the proper elements that would be required. Both technical and creative issues had to be examined to get the best possible quality. Overall this represented a number of challenges that were unique to the world of stereo conversion and we were able to raise the quality of conversion to a new level.
In what follows I’ll cover the basics of the conversion process. I won’t be able to go into specific detail to due the proprietary nature but hopefully this will provide an understanding of the process.
Conversion versus shooting stereo
Many people are under the mistaken belief that shooting stereo is superior to conversion in every way. The key to getting great 3D is to design, shoot and edit for 3D, even if it will be converted. If you shoot stereo but don’t consider the 3D world then you will have problems.
In its current form 3D is a stereo process, which means creating imagery for each eye. It’s actually a big cheat when projecting 2 different images to simulate the look of the 3D because your eyes are always focused on the screen but objects appear at other depths. This is a fairly unnatural process.
The following are some of the basic pros and cons. As with everything there are tradeoffs to be made. Shooting stereo does capture the images in 3D but many of the parameters are locked in at the time of shooting. Shooting stereo also requires a fair bit of post work to actually get the images to match. Films that are shot in stereo will still require a certain number of conversion shots just due to problematic shots or limitations while shooting. This was the case even with Avatar.
Shooting Stereo – PROS
• Capturing stereo images at time of shooting
• Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.
• Preview stereo images on set and on location
• Footage can be edited and reviewed in stereo context with the correct equipment
Shooting Stereo - CONS
• Requires specialized camera rigs
• Camera rigs contain 2 cameras so are larger than non-stereo camera setups
• More time required to shoot (higher shooting costs)
• Requires shooting on digital cameras (no option of film)
• Special stereo monitor and glasses required on set to check settings.
• Requires a certain amount of adjustment, alignment and cleaning.
• Requires special rig technician to handle these adjustments.
• Restrictions on lenses that can obtain good looking stereo
• Requires locking in stereo depth at time of shooting. This cannot be adjusted in post.
• Shooting with ‘toe in’ (non-parallel) camera systems requires a convergence puller similar to a focus puller. Having the correct convergence point is critical when editing shot to shot. This convergence point can only be adjusted so far in post. (With parallel camera systems this convergence is all set in post.)
• Lens flares and shiny reflections can appear different to both cameras and will require post work to correct to prevent viewing problems of final film.
• Because of the polarizing through the beam splitter, water and even asphalt with a sheen will appear different to both cameras and require post correction
• Post work required to align and fix footage. Since one camera is shooting through a beam splitter and another is shooting the beam splitter reflection there is typically an imbalance in terms of color, contrast and alignment.
• Visual Effects require more work and more rendering (added time and cost)
Stereo Conversion - PROS
• Ability to shoot with standard process (Composing with 3D in mind still recommended)
• Choice to shoot on film with any film camera or on digital with any digital camera
• Unlimited choice of camera lenses (Extreme telephoto still not recommended for best 3D results)
• Ability to add depth and volume even on telephoto shots
• Ability to set the 3D depth in post on a shot by shot basis
• Ability to set the convergence as desired for the edit
• Flexibility to adjust the depth and volume of each actor or object in a scene as a creative option not available when shooting stereo.
• Visual Effects are handled in the standard way
Stereo Conversion - CONS
• Extra time required after edit to properly convert or sequences need to be locked during edit to allow conversion to take place while in post.
• Added cost of process
• Reflections, smoke, sparks and rain more difficult to convert but not impossible.
The 2D to 3D process
Each company handles conversion in a slightly different way, sometimes with proprietary tools and techniques but what follows are some of the fundamentals.
The first step in the conversion is to have a stereographer review the shot and determine how it will work in 3D space. How much depth budget there will be (how much total depth in the scene) and where the convergence point will be. The convergence point ends up on the movie screen and imagery in front of the screen is said to be in negative parallax and imagery behind the screen is positive parallax. If you have two images (left and right eye) of a circle and it’s in the exact same spot for both eyes then it will appear on the screen surface. If you offset the images of the circle toward or away (left/right) from each other so they overlap but don’t match then you’ll have the circle float in front of the screen or behind the screen depending on the direction of offset. The amount of offset will determine how far forward or back it appears.
Convergence decisions are based on the action and edit as well as what is in the scene. Frequently the key actors eyes are used for the convergence point so when cutting from one scene to another the audience doesn’t have to visually look back and forth constantly.
With this information the image is broken down. How much detail depends on where it is in space. Stereo eyesight primarily works within 20-25 feet. As objects get further away there tends to be less sense of 3D stereo. Breaking down the image may require roto or some form of image area selection. If you’ve done roto or frame by frame extraction you know how difficult and time consuming it is. But in this case everything in the images is broken down, not just a foreground actor. In vfx if you roto a group of people you may be primarily concerned about the group areas that overlaps the area to be replaced. With 2D to 3D you have to isolate each person and each object on a different depth plane.
So imagine the degree of difficulty extracting every 3D surface from a 2D image and making all the frames and transitions match perfectly and smoothly including wild camera moves and explosions with thousands of sparks. Depending on the exact 2D to 3D process used you may have build actual 3D models or at least replicate that process to some extent. If there is a table in the image the table-top will have to be split into top, front and side.
The amount of work and complexity means that if there is a change to the image (edit, reposition, speed up, slow down, etc) it may require redoing the entire shot.
With traditional roto you can get away with things like losing a few fly away hairs on an actress. If they’re clipped off and a new background is composited no one will know. With conversion however any hairs that have not been extracted may end up at the wrong 3D depth. Imagine a character in the foreground with long flowing hair but having the hair stuck to the mountains in the background. Each company leverages what techniques and software they can to help with this process and compositors can also help the process by leveraging various extraction methods.
3D settings
Once the image is actually broken up it is necessary to create the depth. A conversion stereographer may use propriety software or heavily modified Nuke or other application to help with this step. This can be in a true 3D world or it can be simulated 3D with gap offsets. The stereographer usually works with a stereo monitor and adjusts and animates the very shapes, objects and planes to move in 3D space. This needs to match the surrounding shots and the 3D relationships need to be correct. In the case of Transformers the volume and depth of the objects has to match to the original stereo photography. The show stereographer works with the director as well to provide both creative and technical feedback regarding the depth range and convergence planes. Cory Turner was the show Stereographer on Transformers.
Viewing stereo footage and making adjustments requires developing an eye for stereo. Not everyone can do it and different people have different aptitude for judging stereo. With most vfx work it’s usually obvious when there are matte edges or a color balance problem between elements. Seeing these types of problems is consistent across a number of individuals. Yet viewing stereo footage is much more subjective. Even stereo experts commonly disagree on the latest 3D film. Some may say it’s the best stereo they’ve ever seen and others will say it’s the worst they’ve seen. Technical issues and creative choices by the stereographer and director will affect the final results.
But even at this stage if all you’ve done is cutout the objects you’ll end up with flat images set at different depths. What you need to do is add volume and real 3D shape to the image that matches. This is one of the areas that 2D to 3D conversion provides an advantage over shooting stereo. Telephoto lens images can be provided more volume than would be possible shooting stereo. Different objects can be given more or less 3D emphasis. An actor needs to have their head rounded, their eyes sunken in and their nose coming out. This relationship needs to match even as the actor turns his head and the camera moves. There’s a real art to the process and much of this can not be automated. The skill of the artists combined with the tools at their disposal has a lot to do with the final results.
The end result of this process is 2 images – a left and right eye image. If this is a full 3D process there are essentially two 3D CG cameras setup with an InterAxial offset (lens separation just as if shot with a stereo camera system). Some companies use the original image as one eye and generate only the other eye. Legend tends to convert the original footage as if it’s the center image and so they actually generate both eyes. This tends to make sure both eyes match and the image itself isn’t always offset from the original framing.
Because the original image was 2D when sub-images are offset they create gaps where there is no image. The simple way of thinking of this is to start with 2 identical images and cutting out an actor in each. Offset one cutout of the actor 10 pixels to the right and offset the other 10 pixels to the left. Now there’s no image on the edge where the image was moved from. This gap needs to be filled in so you don’t see a black edge in the finished shots. If the gap is really small you can cheat it but these artifacts can be quite noticeable on the big screen. To do it right in most cases requires treating the edge of each object as a full on rig removal. Creating or shooting a clean plate can greatly help with this process.
Keep in mind all of these steps, adjustments and cleanup have to be done on every single shot in the film if the film was originally shot 2D.
Reflections, lens flares, smoke and other problems
And as if that wasn’t tough enough imagine the main actor wearing glasses. The glasses have to be set at the correct depth to relate to the head. Any thumbprints or dirt on the lenses have to be on that same plane, however the actor’s eyes have to be pushed in. More than likely there is a reflection on the glasses of the room, which may be 20 feet away. And the actor may have a highlight on his eye that reflects something 5 feet away. Each of these different images have to be extracted and removed so you end up with a clean eye with no reflections, the glass reflections, the eye highlight and the thumbprint. This is from live action so requires the compositor to go through a process along the lines of rig removal but splitting in to each depth layer. Each of these layers then has to be placed and positioned in 3D and then composited back together. If this isn’t done then it will look like the actors eyes and the reflections are simply painted on the surface of the glasses. Needless to say this looks is very wrong in 3D.
Same issues if the shot has glass window, mirror, chrome or other reflective surfaces. The issue of seeing partially transparent layers at different depths also comes into play if you have smoke or fog in the scene. If you have partially transparent smoke in front of a building you don’t want the building to be pulled forward to 10 feet from camera nor do you want the smoke to be pushed all the back to the front of the building. Same exact problem happens with lens flares. Each of these has to be removed into a separate element and then added back in at the right depth, usually near the screen plane for lens flares.
Transformers of course was full of fast action with glass windows, smoke, explosions, sparks, and lens flares.
Conversion of VFX shots
VFX shots are usually tough enough without having to deal with two slightly different views for the left and right eye. Working in stereo takes more time and adds an additional level of complexity. When the footage is shot as 2D then the vfx can be finished as 2D shots and approved by the director, which tends to be quicker and easier. But it’s usually best to take advantage of the vfx process when doing the conversion to produce better conversion quicker.
In vfx shots it’s not unusual to add smoke and lens flares among other things. Obviously once these are baked into the shot the conversion artist will have to re-separate them. It makes more sense for speed and quality to simply breakdown the composites in such a way to keep key layers separate and use those layers to set the depth in the conversion stage. Some of the other vfx elements are also useful such as clean plates and potentially some of the roto or mattes.
For shots with CG elements it’s better if the CG elements are rendered separately so there’s no need to extract them and fill in the gaps if only a finished composite was provided. Taking this further the vfx company can render depth maps so the conversion company can leverage it and provide a consistency to the creature (or in this case, transformers)
The majority of 2D vfx shots in Transformers was done this way. The sheer complexity of the Transformers, the number of moving and changing surfaces, really took advantage of this process and allowed them to be consistent with the stereo vfx shots. A few shots were done where the vfx company rendered 2 views of the transformer and then Legend created the 3D world to go around the rendered stereo image. In a few other cases Legend converted the background to stereo and the vfx company worked from that.
Tips for VFX companies
If the film project you’ll be starting is to be converted make sure to work with production and the conversion company. The conversion company should provide a list of requested elements and the ideal formats. This isn’t simply a matter of turning over a Nuke script. Chances are your company has proprietary plugins and scripts. It’s also likely that the composites may be very complex yet the conversion company only needs the layers that make sense in depth. You may not typically render depth maps and potentially other elements independently. The cost and time to do this re-purposing of composites and elements will have to be accounted for either in the vfx bid or as a separate bid. If it’s vfx heavy show it’s likely there will be a small team of people prepping images for the conversion process.
Ideally the production company hires a show stereographer who oversees the process from shot design through to completion.
As with the original photography it’s important for everyone to consider the work as if it were in true 3D space. No more cheating where animated characters are placed or using one element that is supposed to represent multiple depths. These problems will show themselves once converted to real 3D space.
Compositors need to consider building the composite based on 3D space. When holdouts are built in or there are elements with an odd layer order then it becomes difficult to re-build the composites to work in a dimension conversion.
Work out a naming process for these special conversion elements to make it easy for the conversion company to rebuild the composites. Example: RL045.layer_1.over.00128.exr
It’s also important to provide any special lookup tables or color specific nodes
Summary
Conversion can create successful stereo images that match images shot in stereo as long as experienced artists are allowed sufficient time and control to do this complex and vfx heavy process correctly. Just as with vfx companies, the selected conversion companies will determine how successful the final results are. Neither visual effects nor conversion are truly commodities that can be randomly assigned to other companies with the same result.
Legend
Legend3D also did at least some of the conversion work on the following features while Transformers 3 was being done: Green Hornet, Pirates of the Caribbean: On Stranger Tides, Green Lantern, Priest, Smurfs3D. They were able to leverage many of the improvements created for Transformers 3.
[Update: Forgot to post a link to a VFX Guide Podcast that touches on the conversion. I believe it's touched on at about 45+ minutes in.]
[Update: I was reminded when responding to a comment below there was a video of Michael Bay and James Cameron discussing shooting stereo. This was done after the film had been shot and within a few weeks of release. Video of stereo conversation of Bay and Cameron]
[Update: I got a few tweets from @philwittmer that I thought I'd address here since there's still seems to be some confusion.
Phil:
Sorry but that all seems biased towards 2d conversion. With no mention of roto time. granted its an interesting read I never thought of it like that but.... a couple of major points here need to be made
1. U get limitless depth if shooting 3d as opposed to designated planes
2. Moving foc point in post just means bad directing
lastly and most importantly. The conversion will only be as good as the budget and time. Which unfortunately is usually bad
Me:
Yes, any conversion takes time. Roto takes time, extractions take time, setting depth takes time.
That's why I stated you can't do it in 6 weeks and that was listed as one of the CONs of doing conversion. It is a time consuming process.
If the film has a large number of vfx shots then this conversion can be done (especially on the non-vfx shots) while the vfx are being worked on. If the vfx shots are locked and can be worked on as you proceed it still takes time but that post time is already used up by the vfx work. That means it doesn't necessarily mean a large delay after the vfx, depending on the show.
Keep in mind all stereo shoot footage needs to be processed as well. Every shot will have to go through a process to be adjusted for color and alignment and other corrections so the left and right images match correctly. This is not an overnight process and so you need to budget time to do shot by shot corrections of the entire film, like DI but can be much more intense. If any shots are truly broken they may need 2D to 3D conversion or serious vfx work to repair them so expect even more time. This will be less than doing a conversion of the whole film but it's time that should not be discounted.
Unlimited 3D - When shooting 3D you have to determine the amount of depth you want. And that is not unlimited. You can't have something 1 foot from the viewer and things pushed to infinity without compressing that space. If you have too much separation in the background you'll get divergence where people's eyes will go walleye (each looking outward). Not pleasant. Too much negative parallax will cause eye strain as well.
Now you can get nice in camera stereo of complex things in the mid distance such as tall grass, leaves on a tree, smoke, etc. That's why that's a PRO for shooting stereo: Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.
Designated planes - Just to be clear this isn't like a multiplane camera when you have flat cards at different distances and there are 10 designated planes. It's not like the process that seemed to result in some viewmaster displays (i.e. all images flat to camera at different depths). Any flat surface can be at any angle. It has to be to actually work in a 3D movie. That road has to be angled away from camera, that building on the side going away from camera, etc. And more importantly you're creating a dimensional space, at least with the better processes. You have to create the sense of roundness and volume as if you built it as a real 3D model. A shot of a head is a good example (a complex sphere with nose, eye sockets, ears, jaw, etc). You can not get away with having a person on a flat card and expect that to hold up for the audience. Whether the company uses real 3D models, depth maps, complex meshes, etc. you can be sure it's not made up of just designated planes. And yes, that make it very hard, especially for the items that I listed as easier to shoot. Numerous, complex shapes that are changing in 3D space are very difficult.
During shooting of stereo you have the issue of setting the IA (interaxial) to set the overall depth. And you have the issue of setting the convergence. Either of these may be manually changed during the shot depending on the end result. Is it bad for a director to find out later a different edit works better? What if in planning the shots the director originally thought the shots would cut together in a specific way and shot with that in mind. But once the cuts are changed the conversion may have to be adjusted in post so there won't be as much eye strain. Wide shot to closeup, the timing of the convergence wasn't right, etc. What if the initial IA was high but the fast action caused more strain? The director and stereographer have to really do some planning and testing before shooting. And even then there may be an edited version that doesn't work and needs to be adjusted. And some things like IA can't be adjusted cleanly in post currently. So some 3D decisions made on set are what will be.
>The conversion will only be as good as the budget and time.
This goes for every element of a film. You can't do Avatar in 10 days for $10. You can't record sound for a feature film with your iPhone. If the studio doesn't allow the correct amount of time and money to do the conversion correctly then they will have a movie they spent a lot of money on looking terrible in 3D. And this will cause turning more people away from 3D. The studios are starting to understand the correlation between good 3D and a happy audience willing to pay for 3D movies. And many of these 3D films (shot or converted) are large tent pole movies so there is already a large budget so the studios don't want to gamble and go super cheap.
In the end there are pros and cons to both methods. One of the main points of this article is both can produce good results. Which one to use and what balance of the two depends on the specific project. If production and the director can commit to shooting 3D with a real stereographer and take the necessary care and attention then shooting stereo may be their best choice. Just don't rule out 2D to 3D conversion as one of the options to consider simply because it's been done poorly on some films.
Update: Opensource 2D/3D conversion Gimpel3D for those interested.
2D to 3D conversion is the process of converted a film from 2D (normal film) to a 3D (or stereo) film for viewing with a stereographic viewing system. (polarized glasses, shuttered glasses, special screen, etc)
Conversion had gained a bad name after Clash of the Titans and a few other films that had been rushed through at the last minute. Just as you can’t do a 2000 shot vfx show in 6 weeks with great quality, you can’t do a quality conversion of an entire movie in 6 weeks.
Most people think of a 2D to 3D conversion as a factory type of operation but it’s far from it if it’s to be done right. Much of my past year was spent working for Legend3D dealing with some interesting and very technically challenging problems. Legend had completed work on Alice (Tea party sequence and Drink Me sequence) and done the conversion on the first 3 Shrek films to be released for 3D TV.
Legend was starting on Transformers: Dark of the Moon and would need help with the extensive visual effects shots. Legend 3D ended up converting over 77 minutes of footage for Transformers. Most of the vfx shots were converted in a 4-5 month time span. The converted footage was a mix of visual effects shots (from ILM and DD) and non-vfx shots. In most cases these would be shots intercut back and forth with original stereo footage shot on location. This raised the complexity much more than if the entire film had been converted. Just as doing vfx in a live action film is a different task than creating an animated film (or virtual sequence).
This was also more difficult than doing a library title or a film that was completed. The same issues vfx companies face of ever changing edits and creative directions was in full force. In addition the vfx shots were changing and so we had to follow very closely and track all changes such that we could turn around the final converted shots within a day or two of deliver of the last vfx shots.
While at Legend I helped them to develop the pipelines required to do this type of work efficiently and at as high of quality level as possible. Legend had some proprietary software and techniques for different stages in the process. I wrote a number of specialty Nuke plugins and scripts to help leverage these proprietary software packages with existing software. I worked with a number of artists and developers at Legend to review and analyze the shots as they were delivered and worked closely with ILM and DD to get the proper elements that would be required. Both technical and creative issues had to be examined to get the best possible quality. Overall this represented a number of challenges that were unique to the world of stereo conversion and we were able to raise the quality of conversion to a new level.
In what follows I’ll cover the basics of the conversion process. I won’t be able to go into specific detail to due the proprietary nature but hopefully this will provide an understanding of the process.
Conversion versus shooting stereo
Many people are under the mistaken belief that shooting stereo is superior to conversion in every way. The key to getting great 3D is to design, shoot and edit for 3D, even if it will be converted. If you shoot stereo but don’t consider the 3D world then you will have problems.
In its current form 3D is a stereo process, which means creating imagery for each eye. It’s actually a big cheat when projecting 2 different images to simulate the look of the 3D because your eyes are always focused on the screen but objects appear at other depths. This is a fairly unnatural process.
The following are some of the basic pros and cons. As with everything there are tradeoffs to be made. Shooting stereo does capture the images in 3D but many of the parameters are locked in at the time of shooting. Shooting stereo also requires a fair bit of post work to actually get the images to match. Films that are shot in stereo will still require a certain number of conversion shots just due to problematic shots or limitations while shooting. This was the case even with Avatar.
Shooting Stereo – PROS
• Capturing stereo images at time of shooting
• Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.
• Preview stereo images on set and on location
• Footage can be edited and reviewed in stereo context with the correct equipment
Shooting Stereo - CONS
• Requires specialized camera rigs
• Camera rigs contain 2 cameras so are larger than non-stereo camera setups
• More time required to shoot (higher shooting costs)
• Requires shooting on digital cameras (no option of film)
• Special stereo monitor and glasses required on set to check settings.
• Requires a certain amount of adjustment, alignment and cleaning.
• Requires special rig technician to handle these adjustments.
• Restrictions on lenses that can obtain good looking stereo
• Requires locking in stereo depth at time of shooting. This cannot be adjusted in post.
• Shooting with ‘toe in’ (non-parallel) camera systems requires a convergence puller similar to a focus puller. Having the correct convergence point is critical when editing shot to shot. This convergence point can only be adjusted so far in post. (With parallel camera systems this convergence is all set in post.)
• Lens flares and shiny reflections can appear different to both cameras and will require post work to correct to prevent viewing problems of final film.
• Because of the polarizing through the beam splitter, water and even asphalt with a sheen will appear different to both cameras and require post correction
• Post work required to align and fix footage. Since one camera is shooting through a beam splitter and another is shooting the beam splitter reflection there is typically an imbalance in terms of color, contrast and alignment.
• Visual Effects require more work and more rendering (added time and cost)
Stereo Conversion - PROS
• Ability to shoot with standard process (Composing with 3D in mind still recommended)
• Choice to shoot on film with any film camera or on digital with any digital camera
• Unlimited choice of camera lenses (Extreme telephoto still not recommended for best 3D results)
• Ability to add depth and volume even on telephoto shots
• Ability to set the 3D depth in post on a shot by shot basis
• Ability to set the convergence as desired for the edit
• Flexibility to adjust the depth and volume of each actor or object in a scene as a creative option not available when shooting stereo.
• Visual Effects are handled in the standard way
Stereo Conversion - CONS
• Extra time required after edit to properly convert or sequences need to be locked during edit to allow conversion to take place while in post.
• Added cost of process
• Reflections, smoke, sparks and rain more difficult to convert but not impossible.
The 2D to 3D process
Each company handles conversion in a slightly different way, sometimes with proprietary tools and techniques but what follows are some of the fundamentals.
The first step in the conversion is to have a stereographer review the shot and determine how it will work in 3D space. How much depth budget there will be (how much total depth in the scene) and where the convergence point will be. The convergence point ends up on the movie screen and imagery in front of the screen is said to be in negative parallax and imagery behind the screen is positive parallax. If you have two images (left and right eye) of a circle and it’s in the exact same spot for both eyes then it will appear on the screen surface. If you offset the images of the circle toward or away (left/right) from each other so they overlap but don’t match then you’ll have the circle float in front of the screen or behind the screen depending on the direction of offset. The amount of offset will determine how far forward or back it appears.
Convergence decisions are based on the action and edit as well as what is in the scene. Frequently the key actors eyes are used for the convergence point so when cutting from one scene to another the audience doesn’t have to visually look back and forth constantly.
With this information the image is broken down. How much detail depends on where it is in space. Stereo eyesight primarily works within 20-25 feet. As objects get further away there tends to be less sense of 3D stereo. Breaking down the image may require roto or some form of image area selection. If you’ve done roto or frame by frame extraction you know how difficult and time consuming it is. But in this case everything in the images is broken down, not just a foreground actor. In vfx if you roto a group of people you may be primarily concerned about the group areas that overlaps the area to be replaced. With 2D to 3D you have to isolate each person and each object on a different depth plane.
So imagine the degree of difficulty extracting every 3D surface from a 2D image and making all the frames and transitions match perfectly and smoothly including wild camera moves and explosions with thousands of sparks. Depending on the exact 2D to 3D process used you may have build actual 3D models or at least replicate that process to some extent. If there is a table in the image the table-top will have to be split into top, front and side.
The amount of work and complexity means that if there is a change to the image (edit, reposition, speed up, slow down, etc) it may require redoing the entire shot.
With traditional roto you can get away with things like losing a few fly away hairs on an actress. If they’re clipped off and a new background is composited no one will know. With conversion however any hairs that have not been extracted may end up at the wrong 3D depth. Imagine a character in the foreground with long flowing hair but having the hair stuck to the mountains in the background. Each company leverages what techniques and software they can to help with this process and compositors can also help the process by leveraging various extraction methods.
3D settings
Once the image is actually broken up it is necessary to create the depth. A conversion stereographer may use propriety software or heavily modified Nuke or other application to help with this step. This can be in a true 3D world or it can be simulated 3D with gap offsets. The stereographer usually works with a stereo monitor and adjusts and animates the very shapes, objects and planes to move in 3D space. This needs to match the surrounding shots and the 3D relationships need to be correct. In the case of Transformers the volume and depth of the objects has to match to the original stereo photography. The show stereographer works with the director as well to provide both creative and technical feedback regarding the depth range and convergence planes. Cory Turner was the show Stereographer on Transformers.
Viewing stereo footage and making adjustments requires developing an eye for stereo. Not everyone can do it and different people have different aptitude for judging stereo. With most vfx work it’s usually obvious when there are matte edges or a color balance problem between elements. Seeing these types of problems is consistent across a number of individuals. Yet viewing stereo footage is much more subjective. Even stereo experts commonly disagree on the latest 3D film. Some may say it’s the best stereo they’ve ever seen and others will say it’s the worst they’ve seen. Technical issues and creative choices by the stereographer and director will affect the final results.
But even at this stage if all you’ve done is cutout the objects you’ll end up with flat images set at different depths. What you need to do is add volume and real 3D shape to the image that matches. This is one of the areas that 2D to 3D conversion provides an advantage over shooting stereo. Telephoto lens images can be provided more volume than would be possible shooting stereo. Different objects can be given more or less 3D emphasis. An actor needs to have their head rounded, their eyes sunken in and their nose coming out. This relationship needs to match even as the actor turns his head and the camera moves. There’s a real art to the process and much of this can not be automated. The skill of the artists combined with the tools at their disposal has a lot to do with the final results.
The end result of this process is 2 images – a left and right eye image. If this is a full 3D process there are essentially two 3D CG cameras setup with an InterAxial offset (lens separation just as if shot with a stereo camera system). Some companies use the original image as one eye and generate only the other eye. Legend tends to convert the original footage as if it’s the center image and so they actually generate both eyes. This tends to make sure both eyes match and the image itself isn’t always offset from the original framing.
Because the original image was 2D when sub-images are offset they create gaps where there is no image. The simple way of thinking of this is to start with 2 identical images and cutting out an actor in each. Offset one cutout of the actor 10 pixels to the right and offset the other 10 pixels to the left. Now there’s no image on the edge where the image was moved from. This gap needs to be filled in so you don’t see a black edge in the finished shots. If the gap is really small you can cheat it but these artifacts can be quite noticeable on the big screen. To do it right in most cases requires treating the edge of each object as a full on rig removal. Creating or shooting a clean plate can greatly help with this process.
Keep in mind all of these steps, adjustments and cleanup have to be done on every single shot in the film if the film was originally shot 2D.
Reflections, lens flares, smoke and other problems
And as if that wasn’t tough enough imagine the main actor wearing glasses. The glasses have to be set at the correct depth to relate to the head. Any thumbprints or dirt on the lenses have to be on that same plane, however the actor’s eyes have to be pushed in. More than likely there is a reflection on the glasses of the room, which may be 20 feet away. And the actor may have a highlight on his eye that reflects something 5 feet away. Each of these different images have to be extracted and removed so you end up with a clean eye with no reflections, the glass reflections, the eye highlight and the thumbprint. This is from live action so requires the compositor to go through a process along the lines of rig removal but splitting in to each depth layer. Each of these layers then has to be placed and positioned in 3D and then composited back together. If this isn’t done then it will look like the actors eyes and the reflections are simply painted on the surface of the glasses. Needless to say this looks is very wrong in 3D.
Same issues if the shot has glass window, mirror, chrome or other reflective surfaces. The issue of seeing partially transparent layers at different depths also comes into play if you have smoke or fog in the scene. If you have partially transparent smoke in front of a building you don’t want the building to be pulled forward to 10 feet from camera nor do you want the smoke to be pushed all the back to the front of the building. Same exact problem happens with lens flares. Each of these has to be removed into a separate element and then added back in at the right depth, usually near the screen plane for lens flares.
Transformers of course was full of fast action with glass windows, smoke, explosions, sparks, and lens flares.
Conversion of VFX shots
VFX shots are usually tough enough without having to deal with two slightly different views for the left and right eye. Working in stereo takes more time and adds an additional level of complexity. When the footage is shot as 2D then the vfx can be finished as 2D shots and approved by the director, which tends to be quicker and easier. But it’s usually best to take advantage of the vfx process when doing the conversion to produce better conversion quicker.
In vfx shots it’s not unusual to add smoke and lens flares among other things. Obviously once these are baked into the shot the conversion artist will have to re-separate them. It makes more sense for speed and quality to simply breakdown the composites in such a way to keep key layers separate and use those layers to set the depth in the conversion stage. Some of the other vfx elements are also useful such as clean plates and potentially some of the roto or mattes.
For shots with CG elements it’s better if the CG elements are rendered separately so there’s no need to extract them and fill in the gaps if only a finished composite was provided. Taking this further the vfx company can render depth maps so the conversion company can leverage it and provide a consistency to the creature (or in this case, transformers)
The majority of 2D vfx shots in Transformers was done this way. The sheer complexity of the Transformers, the number of moving and changing surfaces, really took advantage of this process and allowed them to be consistent with the stereo vfx shots. A few shots were done where the vfx company rendered 2 views of the transformer and then Legend created the 3D world to go around the rendered stereo image. In a few other cases Legend converted the background to stereo and the vfx company worked from that.
Tips for VFX companies
If the film project you’ll be starting is to be converted make sure to work with production and the conversion company. The conversion company should provide a list of requested elements and the ideal formats. This isn’t simply a matter of turning over a Nuke script. Chances are your company has proprietary plugins and scripts. It’s also likely that the composites may be very complex yet the conversion company only needs the layers that make sense in depth. You may not typically render depth maps and potentially other elements independently. The cost and time to do this re-purposing of composites and elements will have to be accounted for either in the vfx bid or as a separate bid. If it’s vfx heavy show it’s likely there will be a small team of people prepping images for the conversion process.
Ideally the production company hires a show stereographer who oversees the process from shot design through to completion.
As with the original photography it’s important for everyone to consider the work as if it were in true 3D space. No more cheating where animated characters are placed or using one element that is supposed to represent multiple depths. These problems will show themselves once converted to real 3D space.
Compositors need to consider building the composite based on 3D space. When holdouts are built in or there are elements with an odd layer order then it becomes difficult to re-build the composites to work in a dimension conversion.
Work out a naming process for these special conversion elements to make it easy for the conversion company to rebuild the composites. Example: RL045.layer_1.over.00128.exr
It’s also important to provide any special lookup tables or color specific nodes
Summary
Conversion can create successful stereo images that match images shot in stereo as long as experienced artists are allowed sufficient time and control to do this complex and vfx heavy process correctly. Just as with vfx companies, the selected conversion companies will determine how successful the final results are. Neither visual effects nor conversion are truly commodities that can be randomly assigned to other companies with the same result.
Legend
Legend3D also did at least some of the conversion work on the following features while Transformers 3 was being done: Green Hornet, Pirates of the Caribbean: On Stranger Tides, Green Lantern, Priest, Smurfs3D. They were able to leverage many of the improvements created for Transformers 3.
[Update: Forgot to post a link to a VFX Guide Podcast that touches on the conversion. I believe it's touched on at about 45+ minutes in.]
[Update: I was reminded when responding to a comment below there was a video of Michael Bay and James Cameron discussing shooting stereo. This was done after the film had been shot and within a few weeks of release. Video of stereo conversation of Bay and Cameron]
[Update: I got a few tweets from @philwittmer that I thought I'd address here since there's still seems to be some confusion.
Phil:
Sorry but that all seems biased towards 2d conversion. With no mention of roto time. granted its an interesting read I never thought of it like that but.... a couple of major points here need to be made
1. U get limitless depth if shooting 3d as opposed to designated planes
2. Moving foc point in post just means bad directing
lastly and most importantly. The conversion will only be as good as the budget and time. Which unfortunately is usually bad
Me:
Yes, any conversion takes time. Roto takes time, extractions take time, setting depth takes time.
That's why I stated you can't do it in 6 weeks and that was listed as one of the CONs of doing conversion. It is a time consuming process.
If the film has a large number of vfx shots then this conversion can be done (especially on the non-vfx shots) while the vfx are being worked on. If the vfx shots are locked and can be worked on as you proceed it still takes time but that post time is already used up by the vfx work. That means it doesn't necessarily mean a large delay after the vfx, depending on the show.
Keep in mind all stereo shoot footage needs to be processed as well. Every shot will have to go through a process to be adjusted for color and alignment and other corrections so the left and right images match correctly. This is not an overnight process and so you need to budget time to do shot by shot corrections of the entire film, like DI but can be much more intense. If any shots are truly broken they may need 2D to 3D conversion or serious vfx work to repair them so expect even more time. This will be less than doing a conversion of the whole film but it's time that should not be discounted.
Unlimited 3D - When shooting 3D you have to determine the amount of depth you want. And that is not unlimited. You can't have something 1 foot from the viewer and things pushed to infinity without compressing that space. If you have too much separation in the background you'll get divergence where people's eyes will go walleye (each looking outward). Not pleasant. Too much negative parallax will cause eye strain as well.
Now you can get nice in camera stereo of complex things in the mid distance such as tall grass, leaves on a tree, smoke, etc. That's why that's a PRO for shooting stereo: Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.
Designated planes - Just to be clear this isn't like a multiplane camera when you have flat cards at different distances and there are 10 designated planes. It's not like the process that seemed to result in some viewmaster displays (i.e. all images flat to camera at different depths). Any flat surface can be at any angle. It has to be to actually work in a 3D movie. That road has to be angled away from camera, that building on the side going away from camera, etc. And more importantly you're creating a dimensional space, at least with the better processes. You have to create the sense of roundness and volume as if you built it as a real 3D model. A shot of a head is a good example (a complex sphere with nose, eye sockets, ears, jaw, etc). You can not get away with having a person on a flat card and expect that to hold up for the audience. Whether the company uses real 3D models, depth maps, complex meshes, etc. you can be sure it's not made up of just designated planes. And yes, that make it very hard, especially for the items that I listed as easier to shoot. Numerous, complex shapes that are changing in 3D space are very difficult.
During shooting of stereo you have the issue of setting the IA (interaxial) to set the overall depth. And you have the issue of setting the convergence. Either of these may be manually changed during the shot depending on the end result. Is it bad for a director to find out later a different edit works better? What if in planning the shots the director originally thought the shots would cut together in a specific way and shot with that in mind. But once the cuts are changed the conversion may have to be adjusted in post so there won't be as much eye strain. Wide shot to closeup, the timing of the convergence wasn't right, etc. What if the initial IA was high but the fast action caused more strain? The director and stereographer have to really do some planning and testing before shooting. And even then there may be an edited version that doesn't work and needs to be adjusted. And some things like IA can't be adjusted cleanly in post currently. So some 3D decisions made on set are what will be.
>The conversion will only be as good as the budget and time.
This goes for every element of a film. You can't do Avatar in 10 days for $10. You can't record sound for a feature film with your iPhone. If the studio doesn't allow the correct amount of time and money to do the conversion correctly then they will have a movie they spent a lot of money on looking terrible in 3D. And this will cause turning more people away from 3D. The studios are starting to understand the correlation between good 3D and a happy audience willing to pay for 3D movies. And many of these 3D films (shot or converted) are large tent pole movies so there is already a large budget so the studios don't want to gamble and go super cheap.
In the end there are pros and cons to both methods. One of the main points of this article is both can produce good results. Which one to use and what balance of the two depends on the specific project. If production and the director can commit to shooting 3D with a real stereographer and take the necessary care and attention then shooting stereo may be their best choice. Just don't rule out 2D to 3D conversion as one of the options to consider simply because it's been done poorly on some films.
Update: Opensource 2D/3D conversion Gimpel3D for those interested.
Subscribe to:
Posts (Atom)