Wednesday, August 31, 2011

Letter to Animation Guild

The Animation Guild received an email for someone in the VFX industry who documents some of the issues faced by typical vfx workers.   This is somewhat of an extreme but there's certainly a basis of truth.

If any of this sounds familiar then I urge you to make your voice heard. Don't expect someone else to come and fix the problem. Don't expect the company you work for to fix the problems and don't expect the studios to step forward and fix the problems.

 Email VES Leadership group





Saturday, August 27, 2011

VFX Wages Discussion

This post is in response to a comment on a previous post.  See this post and follow the comments there for the full original comment. (Toward the end from Anonymous) Unfortunately once I wrote this response it was bigger than a comment could be so I've done it as a post.

Here's part of that comment (focused for this post) but request you read it all to see it in context:

"Here is my thought on this: We got greedy.


During boom times artist rates went up and up. I now make more money than most doctors. I know plenty of others who do just as well. We are the most expensive part of production on most films. While single individuals may get larger paychecks- the director, producer, lead actor, a massive amount of money goes to VFX.


We are too expensive. I make, on average, 5 to 10 times more than other Americans. Why?! It just happened that way over time, I didn't ever expect to be doing this well as an employee of another company. I am well beyond my own expectations."

We got greedy? I don't recall a mass rally outside vfx company offices calling for huge pay increases. I don't recall any company just deciding one day they will pay more simply because vfx workers would like more.

I'm still not clear why people are apologetic for being paid above a national average as if it were a sin. Especially if you're actually a skilled and knowledgable worker creating something or performing a real service that will ultimately be very profitable. According to some of these people we're all so well paid none of us should get health care or any other benefits.

I'd like to think that decades of experience in a very difficult creative and technical industry would amount to something. That the time and effort put into keeping up with quickly changing software, images and techniques would be of some value. I'd like to be paid more than I was 20+ years ago. Some movies makes hundred of millions of dollars (and some even more than a billion dollars). Movies that we helped create. I'd like to think that those of us who do this are paid enough money to have a reasonable home in the cities we work in, that we can afford to raise children, that we don't have to worry about health care bills, that we don't have to scrape the bottom of the barrel to try to retire one day, and that we we're paid enough so we don't have to worry if we don't get a call for a project next week. Is that being greedy? I don't think so.

There will always be people who make more than you and people who make less than you. Bill Gates isn't always the top of the list. You may not have  made any money last year but there's likely someone who hasn't made money for the last 2+ years.

The average US CEO of a public company makes over $10 million a year. [To put that in perspective many CEOs earn as much in 1 day as a typical US worker does in an entire year. 1 Day] Are they that much smarter? Harder working? Working longer hours? From my experience you could actually replace most CEO's with a rock and it wouldn't make a bit of difference to the bottom line since it tends to be the workers and managers that handle the actual running and in many cases the non-CEOs are the ones that are forward thinking. (Steve Jobs is a rare exception)  I suspect many of these CEO's have assistants that probably make more than a CG supervisor. At the other end of the spectrum are people working multiple jobs cleaning toilets and digging ditches just to try to make a living for themselves and their family. There are some teachers and regional pilots that make less than the poverty level of income. Is any of that fair? No, but all we have at most is a tiny bit of control of our own wages.

I know someone who works in non-vfx that probably puts in half the hours I do and is paid 2-4 times what I make. I know a non-vfx software engineer who was very well paid at a permanent position. More than you are. He received a $300,000 bonus when they shipped on time. When was the last time you got paid a bonus? Any type of bonus? [Supreme Court Clerks now receive over $280,000 signing bonus when they go to work at a law firm.] Most of the time people put in an incredible number of hours at the end of a project and may not even receive a thank you from the company. If the company is really feeling up to it they might have a wrap party as a thank you.

Drive around Los Angeles and look at the pricing of homes in areas you wouldn't mind living in. Base price of a reasonable house (small 2 bedroom) in LA is over $800,000. Many start at $2 million and go up. There are a lot of people in LA making a lot of money and they're not vfx people. Do you think all of these people are going without health care and pensions? Do they all feel guilty?

We are the most expensive part of production on most films.
Well we’re a large expense IF the show is a VFX show.  I’d hardly call vfx the most expensive part of most films. Most films don't have extensive vfx. Most films burn through $100,000-$300,000 per day while shooting. Most films spend tens of millions of dollars on advertising and promotion. Most films employee at least some actors making $2 million to $20 million. On most of the vfx projects I’ve worked on, vfx typically use less than ½ the budget, in many cases 1/3 to ¼.  Published movie budgets are seldom accurate.

While single individuals may get larger paychecks- the director, producer, lead actor, a massive amount of money goes to VFX.”
I cover a few other people below. You act as if it’s our fault and something we should be ashamed of. A studio chooses which projects they feel will make money in the box office. Most of the top 20 moneymaking movies have made extensive use of vfx. It wasn’t the only reason for their financial success but vfx is part of the reason and with some films, it’s very high on the reasons. The studio chooses to make a vfx film. It’s not something we’re pushing on them. A studio makes tradeoffs and may choose to spend more on vfx than on A actors on some projects and just the opposite on other projects. And the thing about vfx is it’s not just one person.  You’ve lumped in all the vfx people costs (which can be hundreds of people) against individuals. When you compare the costs of a full shooting crew on a large vfx movie (including 2nd unit and all the support teams) shooting for 6 months then it’s a much different balance between the ‘expensive vfx crew’ and the shooting crew. The live action crew can be just as expensive, if not more expensive.

Film business: In addition there are quite a few others in the film business making above the national average by week/day/hour. Studio executive, jr executives, editors, DPs, production designers, stunt supervisor, special effects supervisors, sound mixers, DI colorists, etc.
I haven't checked salaries lately but suspect most of the crew make above the national average as well. And why do people make good money in Hollywood? Because they're working on large projects with very good profits if done well. A VFX heavy film will likely make hundreds of millions in profit if done correctly. These people have developed skills and experience that can't be simple bought or learned in a course. They go from project to project so they're not permanently employed. They freelance. They have to be paid more simply to average out and be able to afford the same thing that someone is permanently employed can afford. They put in long hours. Shooting days are 12hr days. Shooting weeks can be 5-7 days. They live in LA and similar areas where the cost of living and housing is above the national average.

Realize the national averages really haven't gone up much in the last 20-30 years.  Most pay has been relatively stagnant with 2% increases a year, if that. Even while many companies have become much more profitable. So where does the extra money go to? The CEO, upper management and shareholders. CEO's 30 years ago made approx 30x their average employee.  It's now over 300x. GE made record profits last year. Paid no taxes. What do they want to do? Cut wages and benefits of their workers.
Why were Verizon workers on strike? Verizon is making very good profits, paying their upper management very, very well and not paying taxes. And yet they're still asking their workers to reduce there benefits. "That loss of health benefits and other givebacks in the proposed contract would net Verizon annually about $1 billion, or $20,000 per worker, according to the unions." ref  And these are benefits that had already been agreed to by both sides and awarded. Now the company wants to reduce them.

From the sounds of it you're in a very sweet spot.  You seem to be permanently employed, paid a very high salary and get paid overtime. "I now make more money than most doctors." Really? You make more than most doctors? What you’re describing is not the average or typical for the majority of vfx workers.

If we're going to even try to compare any of this to the national average there are a few things to do. So let's take a look at what an average vfx worker deals with. First calculate rates 1 1/2x for over 8 hrs and 2x for over 12 hrs to work out an average 40hr week. Most of us work 50 hrs to 90 hr weeks but for comparison we have to put it relative to a 40hr workweek. Don't forget those who work a flat rate (no overtime pay). Use the same formula and see how that works out, especially once you hit 90hrs+ a week. VFX workers who do work a lot of overtime end up sacrificing time with their families and their health in the long run that many other jobs do not.

 "So many of my friends are unemployed right now."
Even at ILM it was common for people to be laid off for 3-6 months during the 'slow' periods and now we're seeing people go for even longer stretches.  So cut the cut the pay period from 12 months to 6-9 months. Now calculate a yearly average or 5-year average.

Oddly when you take that high salary and try to stretch it from 6 months to cover a year, it doesn't look nearly as large. And in the case of most vfx workers there's no guarantee when they will be re-hired. It may seem like the perfect time to take an extended vacation but if you don't know whether you're working again in a few weeks or a year from now it's hard to plan.

Now not everyone in vfx is covered for health care or pension. Even if they were covered if they're off for an extended time or have to switch companies they have to start over. So calculate in the cost of Cobra insurance or self-insurance.  Also calculate a pension fund. Not cheap. And we're adding this because most full time jobs include these as part of their compensation.

If a vfx worker has to work outside of town for a given period and has to cover related expenses (travel, boarding, phone calls, etc) then deduct those as well from the vfx wages. A percentage of non-vfx workers work for companies that offer other benefits. (Discounts on products, discounts on services, profit sharing plans, stock options, bonuses, etc) So now compare the full compensation package of the national average with the average vfx worker and what it costs to create that same level of benefits.  That average vfx salary that may have looked huge on the surface is likely to be much closer to the national average than it first appeared.

Are there 1000's of people who could do what you do? Can they step into the job you're currently at and do just as well? Do you have years of experience, expertise and skills that means you're much more likely to do something much faster and to avoid the pitfalls? Do you have to take responsibility for the project or a team of people? Do you have to manage people? Do you save money for the company even with your salary?  When I work I don't feel bad about being paid well because not only do I accomplish what needs to be done I usually end up saving the vfx company or studio a few times my salary just on avoided expenses. (Avoiding or minimizing costly sets, having the option not to travel the entire cast and crew halfway around the world, improving the pipeline, minimizing the amount of overtime that I can, doing multiple tasks, doing my own mockups, etc)

Companies can and do at times hire poor employees/managers that not only don't accomplish what they need to but that cause losses by making bad decisions. In the end the roto and paint team may have to solve problems because of mistakes made early in the process.

So how does this all work in today’s vfx world?  Yes, workers in China and India are paid less but they typically have a lower cost of living. And even in these places as the workers get more skills and experience their wages are going up.

Why don’t all vfx workers in California cut their wages in half?  Certainly that would have an impact and counter the imbalanced tax incentives? Yes, but probably not in the way most people might expect. First, many vfx companies would be unlikely to pass all of that savings on to the studios.  They would want to keep some to increase their profits and increase salaries of their management. Second, the studios would ask the vfx companies for even lower bids, because after all, the workers cost less.  Those places in other countries and areas would be forced to follow suit because the companies that employee them and the studios would point out those in LA don’t make as much. End result could simply be evenly lower bids with the tax incentives still in place and the work still going else where. And now everyone is paid less.  Doesn't seem to be much of an advantage.

Why did we end up with the salaries in vfx that we do?
It just happened that way over time.."  Nope.
First off there is no standard.  People tend to assume there is because some of the bigger companies have somewhat similar rates. But without a union there’s really no standard rates and people doing the same job at different companies (or even within the same company) could be paid drastically different rates. (and of course different locations and countries differ as well).

Companies don’t just randomly choose to pay a higher rate for workers. Nor do they tend to volunteer to pay increases and other job incentives.

One of the reasons why people are paid the amount they are because when digital vfx started it was made up mainly of union people. ILM was all union when I went to work there. Say what you will about the unions but the fact is in many industries you’re paid a reasonable rate because a group of people organized and asked for given rates. There was a shift to digital as time progressed but many of the people shifted to similar jobs in the digital world if they could. That also meant that 2D animators (with skills and existing experience) tended to move into 3D. Once again people who were already up to speed and able to accomplish the work are worth far more than a number of less expensive people with no experience and that would require training and hand holding.

Those working in VFX are also paid what they are because similar jobs in other industries might pay well. Digital vfx were ramping up as Silicon Valley was at full speed. Whether it’s in graphic arts, software development or other areas, if companies wish to either hire someone from that other industry or want to retain people from going to another industry, then they have to pay a competitive rate. Do you think everyone at Apple, Google, Microsoft and Adobe are being paid minimum wage or even the national average?

And the other reason for the pay to be the way it is because there was a lack of skilled, experienced talent at the beginning and some companies would essentially compete to get key people. The result is there was a spike at one point but much of that has already been scaled back. Will there be more scaling back in the future? Possibly.

But don’t forget the US Justice Dept found that there was collusion between ILM and Pixar regarding non-competing for animators. So that created artificially lower rates for animators and prevented some opportunities for them.

As stated the reason why the rates are where they are elsewhere is due to the rates that have been paid to key vfx people in California. We’re now seeing a flood of vfx students trying to be employed. Many of these people are willing to work for anything, including free and minimum wage. Some would be happy to pay the first year if necessary.  What they don’t understand is by working free or cheap they can end up pushing all wages down, including their future earnings. They may find next year, once they’ve gained more experience, they aren’t given a raise simply because this years batch of students are willing to work for even less. And some producers prey on these types of people and employ essentially students to do their work for free or cheap.

The point here is that we should be looking for solutions to the problems of the vfx industry. Outsourcing and tax incentives are some of the problems. Overtime and overtime pay is another issue. Being forced to be an independent contractor is another problem. I'm sure we could list many more but I'm hoping we focus discussions on potential solutions; throwing out and discussing possible ideas. Simply shrugging our shoulders and saying "woe is me" is not a solution.

It's not easy. This is a global problem but hopefully we'll be able to come up with some ideas to balance this out.

Thanks.

Monday, August 22, 2011

VES 2.0 feedback requested

The VES 2.0 group  (Visual Effects Society)  continues to meet and explore options for the vfx industry.

I'd like to once again ask all vfx artists and companies to submit their thoughts, ideas, solutions and concerns to the VES Leadership group.  Email VES Leadership group You don't have to be a member to submit. You can also post in the comments here or email me directly. **

If you're in the vfx industry you do really owe it to yourself and your future to submit your thoughts. Now is the time, not 6 months from now.

Please consider reading the links on the right side under VFX INDUSTRY - STATE OF THE INDUSTRY heading if you haven't already.  I posted these to try to provide information and different perspectives of some of the issues.  Pass Me That Nail covers some of the key problems with the vfx industry. Using the Nail covers some of the VES 2.0 issues.

I also urge vfx artists to become informed and try to make open minded and balanced decisions and comments. It's easy to repeat the same phrases and become narrow minded, especially under the circumstances. Also keep in mind incentives will not last forever anywhere. Don't assume smooth sailing where you're currently working to last your lifetime. What you perceive as only a US problem could be your problem in the next year or two.

For more info: Visual Effects Society
VES 2.0 Letter to VFX Industry

From Shoot magazine report on VES 2.0


[**Update: Please check out the comments below (you may have to click on the comments link.
I'm listing a few of the types of things to consider in your note and included a few questions from the IA (snippets from the comments)

What are the problems people are facing? You may have some issues I haven't touched on or you may have a totally different priority.

How would you like to see the vfx industry structured?

What should the vfx companies be doing differently to make it better for the workers?

What should the studios be doing to make it better for the workers?

What about the studio and vfx company relationships?

People writing in should mention their views on the IA and why.

What's their views on a vfx trade organization?

From the IA:
"..we continue to seek out some direction as to the type of organization the rank and file really want?
One integrated visual effects organization?
An independent guild?
Affiliation with several different IATSE locals (matching job-skills) such as camera, editorial, animation, or art directors?
We could charter a guild and have it function as a separate entity, benefitting from the experience and strength of the International, and, where appropriate, adopting the Major Studios Basic Agreement. We just need some direction from the affected parties."]



There's also a relatively new take on this which is to focus on management.

While I agree that improving management would help I don't see it being the all in one solution.
Some vfx companies are managed better than others but I don't expect to cut the required time in half or another 20% in profits simply by improved management. Many vfx companies are already reasonably run and while there's room for improvement it's unlikely to be huge. The variables from directors and studios tend to be larger than these gains. Likewise don't assume just because a company is more efficient that it will make more money (it may just get more work) or that benefits, wages and working conditions will improve for employees. The trickle down theory doesn't work. Most general US companies are making 20% more in profits the last couple of years but continue to reduce their workforce and reduce employee benefits. Those added profits go to the executives and shareholders, not the people being more efficient.

What we'll need is improvements/changes to the 3 key parts of this issue (Studios, vfx companies and vfx workers) to get a balance that works and is sustainable.

[Please click on Comments link below if comments aren't automatically shown in your browser]

Thursday, August 04, 2011

2D to 3D Conversions

2D to 3D conversions

2D to 3D conversion is the process of converted a film from 2D (normal film) to a 3D (or stereo) film for viewing with a stereographic viewing system. (polarized glasses, shuttered glasses, special screen, etc)

Conversion had gained a bad name after Clash of the Titans and a few other films that had been rushed through at the last minute. Just as you can’t do a 2000 shot vfx show in 6 weeks with great quality, you can’t do a quality conversion of an entire movie in 6 weeks.

Most people think of a 2D to 3D conversion as a factory type of operation but it’s far from it if it’s to be done right.  Much of my past year was spent working for Legend3D dealing with some interesting and very technically challenging problems. Legend had completed work on Alice (Tea party sequence and Drink Me sequence) and done the conversion on the first 3 Shrek films to be released for 3D TV.

Legend was starting on Transformers: Dark of the Moon and would need help with the extensive visual effects shots. Legend 3D ended up converting over 77 minutes of footage for Transformers.  Most of the vfx shots were converted in a 4-5 month time span. The converted footage was a mix of visual effects shots (from ILM and DD) and non-vfx shots.  In most cases these would be shots intercut back and forth with original stereo footage shot on location.  This raised the complexity much more than if the entire film had been converted. Just as doing vfx in a live action film is a different task than creating an animated film (or virtual sequence).

This was also more difficult than doing a library title or a film that was completed.  The same issues vfx companies face of ever changing edits and creative directions was in full force. In addition the vfx shots were changing and so we had to follow very closely and track all changes such that we could turn around the final converted shots within a day or two of deliver of the last vfx shots.

While at Legend I helped them to develop the pipelines required to do this type of work efficiently and at as high of quality level as possible. Legend had some proprietary software and techniques for different stages in the process. I wrote a number of specialty Nuke plugins and scripts to help leverage these proprietary software packages with existing software. I worked with a number of artists and developers at Legend to review and analyze the shots as they were delivered and worked closely with ILM and DD to get the proper elements that would be required. Both technical and creative issues had to be examined to get the best possible quality. Overall this represented a number of challenges that were unique to the world of stereo conversion and we were able to raise the quality of conversion to a new level.

In what follows I’ll cover the basics of the conversion process. I won’t be able to go into specific detail to due the proprietary nature but hopefully this will provide an understanding of the process.

Conversion versus shooting stereo
Many people are under the mistaken belief that shooting stereo is superior to conversion in every way.  The key to getting great 3D is to design, shoot and edit for 3D, even if it will be converted. If you shoot stereo but don’t consider the 3D world then you will have problems.

In its current form 3D is a stereo process, which means creating imagery for each eye. It’s actually a big cheat when projecting 2 different images to simulate the look of the 3D because your eyes are always focused on the screen but objects appear at other depths. This is a fairly unnatural process.

The following are some of the basic pros and cons. As with everything there are tradeoffs to be made. Shooting stereo does capture the images in 3D but many of the parameters are locked in at the time of shooting. Shooting stereo also requires a fair bit of post work to actually get the images to match. Films that are shot in stereo will still require a certain number of conversion shots just due to problematic shots or limitations while shooting. This was the case even with Avatar.

Shooting Stereo – PROS
Capturing stereo images at time of shooting
Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.
Preview stereo images on set and on location
Footage can be edited and reviewed in stereo context with the correct equipment

Shooting Stereo - CONS
Requires specialized camera rigs
Camera rigs contain 2 cameras so are larger than non-stereo camera setups
More time required to shoot (higher shooting costs)
Requires shooting on digital cameras (no option of film)
Special stereo monitor and glasses required on set to check settings.
Requires a certain amount of adjustment, alignment and cleaning.
Requires special rig technician to handle these adjustments.
Restrictions on lenses that can obtain good looking stereo
Requires locking in stereo depth at time of shooting. This cannot be adjusted in post.
Shooting with ‘toe in’ (non-parallel) camera systems requires a convergence puller similar to a focus puller.  Having the correct convergence point is critical when editing shot to shot.  This convergence point can only be adjusted so far in post. (With parallel camera systems this convergence is all set in post.)
Lens flares and shiny reflections can appear different to both cameras and will require post work to correct to prevent viewing problems of final film.
Because of the polarizing through the beam splitter, water and even asphalt with a sheen will appear different to both cameras and require post correction
Post work required to align and fix footage. Since one camera is shooting through a beam splitter and another is shooting the beam splitter reflection there is typically an imbalance in terms of color, contrast and alignment.
Visual Effects require more work and more rendering (added time and cost)

Stereo Conversion - PROS
Ability to shoot with standard process (Composing with 3D in mind still recommended)
Choice to shoot on film with any film camera or on digital with any digital camera
Unlimited choice of camera lenses (Extreme telephoto still not recommended for best 3D results)
Ability to add depth and volume even on telephoto shots
Ability to set the 3D depth in post on a shot by shot basis
Ability to set the convergence as desired for the edit
Flexibility to adjust the depth and volume of each actor or object in a scene as a creative option not available when shooting stereo.
Visual Effects are handled in the standard way

Stereo Conversion - CONS
Extra time required after edit to properly convert or sequences need to be locked during edit to allow conversion to take place while in post.
Added cost of process
Reflections, smoke, sparks and rain more difficult to convert but not impossible.



The 2D to 3D process
Each company handles conversion in a slightly different way, sometimes with proprietary tools and techniques but what follows are some of the fundamentals.

The first step in the conversion is to have a stereographer review the shot and determine how it will work in 3D space. How much depth budget there will be (how much total depth in the scene) and where the convergence point will be.  The convergence point ends up on the movie screen and imagery in front of the screen is said to be in negative parallax and imagery behind the screen is positive parallax.  If you have two images (left and right eye) of a circle and it’s in the exact same spot for both eyes then it will appear on the screen surface. If you offset the images of the circle toward or away (left/right) from each other so they overlap but don’t match then you’ll have the circle float in front of the screen or behind the screen depending on the direction of offset. The amount of offset will determine how far forward or back it appears.

Convergence decisions are based on the action and edit as well as what is in the scene. Frequently the key actors eyes are used for the convergence point so when cutting from one scene to another the audience doesn’t have to visually look back and forth constantly.

With this information the image is broken down. How much detail depends on where it is in space.  Stereo eyesight primarily works within 20-25 feet. As objects get further away there tends to be less sense of 3D stereo. Breaking down the image may require roto or some form of image area selection. If you’ve done roto or frame by frame extraction you know how difficult and time consuming it is. But in this case everything in the images is broken down, not just a foreground actor. In vfx if you roto a group of people you may be primarily concerned about the group areas that overlaps the area to be replaced. With 2D to 3D you have to isolate each person and each object on a different depth plane.

So imagine the degree of difficulty extracting every 3D surface from a 2D image and making all the frames and transitions match perfectly and smoothly including wild camera moves and explosions with thousands of sparks. Depending on the exact 2D to 3D process used you may have build actual 3D models or at least replicate that process to some extent. If there is a table in the image the table-top will have to be split into top, front and side.

The amount of work and complexity means that if there is a change to the image (edit, reposition, speed up, slow down, etc) it may require redoing the entire shot.

With traditional roto you can get away with things like losing a few fly away hairs on an actress. If they’re clipped off and a new background is composited no one will know. With conversion however any hairs that have not been extracted may end up at the wrong 3D depth. Imagine a character in the foreground with long flowing hair but having the hair stuck to the mountains in the background. Each company leverages what techniques and software they can to help with this process and compositors can also help the process by leveraging various extraction methods.

3D settings
Once the image is actually broken up it is necessary to create the depth. A conversion stereographer may use propriety software or heavily modified Nuke or other application to help with this step. This can be in a true 3D world or it can be simulated 3D with gap offsets.  The stereographer usually works with a stereo monitor and adjusts and animates the very shapes, objects and planes to move in 3D space. This needs to match the surrounding shots and the 3D relationships need to be correct. In the case of Transformers the volume and depth of the objects has to match to the original stereo photography. The show stereographer works with the director as well to provide both creative and technical feedback regarding the depth range and convergence planes.  Cory Turner was the show Stereographer on Transformers.

Viewing stereo footage and making adjustments requires developing an eye for stereo. Not everyone can do it and different people have different aptitude for judging stereo. With most vfx work it’s usually obvious when there are matte edges or a color balance problem between elements. Seeing these types of problems is consistent across a number of individuals. Yet viewing stereo footage is much more subjective. Even stereo experts commonly disagree on the latest 3D film. Some may say it’s the best stereo they’ve ever seen and others will say it’s the worst they’ve seen. Technical issues and creative choices by the stereographer and director will affect the final results.

But even at this stage if all you’ve done is cutout the objects you’ll end up with flat images set at different depths. What you need to do is add volume and real 3D shape to the image that matches. This is one of the areas that 2D to 3D conversion provides an advantage over shooting stereo. Telephoto lens images can be provided more volume than would be possible shooting stereo. Different objects can be given more or less 3D emphasis. An actor needs to have their head rounded, their eyes sunken in and their nose coming out.  This relationship needs to match even as the actor turns his head and the camera moves. There’s a real art to the process and much of this can not be automated. The skill of the artists combined with the tools at their disposal has a lot to do with the final results.

The end result of this process is 2 images – a left and right eye image. If this is a full 3D process there are essentially two 3D CG cameras setup with an InterAxial offset (lens separation just as if shot with a stereo camera system). Some companies use the original image as one eye and generate only the other eye. Legend tends to convert the original footage as if it’s the center image and so they actually generate both eyes. This tends to make sure both eyes match and the image itself isn’t always offset from the original framing.

Because the original image was 2D when sub-images are offset they create gaps where there is no image. The simple way of thinking of this is to start with 2 identical images and cutting out an actor in each. Offset one cutout of the actor 10 pixels to the right and offset the other 10 pixels to the left. Now there’s no image on the edge where the image was moved from. This gap needs to be filled in so you don’t see a black edge in the finished shots. If the gap is really small you can cheat it but these artifacts can be quite noticeable on the big screen. To do it right in most cases requires treating the edge of each object as a full on rig removal. Creating or shooting a clean plate can greatly help with this process.

Keep in mind all of these steps, adjustments and cleanup have to be done on every single shot in the film if the film was originally shot 2D.

Reflections, lens flares, smoke and other problems
And as if that wasn’t tough enough imagine the main actor wearing glasses. The glasses have to be set at the correct depth to relate to the head. Any thumbprints or dirt on the lenses have to be on that same plane, however the actor’s eyes have to be pushed in. More than likely there is a reflection on the glasses of the room, which may be 20 feet away. And the actor may have a highlight on his eye that reflects something 5 feet away.  Each of these different images have to be extracted and removed so you end up with a clean eye with no reflections, the glass reflections, the eye highlight and the thumbprint. This is from live action so requires the compositor to go through a process along the lines of rig removal but splitting in to each depth layer. Each of these layers then has to be placed and positioned in 3D and then composited back together.  If this isn’t done then it will look like the actors eyes and the reflections are simply painted on the surface of the glasses. Needless to say this looks is very wrong in 3D.

Same issues if the shot has glass window, mirror, chrome or other reflective surfaces. The issue of seeing partially transparent layers at different depths also comes into play if you have smoke or fog in the scene. If you have partially transparent smoke in front of a building you don’t want the building to be pulled forward to 10 feet from camera nor do you want the smoke to be pushed all the back to the front of the building.  Same exact problem happens with lens flares.  Each of these has to be removed into a separate element and then added back in at the right depth, usually near the screen plane for lens flares.

Transformers of course was full of fast action with glass windows, smoke, explosions, sparks, and lens flares.

Conversion of VFX shots
VFX shots are usually tough enough without having to deal with two slightly different views for the left and right eye. Working in stereo takes more time and adds an additional level of complexity. When the footage is shot as 2D then the vfx can be finished as 2D shots and approved by the director, which tends to be quicker and easier. But it’s usually best to take advantage of the vfx process when doing the conversion to produce better conversion quicker.

In vfx shots it’s not unusual to add smoke and lens flares among other things. Obviously once these are baked into the shot the conversion artist will have to re-separate them. It makes more sense for speed and quality to simply breakdown the composites in such a way to keep key layers separate and use those layers to set the depth in the conversion stage. Some of the other vfx elements are also useful such as clean plates and potentially some of the roto or mattes.

For shots with CG elements it’s better if the CG elements are rendered separately so there’s no need to extract them and fill in the gaps if only a finished composite was provided. Taking this further the vfx company can render depth maps so the conversion company can leverage it and provide a consistency to the creature (or in this case, transformers)

The majority of 2D vfx shots in Transformers was done this way.  The sheer complexity of the Transformers, the number of moving and changing surfaces, really took advantage of this process and allowed them to be consistent with the stereo vfx shots. A few shots were done where the vfx company rendered 2 views of the transformer and then Legend created the 3D world to go around the rendered stereo image. In a few other cases Legend converted the background to stereo and the vfx company worked from that.

Tips for VFX companies
If the film project you’ll be starting is to be converted make sure to work with production and the conversion company. The conversion company should provide a list of requested elements and the ideal formats. This isn’t simply a matter of turning over a Nuke script.  Chances are your company has proprietary plugins and scripts. It’s also likely that the composites may be very complex yet the conversion company only needs the layers that make sense in depth. You may not typically render depth maps and potentially other elements independently.  The cost and time to do this re-purposing of composites and elements will have to be accounted for either in the vfx bid or as a separate bid. If it’s vfx heavy show it’s likely there will be a small team of people prepping images for the conversion process.

Ideally the production company hires a show stereographer who oversees the process from shot design through to completion.

As with the original photography it’s important for everyone to consider the work as if it were in true 3D space.  No more cheating where animated characters are placed or using one element that is supposed to represent multiple depths. These problems will show themselves once converted to real 3D space.

Compositors need to consider building the composite based on 3D space. When holdouts are built in or there are elements with an odd layer order then it becomes difficult to re-build the composites to work in a dimension conversion.

Work out a naming process for these special conversion elements to make it easy for the conversion company to rebuild the composites.   Example: RL045.layer_1.over.00128.exr

It’s also important to provide any special lookup tables or color specific nodes

Summary
Conversion can create successful stereo images that match images shot in stereo as long as experienced artists are allowed sufficient time and control to do this complex and vfx heavy process correctly. Just as with vfx companies, the selected conversion companies will determine how successful the final results are. Neither visual effects nor conversion are truly commodities that can be randomly assigned to other companies with the same result.

Legend
Legend3D also did at least some of the conversion work on the following features while Transformers 3 was being done: Green Hornet, Pirates of the Caribbean: On Stranger Tides, Green Lantern, Priest,  Smurfs3D.  They were able to leverage many of the improvements created for Transformers 3.

[Update: Forgot to post a link to a VFX Guide Podcast that touches on the conversion.  I believe it's touched on at about 45+ minutes in.]

[Update: I was reminded when responding to a comment below there was a video of Michael Bay and James Cameron discussing shooting stereo.  This was done after the film had been shot and within a few weeks of release.  Video of stereo conversation of Bay and Cameron]

[Update: I got a few tweets from @philwittmer that I thought I'd address here since there's still seems to be some confusion.

Phil:

Sorry but that all seems biased towards 2d conversion. With no mention of roto time. granted its an interesting read I never thought of it like that but.... a couple of major points here need to be made
1. U get limitless depth if shooting 3d as opposed to designated planes
2. Moving foc point in post just means bad directing
lastly and most importantly. The conversion will only be as good as the budget and time. Which unfortunately is usually bad

Me:
Yes, any conversion takes time. Roto takes time, extractions take time, setting depth takes time.
That's why I stated you can't do it in 6 weeks and that was listed as one of the CONs of doing conversion. It is a time consuming process.

If the film has a large number of vfx shots then this conversion can be done (especially on the non-vfx shots) while the vfx are being worked on.  If the vfx shots are locked and can be worked on as you proceed it still takes time but that post time is already used up by the vfx work. That means it doesn't necessarily mean a large delay after the vfx, depending on the show.

Keep in mind all stereo shoot footage needs to be processed as well. Every shot will have to go through a process to be adjusted for color and alignment and other corrections so the left and right images match correctly. This is not an overnight process and so you need to budget time to do shot by shot corrections of the entire film, like DI but can be much more intense. If any shots are truly broken they may need 2D to 3D conversion or serious vfx work to repair them so expect even more time.  This will be less than doing a conversion of the whole film but it's time that should not be discounted.

Unlimited 3D - When shooting 3D you have to determine the amount of depth you want.  And that is not unlimited. You can't have something 1 foot from the viewer and things pushed to infinity without compressing that space.  If you have too much separation in the background you'll get divergence where people's eyes will go walleye (each looking outward). Not pleasant. Too much negative parallax will cause eye strain as well.

Now you can get nice in camera stereo of complex things in the mid distance such as tall grass, leaves on a tree, smoke, etc.  That's why that's a PRO for shooting stereo:  Capturing nuances of complex 3D scenes. Smoke, reflections, rain, leaves, etc.

Designated planes - Just to be clear this isn't like a multiplane camera when you have flat cards at different distances and there are 10 designated planes.  It's not like the process that seemed to result in some viewmaster displays (i.e. all images flat to camera at different depths). Any flat surface can be at any angle. It has to be to actually work in a 3D movie. That road has to be angled away from camera, that building on the side going away from camera, etc. And more importantly you're creating a dimensional space, at least with the better processes.  You have to create the sense of roundness and volume as if you built it as a real 3D model. A shot of a head is a good example (a complex sphere with nose, eye sockets, ears, jaw, etc). You can not get away with having a person on a flat card and expect that to hold up for the audience. Whether the company uses real 3D models, depth maps, complex meshes, etc. you can be sure it's not made up of just designated planes.  And yes, that make it very hard, especially for the items that I listed as easier to shoot. Numerous, complex shapes that are changing in 3D space are very difficult.

During shooting of stereo you have the issue of setting the IA (interaxial)  to set the overall depth. And you have the issue of setting the convergence.  Either of these may be manually changed during the shot depending on the end result.  Is it bad for a director to find out later a different edit works better? What if in planning the shots the director originally thought the shots would cut together in a specific way and shot with that in mind. But once the cuts are changed the conversion may have to be adjusted in post so there won't be as much eye strain. Wide shot to closeup, the timing of the convergence wasn't right, etc. What if the initial IA was high but the fast action caused more strain? The director and stereographer have to really do some planning and testing before shooting. And even then there may be an edited version that doesn't work and needs to be adjusted. And some things like IA can't be adjusted cleanly in post currently. So some 3D decisions made on set are what will be.

>The conversion will only be as good as the budget and time.
This goes for every element of a film. You can't do Avatar in 10 days for $10. You can't record sound for a feature film with your iPhone. If the studio doesn't allow the correct amount of time and money to do the conversion correctly then they will have a movie they spent a lot of money on looking terrible in 3D. And this will cause turning more people away from 3D. The studios are starting to understand the correlation between good 3D and a happy audience willing to pay for 3D movies. And many of these 3D films (shot or converted) are large tent pole movies so there is already a large budget so the studios don't want to gamble and go super cheap.

In the end there are pros and cons to both methods. One of the main points of this article is both can produce good results. Which one to use and what balance of the two depends on the specific project. If production and the director can commit to shooting 3D with a real stereographer and take the necessary care and attention then shooting stereo may be their best choice. Just don't rule out 2D to 3D conversion as one of the options to consider simply because it's been done poorly on some films.

Update: Opensource 2D/3D conversion Gimpel3D for those interested.